2022-09-09T22:20:36.078 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18 2022-09-09T22:20:36.080 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/mbenjamin-2022-09-09_22:09:46-rgw-wip-librgw-frontends-distro-default-smithi/7023840 --name mbenjamin-2022-09-09_22:09:46-rgw-wip-librgw-frontends-distro-default-smithi --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{ubuntu_latest} tasks/test_multi} --verbose --owner scheduled_mbenjamin@teuthology -- /home/teuthworker/archive/mbenjamin-2022-09-09_22:09:46-rgw-wip-librgw-frontends-distro-default-smithi/7023840/orig.config.yaml 2022-09-09T22:20:36.121 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-09-09T22:20:36.194 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mbenjamin-2022-09-09_22:09:46-rgw-wip-librgw-frontends-distro-default-smithi/7023840 branch: wip-librgw-frontends description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{ubuntu_latest} tasks/test_multi} email: null first_in_suite: false job_id: '7023840' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: mbenjamin-2022-09-09_22:09:46-rgw-wip-librgw-frontends-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-librgw-frontends ceph: conf: client: debug rgw: 20 debug rgw sync: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync data inject err probability: 0.1 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) sha1: 28758235ba2ab1c1d8afa0f88d00f07b5687a51e wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 28758235ba2ab1c1d8afa0f88d00f07b5687a51e rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - enabled_features: - resharding endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 rgw-multisite-tests: args: - tests.py - tests_ps.py workunit: branch: wip-librgw-frontends sha1: 28758235ba2ab1c1d8afa0f88d00f07b5687a51e owner: scheduled_mbenjamin@teuthology priority: 70 repo: git://git.ceph.com/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: 28758235ba2ab1c1d8afa0f88d00f07b5687a51e sleep_before_teardown: 0 suite: rgw suite_branch: wip-librgw-frontends suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_28758235ba2ab1c1d8afa0f88d00f07b5687a51e/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 28758235ba2ab1c1d8afa0f88d00f07b5687a51e targets: smithi005.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzFKnoOt2BoS/UBn9KunV/1GYUVB2AcY7cWUKACXEAWiSig9PUvOMwoMCkSvUbgs5stNzMZn58M3TydDGcq1zQwW7asez3uAzAi387ckCknkDmTFJdbrHLGna2mHGF6eL74cRn+9grca/oUUwKrTEQsW8bsgiHtL0gTV/f1rTK/3pyDBRSyy6gfPbIU3LK63P4/e8rRTNm57Jr/RWGt1TwV7dTiHSbBrAJpxD/6qY/4q4fsVM7HJZHB7rmV8tdeoB85+15PDy8JjoyxBdlp1UOE7NIrvBQzu+9V0nR2UvH8N4ryDBG8N1XfHsD9R+3Zg2lCYHenXcKJupnY6xNoSlJfHfkwsard+2GIy+4aJ57s6c3K+UKjnqdxib40X9RdJzNs2Qc+ARp+zbO+sAamr4wWWZFb0nhZUO89u3INoFbpT5gC3mnCI+p2YTeHZG/kJ8U14RnQ37Mai06jQeqFIHivFr8JX/3/pfWiBXg/9bl3y/37pU24aQ/dtBg/K87W68= smithi080.front.sepia.ceph.com: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIC0XOORBR/2Qj/b6x+ngeEme2+rTgZ2w3aDFr4sjpDmu tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b58f2c18636eb10faa77ed3614abd00cb85dfc2c timestamp: 2022-09-09_22:09:46 tube: smithi user: mbenjamin verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1713 2022-09-09T22:20:36.194 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_28758235ba2ab1c1d8afa0f88d00f07b5687a51e/qa; will attempt to use it 2022-09-09T22:20:36.195 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_28758235ba2ab1c1d8afa0f88d00f07b5687a51e/qa/tasks 2022-09-09T22:20:36.196 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-09-09T22:20:36.198 INFO:teuthology.task.internal:Checking packages... 2022-09-09T22:20:36.228 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '28758235ba2ab1c1d8afa0f88d00f07b5687a51e' 2022-09-09T22:20:36.229 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-09-09T22:20:36.229 INFO:teuthology.packaging:ref: None 2022-09-09T22:20:36.230 INFO:teuthology.packaging:tag: None 2022-09-09T22:20:36.230 INFO:teuthology.packaging:branch: wip-librgw-frontends 2022-09-09T22:20:36.230 INFO:teuthology.packaging:sha1: 28758235ba2ab1c1d8afa0f88d00f07b5687a51e 2022-09-09T22:20:36.231 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-librgw-frontends 2022-09-09T22:20:36.445 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-14894-g28758235-1focal 2022-09-09T22:20:36.446 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-09-09T22:20:36.453 INFO:teuthology.task.internal:no buildpackages task found 2022-09-09T22:20:36.453 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-09-09T22:20:36.489 INFO:teuthology.task.internal:Saving configuration 2022-09-09T22:20:36.505 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-09-09T22:20:36.523 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-09-09T22:20:36.550 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi005.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mbenjamin-2022-09-09_22:09:46-rgw-wip-librgw-frontends-distro-default-smithi/7023840', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-09-09 22:15:09.666500', 'locked_by': 'scheduled_mbenjamin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINWewJ4SmPkioIZfEj9dKcbVXV2aqirRxwwpDsYC9h9g'} 2022-09-09T22:20:36.573 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi080.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mbenjamin-2022-09-09_22:09:46-rgw-wip-librgw-frontends-distro-default-smithi/7023840', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-09-09 22:15:09.664522', 'locked_by': 'scheduled_mbenjamin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIC0XOORBR/2Qj/b6x+ngeEme2+rTgZ2w3aDFr4sjpDmu'} 2022-09-09T22:20:36.573 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-09-09T22:20:36.580 INFO:teuthology.task.internal:roles: ubuntu@smithi005.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2022-09-09T22:20:36.580 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2022-09-09T22:20:36.581 INFO:teuthology.run_tasks:Running task console_log... 2022-09-09T22:20:36.682 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f661b26a400>, signals=[15]) 2022-09-09T22:20:36.684 INFO:teuthology.run_tasks:Running task internal.connect... 2022-09-09T22:20:36.691 INFO:teuthology.task.internal:Opening connections... 2022-09-09T22:20:36.692 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi005.front.sepia.ceph.com 2022-09-09T22:20:36.695 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-09T22:20:36.819 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi080.front.sepia.ceph.com 2022-09-09T22:20:36.820 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-09T22:20:36.914 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-09-09T22:20:36.922 DEBUG:teuthology.orchestra.run.smithi005:> uname -m 2022-09-09T22:20:36.930 INFO:teuthology.orchestra.run.smithi005.stdout:x86_64 2022-09-09T22:20:36.930 DEBUG:teuthology.orchestra.run.smithi005:> cat /etc/os-release 2022-09-09T22:20:36.977 INFO:teuthology.orchestra.run.smithi005.stdout:NAME="Ubuntu" 2022-09-09T22:20:36.978 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION="20.04.5 LTS (Focal Fossa)" 2022-09-09T22:20:36.978 INFO:teuthology.orchestra.run.smithi005.stdout:ID=ubuntu 2022-09-09T22:20:36.978 INFO:teuthology.orchestra.run.smithi005.stdout:ID_LIKE=debian 2022-09-09T22:20:36.978 INFO:teuthology.orchestra.run.smithi005.stdout:PRETTY_NAME="Ubuntu 20.04.5 LTS" 2022-09-09T22:20:36.979 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION_ID="20.04" 2022-09-09T22:20:36.979 INFO:teuthology.orchestra.run.smithi005.stdout:HOME_URL="https://www.ubuntu.com/" 2022-09-09T22:20:36.979 INFO:teuthology.orchestra.run.smithi005.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-09-09T22:20:36.979 INFO:teuthology.orchestra.run.smithi005.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-09-09T22:20:36.980 INFO:teuthology.orchestra.run.smithi005.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-09-09T22:20:36.980 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION_CODENAME=focal 2022-09-09T22:20:36.980 INFO:teuthology.orchestra.run.smithi005.stdout:UBUNTU_CODENAME=focal 2022-09-09T22:20:36.981 INFO:teuthology.lock.ops:Updating smithi005.front.sepia.ceph.com on lock server 2022-09-09T22:20:37.006 DEBUG:teuthology.orchestra.run.smithi080:> uname -m 2022-09-09T22:20:37.014 INFO:teuthology.orchestra.run.smithi080.stdout:x86_64 2022-09-09T22:20:37.014 DEBUG:teuthology.orchestra.run.smithi080:> cat /etc/os-release 2022-09-09T22:20:37.021 INFO:teuthology.orchestra.run.smithi080.stdout:NAME="Ubuntu" 2022-09-09T22:20:37.021 INFO:teuthology.orchestra.run.smithi080.stdout:VERSION="20.04.5 LTS (Focal Fossa)" 2022-09-09T22:20:37.021 INFO:teuthology.orchestra.run.smithi080.stdout:ID=ubuntu 2022-09-09T22:20:37.022 INFO:teuthology.orchestra.run.smithi080.stdout:ID_LIKE=debian 2022-09-09T22:20:37.022 INFO:teuthology.orchestra.run.smithi080.stdout:PRETTY_NAME="Ubuntu 20.04.5 LTS" 2022-09-09T22:20:37.023 INFO:teuthology.orchestra.run.smithi080.stdout:VERSION_ID="20.04" 2022-09-09T22:20:37.023 INFO:teuthology.orchestra.run.smithi080.stdout:HOME_URL="https://www.ubuntu.com/" 2022-09-09T22:20:37.023 INFO:teuthology.orchestra.run.smithi080.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-09-09T22:20:37.023 INFO:teuthology.orchestra.run.smithi080.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-09-09T22:20:37.024 INFO:teuthology.orchestra.run.smithi080.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-09-09T22:20:37.024 INFO:teuthology.orchestra.run.smithi080.stdout:VERSION_CODENAME=focal 2022-09-09T22:20:37.024 INFO:teuthology.orchestra.run.smithi080.stdout:UBUNTU_CODENAME=focal 2022-09-09T22:20:37.024 INFO:teuthology.lock.ops:Updating smithi080.front.sepia.ceph.com on lock server 2022-09-09T22:20:37.043 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-09-09T22:20:37.056 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-09-09T22:20:37.080 INFO:teuthology.task.internal:Checking for old test directory... 2022-09-09T22:20:37.081 DEBUG:teuthology.orchestra.run.smithi005:> test '!' -e /home/ubuntu/cephtest 2022-09-09T22:20:37.084 DEBUG:teuthology.orchestra.run.smithi080:> test '!' -e /home/ubuntu/cephtest 2022-09-09T22:20:37.090 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-09-09T22:20:37.114 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-09-09T22:20:37.114 DEBUG:teuthology.orchestra.run.smithi005:> test -z $(ls -A /var/lib/ceph) 2022-09-09T22:20:37.117 DEBUG:teuthology.orchestra.run.smithi080:> test -z $(ls -A /var/lib/ceph) 2022-09-09T22:20:37.132 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-09-09T22:20:37.196 INFO:teuthology.run_tasks:Running task kernel... 2022-09-09T22:20:37.212 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-09-09T22:20:37.212 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2022-09-09T22:20:37.213 DEBUG:teuthology.orchestra.run.smithi005:> test -f /run/.containerenv -o -f /.dockerenv 2022-09-09T22:20:37.214 DEBUG:teuthology.orchestra.run.smithi080:> test -f /run/.containerenv -o -f /.dockerenv 2022-09-09T22:20:37.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-09T22:20:37.220 DEBUG:teuthology.orchestra.run.smithi005:> uname -r 2022-09-09T22:20:37.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-09T22:20:37.222 DEBUG:teuthology.orchestra.run.smithi080:> uname -r 2022-09-09T22:20:37.229 INFO:teuthology.orchestra.run.smithi005.stdout:5.4.0-125-generic 2022-09-09T22:20:37.229 INFO:teuthology.task.kernel:Running kernel on smithi005: 5.4.0-125-generic 2022-09-09T22:20:37.230 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-get clean 2022-09-09T22:20:37.271 INFO:teuthology.orchestra.run.smithi080.stdout:5.4.0-125-generic 2022-09-09T22:20:37.271 INFO:teuthology.task.kernel:Running kernel on smithi080: 5.4.0-125-generic 2022-09-09T22:20:37.272 DEBUG:teuthology.orchestra.run.smithi080:> sudo apt-get clean 2022-09-09T22:20:37.371 DEBUG:teuthology.orchestra.run.smithi080:> sudo apt-get update 2022-09-09T22:20:37.385 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-get update 2022-09-09T22:20:37.529 INFO:teuthology.orchestra.run.smithi080.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-09-09T22:20:37.530 INFO:teuthology.orchestra.run.smithi080.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-09-09T22:20:37.532 INFO:teuthology.orchestra.run.smithi080.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-09-09T22:20:37.564 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-09-09T22:20:37.565 INFO:teuthology.orchestra.run.smithi005.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-09-09T22:20:37.606 INFO:teuthology.orchestra.run.smithi080.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-09-09T22:20:37.626 INFO:teuthology.orchestra.run.smithi005.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-09-09T22:20:37.657 INFO:teuthology.orchestra.run.smithi005.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-09-09T22:20:37.894 INFO:teuthology.orchestra.run.smithi080.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,707 kB] 2022-09-09T22:20:37.954 INFO:teuthology.orchestra.run.smithi005.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [714 kB] 2022-09-09T22:20:38.033 INFO:teuthology.orchestra.run.smithi080.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [714 kB] 2022-09-09T22:20:38.033 INFO:teuthology.orchestra.run.smithi080.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [486 kB] 2022-09-09T22:20:38.039 INFO:teuthology.orchestra.run.smithi005.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,077 kB] 2022-09-09T22:20:38.042 INFO:teuthology.orchestra.run.smithi080.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [287 kB] 2022-09-09T22:20:38.046 INFO:teuthology.orchestra.run.smithi080.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [11.0 kB] 2022-09-09T22:20:38.046 INFO:teuthology.orchestra.run.smithi080.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1,169 kB] 2022-09-09T22:20:38.057 INFO:teuthology.orchestra.run.smithi080.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [166 kB] 2022-09-09T22:20:38.073 INFO:teuthology.orchestra.run.smithi005.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [369 kB] 2022-09-09T22:20:38.076 INFO:teuthology.orchestra.run.smithi005.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [15.9 kB] 2022-09-09T22:20:38.077 INFO:teuthology.orchestra.run.smithi005.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,256 kB] 2022-09-09T22:20:38.086 INFO:teuthology.orchestra.run.smithi005.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [178 kB] 2022-09-09T22:20:38.089 INFO:teuthology.orchestra.run.smithi005.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [949 kB] 2022-09-09T22:20:38.096 INFO:teuthology.orchestra.run.smithi005.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [691 kB] 2022-09-09T22:20:38.097 INFO:teuthology.orchestra.run.smithi080.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,077 kB] 2022-09-09T22:20:38.100 INFO:teuthology.orchestra.run.smithi005.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [216 kB] 2022-09-09T22:20:38.102 INFO:teuthology.orchestra.run.smithi005.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [21.4 kB] 2022-09-09T22:20:38.138 INFO:teuthology.orchestra.run.smithi080.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [369 kB] 2022-09-09T22:20:38.138 INFO:teuthology.orchestra.run.smithi080.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [15.9 kB] 2022-09-09T22:20:38.138 INFO:teuthology.orchestra.run.smithi080.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,256 kB] 2022-09-09T22:20:38.145 INFO:teuthology.orchestra.run.smithi080.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [178 kB] 2022-09-09T22:20:38.146 INFO:teuthology.orchestra.run.smithi080.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [949 kB] 2022-09-09T22:20:38.152 INFO:teuthology.orchestra.run.smithi080.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [691 kB] 2022-09-09T22:20:38.157 INFO:teuthology.orchestra.run.smithi080.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [216 kB] 2022-09-09T22:20:38.158 INFO:teuthology.orchestra.run.smithi080.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [21.4 kB] 2022-09-09T22:20:38.275 INFO:teuthology.orchestra.run.smithi005.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [486 kB] 2022-09-09T22:20:38.558 INFO:teuthology.orchestra.run.smithi005.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,707 kB] 2022-09-09T22:20:38.733 INFO:teuthology.orchestra.run.smithi005.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [287 kB] 2022-09-09T22:20:38.746 INFO:teuthology.orchestra.run.smithi005.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [11.0 kB] 2022-09-09T22:20:38.747 INFO:teuthology.orchestra.run.smithi005.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1,169 kB] 2022-09-09T22:20:38.801 INFO:teuthology.orchestra.run.smithi005.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [166 kB] 2022-09-09T22:20:41.069 INFO:teuthology.orchestra.run.smithi080.stdout:Fetched 10.6 MB in 2s (6,082 kB/s) 2022-09-09T22:20:41.496 INFO:teuthology.orchestra.run.smithi005.stdout:Fetched 10.6 MB in 2s (5,949 kB/s) 2022-09-09T22:20:42.446 INFO:teuthology.orchestra.run.smithi080.stdout:Reading package lists... 2022-09-09T22:20:42.470 DEBUG:teuthology.orchestra.run.smithi080:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-09-09T22:20:42.537 INFO:teuthology.orchestra.run.smithi080.stdout:Reading package lists... 2022-09-09T22:20:42.723 INFO:teuthology.orchestra.run.smithi080.stdout:Building dependency tree... 2022-09-09T22:20:42.724 INFO:teuthology.orchestra.run.smithi080.stdout:Reading state information... 2022-09-09T22:20:42.876 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2022-09-09T22:20:42.902 DEBUG:teuthology.orchestra.run.smithi005:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-09-09T22:20:42.945 INFO:teuthology.orchestra.run.smithi080.stdout:linux-image-generic is already the newest version (5.4.0.125.126). 2022-09-09T22:20:42.945 INFO:teuthology.orchestra.run.smithi080.stdout:linux-image-generic set to manually installed. 2022-09-09T22:20:42.946 INFO:teuthology.orchestra.run.smithi080.stdout:0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. 2022-09-09T22:20:42.948 DEBUG:teuthology.orchestra.run.smithi080:> dpkg -s linux-image-generic 2022-09-09T22:20:42.969 INFO:teuthology.orchestra.run.smithi080.stdout:Package: linux-image-generic 2022-09-09T22:20:42.970 INFO:teuthology.orchestra.run.smithi080.stdout:Status: install ok installed 2022-09-09T22:20:42.970 INFO:teuthology.orchestra.run.smithi080.stdout:Priority: optional 2022-09-09T22:20:42.970 INFO:teuthology.orchestra.run.smithi080.stdout:Section: kernel 2022-09-09T22:20:42.971 INFO:teuthology.orchestra.run.smithi080.stdout:Installed-Size: 19 2022-09-09T22:20:42.971 INFO:teuthology.orchestra.run.smithi080.stdout:Maintainer: Ubuntu Kernel Team 2022-09-09T22:20:42.971 INFO:teuthology.orchestra.run.smithi080.stdout:Architecture: amd64 2022-09-09T22:20:42.971 INFO:teuthology.orchestra.run.smithi080.stdout:Source: linux-meta 2022-09-09T22:20:42.972 INFO:teuthology.orchestra.run.smithi080.stdout:Version: 5.4.0.125.126 2022-09-09T22:20:42.972 INFO:teuthology.orchestra.run.smithi080.stdout:Provides: virtualbox-guest-modules (= 6.1.34-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-09-09T22:20:42.972 INFO:teuthology.orchestra.run.smithi080.stdout:Depends: linux-image-5.4.0-125-generic, linux-modules-extra-5.4.0-125-generic, linux-firmware, intel-microcode, amd64-microcode 2022-09-09T22:20:42.972 INFO:teuthology.orchestra.run.smithi080.stdout:Recommends: thermald 2022-09-09T22:20:42.973 INFO:teuthology.orchestra.run.smithi080.stdout:Description: Generic Linux kernel image 2022-09-09T22:20:42.973 INFO:teuthology.orchestra.run.smithi080.stdout: This package will always depend on the latest generic kernel image 2022-09-09T22:20:42.973 INFO:teuthology.orchestra.run.smithi080.stdout: available. 2022-09-09T22:20:42.974 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-09-09T22:20:42.974 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2022-09-09T22:20:42.974 DEBUG:teuthology.orchestra.run.smithi080:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-09-09T22:20:42.975 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2022-09-09T22:20:42.993 INFO:teuthology.orchestra.run.smithi080.stdout:ttyS1 2022-09-09T22:20:43.015 DEBUG:teuthology.parallel:result is None 2022-09-09T22:20:43.156 INFO:teuthology.orchestra.run.smithi005.stdout:Building dependency tree... 2022-09-09T22:20:43.156 INFO:teuthology.orchestra.run.smithi005.stdout:Reading state information... 2022-09-09T22:20:43.367 INFO:teuthology.orchestra.run.smithi005.stdout:linux-image-generic is already the newest version (5.4.0.125.126). 2022-09-09T22:20:43.368 INFO:teuthology.orchestra.run.smithi005.stdout:linux-image-generic set to manually installed. 2022-09-09T22:20:43.368 INFO:teuthology.orchestra.run.smithi005.stdout:0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. 2022-09-09T22:20:43.370 DEBUG:teuthology.orchestra.run.smithi005:> dpkg -s linux-image-generic 2022-09-09T22:20:43.391 INFO:teuthology.orchestra.run.smithi005.stdout:Package: linux-image-generic 2022-09-09T22:20:43.392 INFO:teuthology.orchestra.run.smithi005.stdout:Status: install ok installed 2022-09-09T22:20:43.392 INFO:teuthology.orchestra.run.smithi005.stdout:Priority: optional 2022-09-09T22:20:43.392 INFO:teuthology.orchestra.run.smithi005.stdout:Section: kernel 2022-09-09T22:20:43.392 INFO:teuthology.orchestra.run.smithi005.stdout:Installed-Size: 19 2022-09-09T22:20:43.393 INFO:teuthology.orchestra.run.smithi005.stdout:Maintainer: Ubuntu Kernel Team 2022-09-09T22:20:43.393 INFO:teuthology.orchestra.run.smithi005.stdout:Architecture: amd64 2022-09-09T22:20:43.393 INFO:teuthology.orchestra.run.smithi005.stdout:Source: linux-meta 2022-09-09T22:20:43.393 INFO:teuthology.orchestra.run.smithi005.stdout:Version: 5.4.0.125.126 2022-09-09T22:20:43.393 INFO:teuthology.orchestra.run.smithi005.stdout:Provides: virtualbox-guest-modules (= 6.1.34-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-09-09T22:20:43.394 INFO:teuthology.orchestra.run.smithi005.stdout:Depends: linux-image-5.4.0-125-generic, linux-modules-extra-5.4.0-125-generic, linux-firmware, intel-microcode, amd64-microcode 2022-09-09T22:20:43.394 INFO:teuthology.orchestra.run.smithi005.stdout:Recommends: thermald 2022-09-09T22:20:43.394 INFO:teuthology.orchestra.run.smithi005.stdout:Description: Generic Linux kernel image 2022-09-09T22:20:43.395 INFO:teuthology.orchestra.run.smithi005.stdout: This package will always depend on the latest generic kernel image 2022-09-09T22:20:43.395 INFO:teuthology.orchestra.run.smithi005.stdout: available. 2022-09-09T22:20:43.395 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-09-09T22:20:43.396 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2022-09-09T22:20:43.396 DEBUG:teuthology.orchestra.run.smithi005:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-09-09T22:20:43.460 INFO:teuthology.orchestra.run.smithi005.stdout:ttyS1 2022-09-09T22:20:43.485 DEBUG:teuthology.parallel:result is None 2022-09-09T22:20:43.486 INFO:teuthology.run_tasks:Running task internal.base... 2022-09-09T22:20:43.494 INFO:teuthology.task.internal:Creating test directory... 2022-09-09T22:20:43.495 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-09-09T22:20:43.497 DEBUG:teuthology.orchestra.run.smithi080:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-09-09T22:20:43.504 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-09-09T22:20:43.530 INFO:teuthology.run_tasks:Running task internal.archive... 2022-09-09T22:20:43.566 INFO:teuthology.task.internal:Creating archive directory... 2022-09-09T22:20:43.566 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-09-09T22:20:43.569 DEBUG:teuthology.orchestra.run.smithi080:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-09-09T22:20:43.585 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-09-09T22:20:43.597 INFO:teuthology.task.internal:Enabling coredump saving... 2022-09-09T22:20:43.598 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-09-09T22:20:43.617 DEBUG:teuthology.orchestra.run.smithi080:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-09-09T22:20:43.642 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-09-09T22:20:43.649 INFO:teuthology.orchestra.run.smithi080.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-09-09T22:20:43.653 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-09-09T22:20:43.660 INFO:teuthology.orchestra.run.smithi080.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-09-09T22:20:43.663 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-09-09T22:20:43.686 INFO:teuthology.task.internal:Configuring sudo... 2022-09-09T22:20:43.687 DEBUG:teuthology.orchestra.run.smithi005:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-09-09T22:20:43.690 DEBUG:teuthology.orchestra.run.smithi080:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-09-09T22:20:43.720 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-09-09T22:20:43.731 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-09-09T22:20:43.732 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-09-09T22:20:43.750 DEBUG:teuthology.orchestra.run.smithi080:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-09-09T22:20:43.759 DEBUG:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-09-09T22:20:43.767 DEBUG:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-09-09T22:20:43.818 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-09-09T22:20:43.818 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-09-09T22:20:43.838 DEBUG:teuthology.orchestra.run.smithi080:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-09-09T22:20:43.845 DEBUG:teuthology.orchestra.run.smithi080:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-09-09T22:20:43.853 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2022-09-09T22:20:43.853 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-09-09T22:20:43.871 DEBUG:teuthology.orchestra.run.smithi005:> sudo service rsyslog restart 2022-09-09T22:20:43.874 DEBUG:teuthology.orchestra.run.smithi080:> sudo service rsyslog restart 2022-09-09T22:20:43.955 INFO:teuthology.run_tasks:Running task internal.timer... 2022-09-09T22:20:43.963 INFO:teuthology.task.internal:Starting timer... 2022-09-09T22:20:43.963 INFO:teuthology.run_tasks:Running task pcp... 2022-09-09T22:20:44.014 INFO:teuthology.run_tasks:Running task selinux... 2022-09-09T22:20:44.042 DEBUG:teuthology.task.selinux:Excluding smithi005: OS 'ubuntu' does not support SELinux 2022-09-09T22:20:44.043 DEBUG:teuthology.task.selinux:Excluding smithi080: OS 'ubuntu' does not support SELinux 2022-09-09T22:20:44.043 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-09-09T22:20:44.044 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-09-09T22:20:44.044 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-09-09T22:20:44.044 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-09-09T22:20:44.075 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2022-09-09T22:20:44.084 INFO:teuthology.repo_utils:Fetching main from origin 2022-09-09T22:20:44.129 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main to origin/main 2022-09-09T22:20:44.163 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-09-09T22:20:44.165 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi005.front.sepia.ceph.com,smithi080.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main/cephlab.yml 2022-09-09T22:23:13.265 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi005.front.sepia.ceph.com'), Remote(name='ubuntu@smithi080.front.sepia.ceph.com')] 2022-09-09T22:23:13.268 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2022-09-09T22:23:13.269 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-09T22:23:13.385 DEBUG:teuthology.orchestra.run.smithi005:> true 2022-09-09T22:23:13.512 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2022-09-09T22:23:13.514 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-09T22:23:13.615 DEBUG:teuthology.orchestra.run.smithi080:> true 2022-09-09T22:23:13.744 INFO:teuthology.run_tasks:Running task clock... 2022-09-09T22:23:13.763 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-09-09T22:23:13.764 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-09-09T22:23:13.768 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-09-09T22:23:13.776 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-09-09T22:23:13.778 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-09-09T22:23:13.805 INFO:teuthology.orchestra.run.smithi005.stdout: 9 Sep 22:23:13 ntpd[8463]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-09-09T22:23:13.806 INFO:teuthology.orchestra.run.smithi005.stdout: 9 Sep 22:23:13 ntpd[8463]: Command line: ntpd -gq 2022-09-09T22:23:13.807 INFO:teuthology.orchestra.run.smithi005.stdout: 9 Sep 22:23:13 ntpd[8463]: proto: precision = 0.073 usec (-24) 2022-09-09T22:23:13.807 INFO:teuthology.orchestra.run.smithi005.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-09-09T22:23:13.809 INFO:teuthology.orchestra.run.smithi005.stdout: 9 Sep 22:23:13 ntpd[8463]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-09-09T22:23:13.809 INFO:teuthology.orchestra.run.smithi005.stderr:restrict ::: KOD does nothing without LIMITED. 2022-09-09T22:23:13.811 INFO:teuthology.orchestra.run.smithi005.stdout: 9 Sep 22:23:13 ntpd[8463]: restrict ::: KOD does nothing without LIMITED. 2022-09-09T22:23:13.812 INFO:teuthology.orchestra.run.smithi005.stdout: 9 Sep 22:23:13 ntpd[8463]: Listen and drop on 0 v6wildcard [::]:123 2022-09-09T22:23:13.812 INFO:teuthology.orchestra.run.smithi005.stdout: 9 Sep 22:23:13 ntpd[8463]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-09-09T22:23:13.813 INFO:teuthology.orchestra.run.smithi005.stdout: 9 Sep 22:23:13 ntpd[8463]: Listen normally on 2 lo 127.0.0.1:123 2022-09-09T22:23:13.813 INFO:teuthology.orchestra.run.smithi005.stdout: 9 Sep 22:23:13 ntpd[8463]: Listen normally on 3 ens1f0 172.21.15.5:123 2022-09-09T22:23:13.813 INFO:teuthology.orchestra.run.smithi005.stdout: 9 Sep 22:23:13 ntpd[8463]: Listen normally on 4 lo [::1]:123 2022-09-09T22:23:13.814 INFO:teuthology.orchestra.run.smithi005.stdout: 9 Sep 22:23:13 ntpd[8463]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1442%4]:123 2022-09-09T22:23:13.814 INFO:teuthology.orchestra.run.smithi005.stdout: 9 Sep 22:23:13 ntpd[8463]: Listening on routing socket on fd #22 for interface updates 2022-09-09T22:23:13.817 INFO:teuthology.orchestra.run.smithi080.stdout: 9 Sep 22:23:14 ntpd[8717]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-09-09T22:23:13.817 INFO:teuthology.orchestra.run.smithi080.stdout: 9 Sep 22:23:14 ntpd[8717]: Command line: ntpd -gq 2022-09-09T22:23:13.818 INFO:teuthology.orchestra.run.smithi080.stdout: 9 Sep 22:23:14 ntpd[8717]: proto: precision = 0.120 usec (-23) 2022-09-09T22:23:13.820 INFO:teuthology.orchestra.run.smithi080.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-09-09T22:23:13.820 INFO:teuthology.orchestra.run.smithi080.stdout: 9 Sep 22:23:14 ntpd[8717]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-09-09T22:23:13.821 INFO:teuthology.orchestra.run.smithi080.stderr:restrict ::: KOD does nothing without LIMITED. 2022-09-09T22:23:13.823 INFO:teuthology.orchestra.run.smithi080.stdout: 9 Sep 22:23:14 ntpd[8717]: restrict ::: KOD does nothing without LIMITED. 2022-09-09T22:23:13.823 INFO:teuthology.orchestra.run.smithi080.stdout: 9 Sep 22:23:14 ntpd[8717]: Listen and drop on 0 v6wildcard [::]:123 2022-09-09T22:23:13.824 INFO:teuthology.orchestra.run.smithi080.stdout: 9 Sep 22:23:14 ntpd[8717]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-09-09T22:23:13.824 INFO:teuthology.orchestra.run.smithi080.stdout: 9 Sep 22:23:14 ntpd[8717]: Listen normally on 2 lo 127.0.0.1:123 2022-09-09T22:23:13.824 INFO:teuthology.orchestra.run.smithi080.stdout: 9 Sep 22:23:14 ntpd[8717]: Listen normally on 3 enp3s0f1 172.21.15.80:123 2022-09-09T22:23:13.824 INFO:teuthology.orchestra.run.smithi080.stdout: 9 Sep 22:23:14 ntpd[8717]: Listen normally on 4 lo [::1]:123 2022-09-09T22:23:13.824 INFO:teuthology.orchestra.run.smithi080.stdout: 9 Sep 22:23:14 ntpd[8717]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cba9%5]:123 2022-09-09T22:23:13.825 INFO:teuthology.orchestra.run.smithi080.stdout: 9 Sep 22:23:14 ntpd[8717]: Listening on routing socket on fd #22 for interface updates 2022-09-09T22:23:14.819 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:15 ntpd[8717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:14.820 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:15 ntpd[8717]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:14.820 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:15 ntpd[8717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:14.820 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:15 ntpd[8717]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:14.820 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:15 ntpd[8717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:14.821 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:15 ntpd[8717]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:14.821 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:15 ntpd[8717]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-09-09T22:23:14.821 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:15 ntpd[8717]: can't open /var/log/ntpstats/peerstats.20220909: Permission denied 2022-09-09T22:23:14.821 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:15 ntpd[8717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:14.822 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:15 ntpd[8717]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:14.822 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:15 ntpd[8717]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-09-09T22:23:14.822 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:15 ntpd[8717]: can't open /var/log/ntpstats/peerstats.20220909: Permission denied 2022-09-09T22:23:15.808 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:15 ntpd[8463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:15.808 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:15 ntpd[8463]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:15.808 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:15 ntpd[8463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:15.809 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:15 ntpd[8463]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:15.809 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:15 ntpd[8463]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-09-09T22:23:15.809 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:15 ntpd[8463]: can't open /var/log/ntpstats/peerstats.20220909: Permission denied 2022-09-09T22:23:15.817 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:16 ntpd[8717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:15.817 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:16 ntpd[8717]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:15.819 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:16 ntpd[8717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:15.819 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:16 ntpd[8717]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:15.819 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:16 ntpd[8717]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-09-09T22:23:15.820 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:16 ntpd[8717]: can't open /var/log/ntpstats/peerstats.20220909: Permission denied 2022-09-09T22:23:16.807 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:16 ntpd[8463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:16.807 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:16 ntpd[8463]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:16.808 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:16 ntpd[8463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:16.808 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:16 ntpd[8463]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:16.808 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:16 ntpd[8463]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-09-09T22:23:16.808 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:16 ntpd[8463]: can't open /var/log/ntpstats/peerstats.20220909: Permission denied 2022-09-09T22:23:16.820 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:17 ntpd[8717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:16.821 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:17 ntpd[8717]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:16.821 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:17 ntpd[8717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:16.821 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:17 ntpd[8717]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:16.822 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:17 ntpd[8717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:16.822 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:17 ntpd[8717]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:16.823 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:17 ntpd[8717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:16.824 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:17 ntpd[8717]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:16.824 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:17 ntpd[8717]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-09-09T22:23:16.825 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:17 ntpd[8717]: can't open /var/log/ntpstats/peerstats.20220909: Permission denied 2022-09-09T22:23:16.825 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:17 ntpd[8717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:16.826 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:17 ntpd[8717]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:16.826 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:17 ntpd[8717]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-09-09T22:23:16.827 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:17 ntpd[8717]: can't open /var/log/ntpstats/peerstats.20220909: Permission denied 2022-09-09T22:23:17.809 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:17 ntpd[8463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:17.809 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:17 ntpd[8463]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:17.810 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:17 ntpd[8463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:17.811 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:17 ntpd[8463]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:17.811 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:17 ntpd[8463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:17.811 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:17 ntpd[8463]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:17.812 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:17 ntpd[8463]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-09-09T22:23:17.812 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:17 ntpd[8463]: can't open /var/log/ntpstats/peerstats.20220909: Permission denied 2022-09-09T22:23:17.813 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:17 ntpd[8463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:17.813 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:17 ntpd[8463]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:17.814 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:17 ntpd[8463]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-09-09T22:23:17.814 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:17 ntpd[8463]: can't open /var/log/ntpstats/peerstats.20220909: Permission denied 2022-09-09T22:23:17.817 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:18 ntpd[8717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:17.818 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:18 ntpd[8717]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:17.818 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:18 ntpd[8717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:17.818 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:18 ntpd[8717]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:17.819 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:18 ntpd[8717]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-09-09T22:23:17.819 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:18 ntpd[8717]: can't open /var/log/ntpstats/peerstats.20220909: Permission denied 2022-09-09T22:23:18.808 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:18 ntpd[8463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:18.809 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:18 ntpd[8463]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:18.809 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:18 ntpd[8463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:18.811 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:18 ntpd[8463]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:18.812 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:18 ntpd[8463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:18.812 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:18 ntpd[8463]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:18.813 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:18 ntpd[8463]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-09-09T22:23:18.813 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:18 ntpd[8463]: can't open /var/log/ntpstats/peerstats.20220909: Permission denied 2022-09-09T22:23:18.822 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:19 ntpd[8717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:18.822 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:19 ntpd[8717]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:18.823 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:19 ntpd[8717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:18.823 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:19 ntpd[8717]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:18.824 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:19 ntpd[8717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:18.824 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:19 ntpd[8717]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:18.825 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:19 ntpd[8717]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-09-09T22:23:18.825 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:19 ntpd[8717]: can't open /var/log/ntpstats/peerstats.20220909: Permission denied 2022-09-09T22:23:18.825 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:19 ntpd[8717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:18.826 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:19 ntpd[8717]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:18.826 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:19 ntpd[8717]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-09-09T22:23:18.827 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:19 ntpd[8717]: can't open /var/log/ntpstats/peerstats.20220909: Permission denied 2022-09-09T22:23:19.807 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:19 ntpd[8463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:19.808 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:19 ntpd[8463]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:19.808 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:19 ntpd[8463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:19.809 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:19 ntpd[8463]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:19.809 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:19 ntpd[8463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:19.809 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:19 ntpd[8463]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:19.810 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:19 ntpd[8463]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-09-09T22:23:19.810 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:19 ntpd[8463]: can't open /var/log/ntpstats/peerstats.20220909: Permission denied 2022-09-09T22:23:19.810 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:19 ntpd[8463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:19.811 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:19 ntpd[8463]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:19.811 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:19 ntpd[8463]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-09-09T22:23:19.811 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:19 ntpd[8463]: can't open /var/log/ntpstats/peerstats.20220909: Permission denied 2022-09-09T22:23:19.819 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:20 ntpd[8717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:19.820 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:20 ntpd[8717]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:19.820 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:20 ntpd[8717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:19.820 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:20 ntpd[8717]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:19.820 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:20 ntpd[8717]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-09-09T22:23:19.821 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:20 ntpd[8717]: can't open /var/log/ntpstats/peerstats.20220909: Permission denied 2022-09-09T22:23:20.806 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:20 ntpd[8463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:20.807 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:20 ntpd[8463]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:20.807 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:20 ntpd[8463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:20.807 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:20 ntpd[8463]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:20.808 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:20 ntpd[8463]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-09-09T22:23:20.808 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:20 ntpd[8463]: can't open /var/log/ntpstats/peerstats.20220909: Permission denied 2022-09-09T22:23:20.819 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:21 ntpd[8717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:20.820 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:21 ntpd[8717]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:20.821 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:21 ntpd[8717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:20.821 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:21 ntpd[8717]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:20.822 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:21 ntpd[8717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:20.822 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:21 ntpd[8717]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:20.823 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:21 ntpd[8717]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-09-09T22:23:20.823 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:21 ntpd[8717]: can't open /var/log/ntpstats/peerstats.20220909: Permission denied 2022-09-09T22:23:20.824 INFO:teuthology.orchestra.run.smithi080.stdout: 9 Sep 22:23:20 ntpd[8717]: ntpd: time set -0.409577 s 2022-09-09T22:23:20.824 INFO:teuthology.orchestra.run.smithi080.stdout:ntpd: time set -0.409577s 2022-09-09T22:23:20.825 INFO:teuthology.orchestra.run.smithi080.stderr: 9 Sep 22:23:20 ntpd[8717]: can't open /var/log/ntpstats/loopstats.20220909: Permission denied 2022-09-09T22:23:20.899 INFO:teuthology.orchestra.run.smithi080.stdout: remote refid st t when poll reach delay offset jitter 2022-09-09T22:23:20.900 INFO:teuthology.orchestra.run.smithi080.stdout:============================================================================== 2022-09-09T22:23:20.900 INFO:teuthology.orchestra.run.smithi080.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-09-09T22:23:20.900 INFO:teuthology.orchestra.run.smithi080.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-09-09T22:23:20.900 INFO:teuthology.orchestra.run.smithi080.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-09-09T22:23:20.901 INFO:teuthology.orchestra.run.smithi080.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-09-09T22:23:21.807 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:21 ntpd[8463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:21.808 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:21 ntpd[8463]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:21.808 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:21 ntpd[8463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:21.808 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:21 ntpd[8463]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:21.808 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:21 ntpd[8463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-09-09T22:23:21.809 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:21 ntpd[8463]: can't open /var/log/ntpstats/rawstats.20220909: Permission denied 2022-09-09T22:23:21.809 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:21 ntpd[8463]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-09-09T22:23:21.809 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:21 ntpd[8463]: can't open /var/log/ntpstats/peerstats.20220909: Permission denied 2022-09-09T22:23:21.810 INFO:teuthology.orchestra.run.smithi005.stdout: 9 Sep 22:23:21 ntpd[8463]: ntpd: time slew +0.000246 s 2022-09-09T22:23:21.810 INFO:teuthology.orchestra.run.smithi005.stdout:ntpd: time slew +0.000246s 2022-09-09T22:23:21.811 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Sep 22:23:21 ntpd[8463]: can't open /var/log/ntpstats/loopstats.20220909: Permission denied 2022-09-09T22:23:21.862 INFO:teuthology.orchestra.run.smithi005.stdout: remote refid st t when poll reach delay offset jitter 2022-09-09T22:23:21.863 INFO:teuthology.orchestra.run.smithi005.stdout:============================================================================== 2022-09-09T22:23:21.863 INFO:teuthology.orchestra.run.smithi005.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-09-09T22:23:21.863 INFO:teuthology.orchestra.run.smithi005.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-09-09T22:23:21.863 INFO:teuthology.orchestra.run.smithi005.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-09-09T22:23:21.864 INFO:teuthology.orchestra.run.smithi005.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-09-09T22:23:21.865 INFO:teuthology.run_tasks:Running task install... 2022-09-09T22:23:21.875 DEBUG:teuthology.task.install:project ceph 2022-09-09T22:23:21.875 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '28758235ba2ab1c1d8afa0f88d00f07b5687a51e'}} 2022-09-09T22:23:21.876 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '28758235ba2ab1c1d8afa0f88d00f07b5687a51e'} 2022-09-09T22:23:21.876 INFO:teuthology.task.install:Using flavor: default 2022-09-09T22:23:21.883 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-09-09T22:23:21.884 INFO:teuthology.task.install:extra packages: [] 2022-09-09T22:23:21.884 DEBUG:teuthology.orchestra.run.smithi005:> sudo lsb_release -is 2022-09-09T22:23:21.960 INFO:teuthology.orchestra.run.smithi005.stdout:Ubuntu 2022-09-09T22:23:21.966 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-09-09T22:23:21.966 DEBUG:teuthology.orchestra.run.smithi080:> sudo lsb_release -is 2022-09-09T22:23:21.967 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-key list | grep Ceph 2022-09-09T22:23:22.017 INFO:teuthology.orchestra.run.smithi080.stdout:Ubuntu 2022-09-09T22:23:22.024 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-09-09T22:23:22.025 DEBUG:teuthology.orchestra.run.smithi080:> sudo apt-key list | grep Ceph 2022-09-09T22:23:22.067 INFO:teuthology.orchestra.run.smithi005.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-09-09T22:23:22.132 INFO:teuthology.orchestra.run.smithi080.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-09-09T22:23:22.134 INFO:teuthology.orchestra.run.smithi005.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-09-09T22:23:22.135 INFO:teuthology.orchestra.run.smithi005.stdout:uid [ unknown] Ceph.com (release key) 2022-09-09T22:23:22.138 DEBUG:teuthology.orchestra.run.smithi005:> sudo lsb_release -is 2022-09-09T22:23:22.190 INFO:teuthology.orchestra.run.smithi080.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-09-09T22:23:22.190 INFO:teuthology.orchestra.run.smithi080.stdout:uid [ unknown] Ceph.com (release key) 2022-09-09T22:23:22.191 DEBUG:teuthology.orchestra.run.smithi080:> sudo lsb_release -is 2022-09-09T22:23:22.192 INFO:teuthology.orchestra.run.smithi005.stdout:Ubuntu 2022-09-09T22:23:22.202 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-09-09T22:23:22.203 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2022-09-09T22:23:22.204 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=28758235ba2ab1c1d8afa0f88d00f07b5687a51e 2022-09-09T22:23:22.247 INFO:teuthology.orchestra.run.smithi080.stdout:Ubuntu 2022-09-09T22:23:22.249 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-09-09T22:23:22.250 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2022-09-09T22:23:22.250 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=28758235ba2ab1c1d8afa0f88d00f07b5687a51e 2022-09-09T22:23:22.373 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default/ 2022-09-09T22:23:22.374 INFO:teuthology.task.install.deb:Package version is 17.0.0-14894-g28758235-1focal 2022-09-09T22:23:22.416 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default/ 2022-09-09T22:23:22.417 INFO:teuthology.task.install.deb:Package version is 17.0.0-14894-g28758235-1focal 2022-09-09T22:23:22.500 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-09-09T22:23:22.501 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-09-09T22:23:22.518 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-get update 2022-09-09T22:23:22.536 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2022-09-09T22:23:22.537 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-09-09T22:23:22.554 DEBUG:teuthology.orchestra.run.smithi080:> sudo apt-get update 2022-09-09T22:23:22.660 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-09-09T22:23:22.665 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-09-09T22:23:22.682 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-09-09T22:23:22.702 INFO:teuthology.orchestra.run.smithi080.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-09-09T22:23:22.703 INFO:teuthology.orchestra.run.smithi080.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-09-09T22:23:22.707 INFO:teuthology.orchestra.run.smithi080.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-09-09T22:23:22.722 INFO:teuthology.orchestra.run.smithi005.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal InRelease 2022-09-09T22:23:22.724 INFO:teuthology.orchestra.run.smithi080.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-09-09T22:23:22.746 INFO:teuthology.orchestra.run.smithi005.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal Release [7,662 B] 2022-09-09T22:23:22.766 INFO:teuthology.orchestra.run.smithi080.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal InRelease 2022-09-09T22:23:22.770 INFO:teuthology.orchestra.run.smithi005.stdout:Ign:6 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal Release.gpg 2022-09-09T22:23:22.775 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:7 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-09-09T22:23:22.789 INFO:teuthology.orchestra.run.smithi080.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal Release [7,662 B] 2022-09-09T22:23:22.794 INFO:teuthology.orchestra.run.smithi005.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main i386 Packages [3,449 B] 2022-09-09T22:23:22.813 INFO:teuthology.orchestra.run.smithi080.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal Release.gpg 2022-09-09T22:23:22.827 INFO:teuthology.orchestra.run.smithi005.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2022-09-09T22:23:22.839 INFO:teuthology.orchestra.run.smithi080.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main i386 Packages [3,449 B] 2022-09-09T22:23:22.882 INFO:teuthology.orchestra.run.smithi080.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2022-09-09T22:23:23.345 INFO:teuthology.orchestra.run.smithi005.stdout:Fetched 28.5 kB in 1s (43.5 kB/s) 2022-09-09T22:23:23.360 INFO:teuthology.orchestra.run.smithi080.stdout:Fetched 28.5 kB in 1s (44.4 kB/s) 2022-09-09T22:23:24.249 INFO:teuthology.orchestra.run.smithi080.stdout:Reading package lists... 2022-09-09T22:23:24.271 DEBUG:teuthology.orchestra.run.smithi080:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-14894-g28758235-1focal cephadm=17.0.0-14894-g28758235-1focal ceph-mds=17.0.0-14894-g28758235-1focal ceph-mgr=17.0.0-14894-g28758235-1focal ceph-common=17.0.0-14894-g28758235-1focal ceph-fuse=17.0.0-14894-g28758235-1focal ceph-test=17.0.0-14894-g28758235-1focal ceph-volume=17.0.0-14894-g28758235-1focal radosgw=17.0.0-14894-g28758235-1focal python3-rados=17.0.0-14894-g28758235-1focal python3-rgw=17.0.0-14894-g28758235-1focal python3-cephfs=17.0.0-14894-g28758235-1focal python3-rbd=17.0.0-14894-g28758235-1focal libcephfs2=17.0.0-14894-g28758235-1focal libcephfs-dev=17.0.0-14894-g28758235-1focal librados2=17.0.0-14894-g28758235-1focal librbd1=17.0.0-14894-g28758235-1focal rbd-fuse=17.0.0-14894-g28758235-1focal 2022-09-09T22:23:24.273 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2022-09-09T22:23:24.299 DEBUG:teuthology.orchestra.run.smithi005:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-14894-g28758235-1focal cephadm=17.0.0-14894-g28758235-1focal ceph-mds=17.0.0-14894-g28758235-1focal ceph-mgr=17.0.0-14894-g28758235-1focal ceph-common=17.0.0-14894-g28758235-1focal ceph-fuse=17.0.0-14894-g28758235-1focal ceph-test=17.0.0-14894-g28758235-1focal ceph-volume=17.0.0-14894-g28758235-1focal radosgw=17.0.0-14894-g28758235-1focal python3-rados=17.0.0-14894-g28758235-1focal python3-rgw=17.0.0-14894-g28758235-1focal python3-cephfs=17.0.0-14894-g28758235-1focal python3-rbd=17.0.0-14894-g28758235-1focal libcephfs2=17.0.0-14894-g28758235-1focal libcephfs-dev=17.0.0-14894-g28758235-1focal librados2=17.0.0-14894-g28758235-1focal librbd1=17.0.0-14894-g28758235-1focal rbd-fuse=17.0.0-14894-g28758235-1focal 2022-09-09T22:23:24.343 INFO:teuthology.orchestra.run.smithi080.stdout:Reading package lists... 2022-09-09T22:23:24.366 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2022-09-09T22:23:24.521 INFO:teuthology.orchestra.run.smithi080.stdout:Building dependency tree... 2022-09-09T22:23:24.522 INFO:teuthology.orchestra.run.smithi080.stdout:Reading state information... 2022-09-09T22:23:24.548 INFO:teuthology.orchestra.run.smithi005.stdout:Building dependency tree... 2022-09-09T22:23:24.549 INFO:teuthology.orchestra.run.smithi005.stdout:Reading state information... 2022-09-09T22:23:24.672 INFO:teuthology.orchestra.run.smithi080.stdout:The following packages were automatically installed and are no longer required: 2022-09-09T22:23:24.673 INFO:teuthology.orchestra.run.smithi080.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-09-09T22:23:24.673 INFO:teuthology.orchestra.run.smithi080.stdout:Use 'sudo apt autoremove' to remove them. 2022-09-09T22:23:24.673 INFO:teuthology.orchestra.run.smithi080.stdout:The following additional packages will be installed: 2022-09-09T22:23:24.674 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2022-09-09T22:23:24.674 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2022-09-09T22:23:24.674 INFO:teuthology.orchestra.run.smithi080.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2022-09-09T22:23:24.675 INFO:teuthology.orchestra.run.smithi080.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2022-09-09T22:23:24.675 INFO:teuthology.orchestra.run.smithi080.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2022-09-09T22:23:24.675 INFO:teuthology.orchestra.run.smithi080.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 libxslt1.1 nvme-cli 2022-09-09T22:23:24.675 INFO:teuthology.orchestra.run.smithi080.stdout: python-asyncssh-doc python-natsort-doc python-pastedeploy-tpl 2022-09-09T22:23:24.676 INFO:teuthology.orchestra.run.smithi080.stdout: python3-asyncssh python3-atomicwrites python3-attr python3-bcrypt 2022-09-09T22:23:24.676 INFO:teuthology.orchestra.run.smithi080.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2022-09-09T22:23:24.676 INFO:teuthology.orchestra.run.smithi080.stdout: python3-ceph-common python3-cherrypy3 python3-cryptography python3-decorator 2022-09-09T22:23:24.676 INFO:teuthology.orchestra.run.smithi080.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2022-09-09T22:23:24.677 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2022-09-09T22:23:24.677 INFO:teuthology.orchestra.run.smithi080.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2022-09-09T22:23:24.677 INFO:teuthology.orchestra.run.smithi080.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2022-09-09T22:23:24.677 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2022-09-09T22:23:24.678 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2022-09-09T22:23:24.678 INFO:teuthology.orchestra.run.smithi080.stdout: python3-repoze.lru python3-requests-oauthlib python3-routes python3-rsa 2022-09-09T22:23:24.678 INFO:teuthology.orchestra.run.smithi080.stdout: python3-scipy python3-simplegeneric python3-simplejson 2022-09-09T22:23:24.678 INFO:teuthology.orchestra.run.smithi080.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2022-09-09T22:23:24.679 INFO:teuthology.orchestra.run.smithi080.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2022-09-09T22:23:24.679 INFO:teuthology.orchestra.run.smithi080.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2022-09-09T22:23:24.679 INFO:teuthology.orchestra.run.smithi080.stdout: qttranslations5-l10n socat xmlstarlet 2022-09-09T22:23:24.680 INFO:teuthology.orchestra.run.smithi080.stdout:Suggested packages: 2022-09-09T22:23:24.680 INFO:teuthology.orchestra.run.smithi080.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2022-09-09T22:23:24.681 INFO:teuthology.orchestra.run.smithi080.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2022-09-09T22:23:24.681 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2022-09-09T22:23:24.681 INFO:teuthology.orchestra.run.smithi080.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2022-09-09T22:23:24.682 INFO:teuthology.orchestra.run.smithi080.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2022-09-09T22:23:24.682 INFO:teuthology.orchestra.run.smithi080.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2022-09-09T22:23:24.682 INFO:teuthology.orchestra.run.smithi080.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2022-09-09T22:23:24.683 INFO:teuthology.orchestra.run.smithi080.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2022-09-09T22:23:24.683 INFO:teuthology.orchestra.run.smithi080.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2022-09-09T22:23:24.684 INFO:teuthology.orchestra.run.smithi080.stdout:Recommended packages: 2022-09-09T22:23:24.684 INFO:teuthology.orchestra.run.smithi080.stdout: btrfs-tools 2022-09-09T22:23:24.714 INFO:teuthology.orchestra.run.smithi005.stdout:The following packages were automatically installed and are no longer required: 2022-09-09T22:23:24.716 INFO:teuthology.orchestra.run.smithi005.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-09-09T22:23:24.716 INFO:teuthology.orchestra.run.smithi005.stdout:Use 'sudo apt autoremove' to remove them. 2022-09-09T22:23:24.718 INFO:teuthology.orchestra.run.smithi005.stdout:The following additional packages will be installed: 2022-09-09T22:23:24.719 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2022-09-09T22:23:24.720 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2022-09-09T22:23:24.720 INFO:teuthology.orchestra.run.smithi005.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2022-09-09T22:23:24.721 INFO:teuthology.orchestra.run.smithi005.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2022-09-09T22:23:24.722 INFO:teuthology.orchestra.run.smithi005.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2022-09-09T22:23:24.722 INFO:teuthology.orchestra.run.smithi005.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 libxslt1.1 nvme-cli 2022-09-09T22:23:24.723 INFO:teuthology.orchestra.run.smithi005.stdout: python-asyncssh-doc python-natsort-doc python-pastedeploy-tpl 2022-09-09T22:23:24.724 INFO:teuthology.orchestra.run.smithi005.stdout: python3-asyncssh python3-atomicwrites python3-attr python3-bcrypt 2022-09-09T22:23:24.724 INFO:teuthology.orchestra.run.smithi005.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2022-09-09T22:23:24.724 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-common python3-cherrypy3 python3-cryptography python3-decorator 2022-09-09T22:23:24.724 INFO:teuthology.orchestra.run.smithi005.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2022-09-09T22:23:24.725 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2022-09-09T22:23:24.725 INFO:teuthology.orchestra.run.smithi005.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2022-09-09T22:23:24.725 INFO:teuthology.orchestra.run.smithi005.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2022-09-09T22:23:24.725 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2022-09-09T22:23:24.726 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2022-09-09T22:23:24.726 INFO:teuthology.orchestra.run.smithi005.stdout: python3-repoze.lru python3-requests-oauthlib python3-routes python3-rsa 2022-09-09T22:23:24.726 INFO:teuthology.orchestra.run.smithi005.stdout: python3-scipy python3-simplegeneric python3-simplejson 2022-09-09T22:23:24.727 INFO:teuthology.orchestra.run.smithi005.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2022-09-09T22:23:24.727 INFO:teuthology.orchestra.run.smithi005.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2022-09-09T22:23:24.727 INFO:teuthology.orchestra.run.smithi005.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2022-09-09T22:23:24.728 INFO:teuthology.orchestra.run.smithi005.stdout: qttranslations5-l10n socat xmlstarlet 2022-09-09T22:23:24.728 INFO:teuthology.orchestra.run.smithi005.stdout:Suggested packages: 2022-09-09T22:23:24.729 INFO:teuthology.orchestra.run.smithi005.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2022-09-09T22:23:24.729 INFO:teuthology.orchestra.run.smithi005.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2022-09-09T22:23:24.729 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2022-09-09T22:23:24.730 INFO:teuthology.orchestra.run.smithi005.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2022-09-09T22:23:24.730 INFO:teuthology.orchestra.run.smithi005.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2022-09-09T22:23:24.731 INFO:teuthology.orchestra.run.smithi005.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2022-09-09T22:23:24.731 INFO:teuthology.orchestra.run.smithi005.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2022-09-09T22:23:24.732 INFO:teuthology.orchestra.run.smithi005.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2022-09-09T22:23:24.732 INFO:teuthology.orchestra.run.smithi005.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2022-09-09T22:23:24.734 INFO:teuthology.orchestra.run.smithi005.stdout:Recommended packages: 2022-09-09T22:23:24.735 INFO:teuthology.orchestra.run.smithi005.stdout: btrfs-tools 2022-09-09T22:23:24.792 INFO:teuthology.orchestra.run.smithi080.stdout:The following NEW packages will be installed: 2022-09-09T22:23:24.793 INFO:teuthology.orchestra.run.smithi080.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2022-09-09T22:23:24.793 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2022-09-09T22:23:24.793 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2022-09-09T22:23:24.793 INFO:teuthology.orchestra.run.smithi080.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2022-09-09T22:23:24.794 INFO:teuthology.orchestra.run.smithi080.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2022-09-09T22:23:24.794 INFO:teuthology.orchestra.run.smithi080.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2022-09-09T22:23:24.794 INFO:teuthology.orchestra.run.smithi080.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2022-09-09T22:23:24.794 INFO:teuthology.orchestra.run.smithi080.stdout: libsqlite3-mod-ceph libthrift-0.13.0 libxslt1.1 nvme-cli python-asyncssh-doc 2022-09-09T22:23:24.795 INFO:teuthology.orchestra.run.smithi080.stdout: python-natsort-doc python-pastedeploy-tpl python3-asyncssh 2022-09-09T22:23:24.795 INFO:teuthology.orchestra.run.smithi080.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-09-09T22:23:24.795 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2022-09-09T22:23:24.795 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2022-09-09T22:23:24.795 INFO:teuthology.orchestra.run.smithi080.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2022-09-09T22:23:24.796 INFO:teuthology.orchestra.run.smithi080.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2022-09-09T22:23:24.796 INFO:teuthology.orchestra.run.smithi080.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2022-09-09T22:23:24.796 INFO:teuthology.orchestra.run.smithi080.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2022-09-09T22:23:24.796 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2022-09-09T22:23:24.797 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2022-09-09T22:23:24.797 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2022-09-09T22:23:24.797 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-09-09T22:23:24.797 INFO:teuthology.orchestra.run.smithi080.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-09-09T22:23:24.798 INFO:teuthology.orchestra.run.smithi080.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-09-09T22:23:24.798 INFO:teuthology.orchestra.run.smithi080.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-09-09T22:23:24.799 INFO:teuthology.orchestra.run.smithi080.stdout: python3-webtest python3-werkzeug qttranslations5-l10n radosgw rbd-fuse socat 2022-09-09T22:23:24.799 INFO:teuthology.orchestra.run.smithi080.stdout: xmlstarlet 2022-09-09T22:23:24.799 INFO:teuthology.orchestra.run.smithi080.stdout:The following packages will be upgraded: 2022-09-09T22:23:24.800 INFO:teuthology.orchestra.run.smithi080.stdout: librados2 librbd1 2022-09-09T22:23:24.837 INFO:teuthology.orchestra.run.smithi005.stdout:The following NEW packages will be installed: 2022-09-09T22:23:24.837 INFO:teuthology.orchestra.run.smithi005.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2022-09-09T22:23:24.838 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2022-09-09T22:23:24.838 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2022-09-09T22:23:24.838 INFO:teuthology.orchestra.run.smithi005.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2022-09-09T22:23:24.839 INFO:teuthology.orchestra.run.smithi005.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2022-09-09T22:23:24.839 INFO:teuthology.orchestra.run.smithi005.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2022-09-09T22:23:24.840 INFO:teuthology.orchestra.run.smithi005.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2022-09-09T22:23:24.842 INFO:teuthology.orchestra.run.smithi005.stdout: libsqlite3-mod-ceph libthrift-0.13.0 libxslt1.1 nvme-cli python-asyncssh-doc 2022-09-09T22:23:24.842 INFO:teuthology.orchestra.run.smithi005.stdout: python-natsort-doc python-pastedeploy-tpl python3-asyncssh 2022-09-09T22:23:24.842 INFO:teuthology.orchestra.run.smithi005.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-09-09T22:23:24.842 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2022-09-09T22:23:24.843 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2022-09-09T22:23:24.843 INFO:teuthology.orchestra.run.smithi005.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2022-09-09T22:23:24.843 INFO:teuthology.orchestra.run.smithi005.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2022-09-09T22:23:24.843 INFO:teuthology.orchestra.run.smithi005.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2022-09-09T22:23:24.844 INFO:teuthology.orchestra.run.smithi005.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2022-09-09T22:23:24.844 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2022-09-09T22:23:24.844 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2022-09-09T22:23:24.844 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2022-09-09T22:23:24.844 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-09-09T22:23:24.845 INFO:teuthology.orchestra.run.smithi005.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-09-09T22:23:24.845 INFO:teuthology.orchestra.run.smithi005.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-09-09T22:23:24.845 INFO:teuthology.orchestra.run.smithi005.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-09-09T22:23:24.845 INFO:teuthology.orchestra.run.smithi005.stdout: python3-webtest python3-werkzeug qttranslations5-l10n radosgw rbd-fuse socat 2022-09-09T22:23:24.846 INFO:teuthology.orchestra.run.smithi005.stdout: xmlstarlet 2022-09-09T22:23:24.846 INFO:teuthology.orchestra.run.smithi005.stdout:The following packages will be upgraded: 2022-09-09T22:23:24.846 INFO:teuthology.orchestra.run.smithi005.stdout: librados2 librbd1 2022-09-09T22:23:24.854 INFO:teuthology.orchestra.run.smithi080.stdout:2 upgraded, 111 newly installed, 0 to remove and 0 not upgraded. 2022-09-09T22:23:24.854 INFO:teuthology.orchestra.run.smithi080.stdout:Need to get 221 MB of archives. 2022-09-09T22:23:24.855 INFO:teuthology.orchestra.run.smithi080.stdout:After this operation, 1,068 MB of additional disk space will be used. 2022-09-09T22:23:24.855 INFO:teuthology.orchestra.run.smithi080.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-09-09T22:23:24.898 INFO:teuthology.orchestra.run.smithi005.stdout:2 upgraded, 111 newly installed, 0 to remove and 0 not upgraded. 2022-09-09T22:23:24.899 INFO:teuthology.orchestra.run.smithi005.stdout:Need to get 221 MB of archives. 2022-09-09T22:23:24.899 INFO:teuthology.orchestra.run.smithi005.stdout:After this operation, 1,068 MB of additional disk space will be used. 2022-09-09T22:23:24.900 INFO:teuthology.orchestra.run.smithi005.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-09-09T22:23:24.940 INFO:teuthology.orchestra.run.smithi080.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2022-09-09T22:23:24.943 INFO:teuthology.orchestra.run.smithi080.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libpcre2-16-0 amd64 10.34-7 [181 kB] 2022-09-09T22:23:24.949 INFO:teuthology.orchestra.run.smithi080.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.0.0-14894-g28758235-1focal [3,445 kB] 2022-09-09T22:23:24.960 INFO:teuthology.orchestra.run.smithi080.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2022-09-09T22:23:24.981 INFO:teuthology.orchestra.run.smithi005.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2022-09-09T22:23:24.983 INFO:teuthology.orchestra.run.smithi005.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libpcre2-16-0 amd64 10.34-7 [181 kB] 2022-09-09T22:23:24.994 INFO:teuthology.orchestra.run.smithi005.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.0.0-14894-g28758235-1focal [3,445 kB] 2022-09-09T22:23:25.000 INFO:teuthology.orchestra.run.smithi005.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2022-09-09T22:23:25.012 INFO:teuthology.orchestra.run.smithi080.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2022-09-09T22:23:25.017 INFO:teuthology.orchestra.run.smithi080.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2022-09-09T22:23:25.023 INFO:teuthology.orchestra.run.smithi080.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2022-09-09T22:23:25.025 INFO:teuthology.orchestra.run.smithi080.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-09-09T22:23:25.025 INFO:teuthology.orchestra.run.smithi080.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2022-09-09T22:23:25.026 INFO:teuthology.orchestra.run.smithi080.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-09-09T22:23:25.026 INFO:teuthology.orchestra.run.smithi080.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-09-09T22:23:25.029 INFO:teuthology.orchestra.run.smithi080.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-09-09T22:23:25.030 INFO:teuthology.orchestra.run.smithi080.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-09-09T22:23:25.031 INFO:teuthology.orchestra.run.smithi080.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2022-09-09T22:23:25.048 INFO:teuthology.orchestra.run.smithi080.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-09-09T22:23:25.048 INFO:teuthology.orchestra.run.smithi080.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-09-09T22:23:25.049 INFO:teuthology.orchestra.run.smithi080.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-09-09T22:23:25.050 INFO:teuthology.orchestra.run.smithi080.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-09-09T22:23:25.050 INFO:teuthology.orchestra.run.smithi080.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-09-09T22:23:25.050 INFO:teuthology.orchestra.run.smithi080.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-09-09T22:23:25.052 INFO:teuthology.orchestra.run.smithi005.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2022-09-09T22:23:25.055 INFO:teuthology.orchestra.run.smithi005.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2022-09-09T22:23:25.064 INFO:teuthology.orchestra.run.smithi005.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2022-09-09T22:23:25.066 INFO:teuthology.orchestra.run.smithi005.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-09-09T22:23:25.067 INFO:teuthology.orchestra.run.smithi080.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-09-09T22:23:25.067 INFO:teuthology.orchestra.run.smithi005.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2022-09-09T22:23:25.068 INFO:teuthology.orchestra.run.smithi080.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-09-09T22:23:25.069 INFO:teuthology.orchestra.run.smithi005.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-09-09T22:23:25.069 INFO:teuthology.orchestra.run.smithi005.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-09-09T22:23:25.069 INFO:teuthology.orchestra.run.smithi080.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2022-09-09T22:23:25.071 INFO:teuthology.orchestra.run.smithi005.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-09-09T22:23:25.071 INFO:teuthology.orchestra.run.smithi080.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-09-09T22:23:25.072 INFO:teuthology.orchestra.run.smithi005.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-09-09T22:23:25.073 INFO:teuthology.orchestra.run.smithi005.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2022-09-09T22:23:25.085 INFO:teuthology.orchestra.run.smithi080.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-09-09T22:23:25.089 INFO:teuthology.orchestra.run.smithi005.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-09-09T22:23:25.089 INFO:teuthology.orchestra.run.smithi005.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-09-09T22:23:25.091 INFO:teuthology.orchestra.run.smithi005.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-09-09T22:23:25.092 INFO:teuthology.orchestra.run.smithi005.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-09-09T22:23:25.094 INFO:teuthology.orchestra.run.smithi005.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-09-09T22:23:25.094 INFO:teuthology.orchestra.run.smithi005.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-09-09T22:23:25.095 INFO:teuthology.orchestra.run.smithi080.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-09-09T22:23:25.096 INFO:teuthology.orchestra.run.smithi080.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-09-09T22:23:25.096 INFO:teuthology.orchestra.run.smithi080.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-09-09T22:23:25.096 INFO:teuthology.orchestra.run.smithi080.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-09-09T22:23:25.096 INFO:teuthology.orchestra.run.smithi080.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-09-09T22:23:25.102 INFO:teuthology.orchestra.run.smithi080.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-09-09T22:23:25.105 INFO:teuthology.orchestra.run.smithi080.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-09-09T22:23:25.106 INFO:teuthology.orchestra.run.smithi080.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-asyncssh all 1.12.2-1 [152 kB] 2022-09-09T22:23:25.107 INFO:teuthology.orchestra.run.smithi005.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-09-09T22:23:25.107 INFO:teuthology.orchestra.run.smithi005.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-09-09T22:23:25.108 INFO:teuthology.orchestra.run.smithi080.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2022-09-09T22:23:25.108 INFO:teuthology.orchestra.run.smithi005.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2022-09-09T22:23:25.110 INFO:teuthology.orchestra.run.smithi005.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-09-09T22:23:25.121 INFO:teuthology.orchestra.run.smithi080.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-09-09T22:23:25.122 INFO:teuthology.orchestra.run.smithi080.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2022-09-09T22:23:25.122 INFO:teuthology.orchestra.run.smithi080.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-09-09T22:23:25.123 INFO:teuthology.orchestra.run.smithi080.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-09-09T22:23:25.123 INFO:teuthology.orchestra.run.smithi080.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-09-09T22:23:25.125 INFO:teuthology.orchestra.run.smithi005.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-09-09T22:23:25.126 INFO:teuthology.orchestra.run.smithi080.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-09-09T22:23:25.128 INFO:teuthology.orchestra.run.smithi005.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-09-09T22:23:25.129 INFO:teuthology.orchestra.run.smithi005.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-09-09T22:23:25.129 INFO:teuthology.orchestra.run.smithi005.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-09-09T22:23:25.130 INFO:teuthology.orchestra.run.smithi005.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-09-09T22:23:25.131 INFO:teuthology.orchestra.run.smithi005.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-09-09T22:23:25.137 INFO:teuthology.orchestra.run.smithi080.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-09-09T22:23:25.137 INFO:teuthology.orchestra.run.smithi080.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-09-09T22:23:25.138 INFO:teuthology.orchestra.run.smithi080.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-09-09T22:23:25.139 INFO:teuthology.orchestra.run.smithi005.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-09-09T22:23:25.143 INFO:teuthology.orchestra.run.smithi005.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-09-09T22:23:25.144 INFO:teuthology.orchestra.run.smithi080.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-09-09T22:23:25.144 INFO:teuthology.orchestra.run.smithi005.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-asyncssh all 1.12.2-1 [152 kB] 2022-09-09T22:23:25.145 INFO:teuthology.orchestra.run.smithi005.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2022-09-09T22:23:25.153 INFO:teuthology.orchestra.run.smithi080.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-09-09T22:23:25.154 INFO:teuthology.orchestra.run.smithi080.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-09-09T22:23:25.155 INFO:teuthology.orchestra.run.smithi080.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-09-09T22:23:25.158 INFO:teuthology.orchestra.run.smithi080.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-09-09T22:23:25.159 INFO:teuthology.orchestra.run.smithi080.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-09-09T22:23:25.160 INFO:teuthology.orchestra.run.smithi080.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-09-09T22:23:25.161 INFO:teuthology.orchestra.run.smithi005.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-09-09T22:23:25.161 INFO:teuthology.orchestra.run.smithi005.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2022-09-09T22:23:25.162 INFO:teuthology.orchestra.run.smithi005.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-09-09T22:23:25.163 INFO:teuthology.orchestra.run.smithi005.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-09-09T22:23:25.164 INFO:teuthology.orchestra.run.smithi005.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-09-09T22:23:25.164 INFO:teuthology.orchestra.run.smithi005.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-09-09T22:23:25.174 INFO:teuthology.orchestra.run.smithi005.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-09-09T22:23:25.175 INFO:teuthology.orchestra.run.smithi005.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-09-09T22:23:25.176 INFO:teuthology.orchestra.run.smithi005.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-09-09T22:23:25.178 INFO:teuthology.orchestra.run.smithi005.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-09-09T22:23:25.190 INFO:teuthology.orchestra.run.smithi005.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-09-09T22:23:25.193 INFO:teuthology.orchestra.run.smithi005.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-09-09T22:23:25.195 INFO:teuthology.orchestra.run.smithi005.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-09-09T22:23:25.198 INFO:teuthology.orchestra.run.smithi005.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-09-09T22:23:25.199 INFO:teuthology.orchestra.run.smithi005.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-09-09T22:23:25.199 INFO:teuthology.orchestra.run.smithi005.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-09-09T22:23:25.203 INFO:teuthology.orchestra.run.smithi080.stdout:Get:52 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.0.0-14894-g28758235-1focal [3,941 kB] 2022-09-09T22:23:25.245 INFO:teuthology.orchestra.run.smithi080.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-09-09T22:23:25.264 INFO:teuthology.orchestra.run.smithi080.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-09-09T22:23:25.266 INFO:teuthology.orchestra.run.smithi080.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-09-09T22:23:25.273 INFO:teuthology.orchestra.run.smithi005.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-09-09T22:23:25.278 INFO:teuthology.orchestra.run.smithi080.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-09-09T22:23:25.278 INFO:teuthology.orchestra.run.smithi080.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-09-09T22:23:25.279 INFO:teuthology.orchestra.run.smithi080.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-09-09T22:23:25.287 INFO:teuthology.orchestra.run.smithi005.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-09-09T22:23:25.289 INFO:teuthology.orchestra.run.smithi005.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-09-09T22:23:25.300 INFO:teuthology.orchestra.run.smithi005.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-09-09T22:23:25.329 INFO:teuthology.orchestra.run.smithi080.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-09-09T22:23:25.330 INFO:teuthology.orchestra.run.smithi080.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-09-09T22:23:25.331 INFO:teuthology.orchestra.run.smithi080.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-09-09T22:23:25.332 INFO:teuthology.orchestra.run.smithi080.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-09-09T22:23:25.334 INFO:teuthology.orchestra.run.smithi080.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-09-09T22:23:25.335 INFO:teuthology.orchestra.run.smithi080.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-09-09T22:23:25.336 INFO:teuthology.orchestra.run.smithi080.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-09-09T22:23:25.338 INFO:teuthology.orchestra.run.smithi080.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-09-09T22:23:25.346 INFO:teuthology.orchestra.run.smithi080.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-09-09T22:23:25.356 INFO:teuthology.orchestra.run.smithi005.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-09-09T22:23:25.357 INFO:teuthology.orchestra.run.smithi080.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-09-09T22:23:25.357 INFO:teuthology.orchestra.run.smithi005.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-09-09T22:23:25.359 INFO:teuthology.orchestra.run.smithi080.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-09-09T22:23:25.359 INFO:teuthology.orchestra.run.smithi005.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-09-09T22:23:25.360 INFO:teuthology.orchestra.run.smithi005.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-09-09T22:23:25.361 INFO:teuthology.orchestra.run.smithi005.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-09-09T22:23:25.361 INFO:teuthology.orchestra.run.smithi005.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-09-09T22:23:25.363 INFO:teuthology.orchestra.run.smithi005.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-09-09T22:23:25.364 INFO:teuthology.orchestra.run.smithi080.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libxslt1.1 amd64 1.1.34-4ubuntu0.20.04.1 [151 kB] 2022-09-09T22:23:25.364 INFO:teuthology.orchestra.run.smithi005.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-09-09T22:23:25.367 INFO:teuthology.orchestra.run.smithi080.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-09-09T22:23:25.371 INFO:teuthology.orchestra.run.smithi080.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2022-09-09T22:23:25.372 INFO:teuthology.orchestra.run.smithi080.stdout:Get:73 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.0.0-14894-g28758235-1focal [719 kB] 2022-09-09T22:23:25.373 INFO:teuthology.orchestra.run.smithi005.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-09-09T22:23:25.374 INFO:teuthology.orchestra.run.smithi005.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-09-09T22:23:25.389 INFO:teuthology.orchestra.run.smithi080.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-09-09T22:23:25.390 INFO:teuthology.orchestra.run.smithi005.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-09-09T22:23:25.391 INFO:teuthology.orchestra.run.smithi005.stdout:Get:67 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.0.0-14894-g28758235-1focal [3,941 kB] 2022-09-09T22:23:25.392 INFO:teuthology.orchestra.run.smithi080.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-asyncssh-doc all 1.12.2-1 [258 kB] 2022-09-09T22:23:25.392 INFO:teuthology.orchestra.run.smithi005.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-09-09T22:23:25.393 INFO:teuthology.orchestra.run.smithi005.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-09-09T22:23:25.394 INFO:teuthology.orchestra.run.smithi080.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2022-09-09T22:23:25.397 INFO:teuthology.orchestra.run.smithi005.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libxslt1.1 amd64 1.1.34-4ubuntu0.20.04.1 [151 kB] 2022-09-09T22:23:25.400 INFO:teuthology.orchestra.run.smithi005.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-09-09T22:23:25.404 INFO:teuthology.orchestra.run.smithi005.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2022-09-09T22:23:25.412 INFO:teuthology.orchestra.run.smithi080.stdout:Get:77 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.0.0-14894-g28758235-1focal [376 kB] 2022-09-09T22:23:25.419 INFO:teuthology.orchestra.run.smithi005.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-09-09T22:23:25.421 INFO:teuthology.orchestra.run.smithi005.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-asyncssh-doc all 1.12.2-1 [258 kB] 2022-09-09T22:23:25.423 INFO:teuthology.orchestra.run.smithi005.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2022-09-09T22:23:25.424 INFO:teuthology.orchestra.run.smithi080.stdout:Get:78 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.0.0-14894-g28758235-1focal [22.4 kB] 2022-09-09T22:23:25.427 INFO:teuthology.orchestra.run.smithi080.stdout:Get:79 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.0.0-14894-g28758235-1focal [184 kB] 2022-09-09T22:23:25.434 INFO:teuthology.orchestra.run.smithi080.stdout:Get:80 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.0.0-14894-g28758235-1focal [43.7 kB] 2022-09-09T22:23:25.436 INFO:teuthology.orchestra.run.smithi080.stdout:Get:81 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.0.0-14894-g28758235-1focal [346 kB] 2022-09-09T22:23:25.439 INFO:teuthology.orchestra.run.smithi080.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2022-09-09T22:23:25.439 INFO:teuthology.orchestra.run.smithi080.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-09-09T22:23:25.440 INFO:teuthology.orchestra.run.smithi080.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-09-09T22:23:25.441 INFO:teuthology.orchestra.run.smithi080.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-09-09T22:23:25.453 INFO:teuthology.orchestra.run.smithi080.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-09-09T22:23:25.454 INFO:teuthology.orchestra.run.smithi080.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-09-09T22:23:25.455 INFO:teuthology.orchestra.run.smithi080.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-09-09T22:23:25.455 INFO:teuthology.orchestra.run.smithi080.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-09-09T22:23:25.456 INFO:teuthology.orchestra.run.smithi080.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2022-09-09T22:23:25.463 INFO:teuthology.orchestra.run.smithi005.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2022-09-09T22:23:25.463 INFO:teuthology.orchestra.run.smithi080.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.0.0-14894-g28758235-1focal [6,387 kB] 2022-09-09T22:23:25.464 INFO:teuthology.orchestra.run.smithi005.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-09-09T22:23:25.465 INFO:teuthology.orchestra.run.smithi005.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-09-09T22:23:25.465 INFO:teuthology.orchestra.run.smithi005.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-09-09T22:23:25.477 INFO:teuthology.orchestra.run.smithi005.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-09-09T22:23:25.478 INFO:teuthology.orchestra.run.smithi005.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-09-09T22:23:25.478 INFO:teuthology.orchestra.run.smithi005.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-09-09T22:23:25.479 INFO:teuthology.orchestra.run.smithi005.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-09-09T22:23:25.479 INFO:teuthology.orchestra.run.smithi005.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2022-09-09T22:23:25.766 INFO:teuthology.orchestra.run.smithi005.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.0.0-14894-g28758235-1focal [719 kB] 2022-09-09T22:23:25.797 INFO:teuthology.orchestra.run.smithi080.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.0.0-14894-g28758235-1focal [104 kB] 2022-09-09T22:23:25.797 INFO:teuthology.orchestra.run.smithi080.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.0.0-14894-g28758235-1focal [466 kB] 2022-09-09T22:23:25.812 INFO:teuthology.orchestra.run.smithi080.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.0.0-14894-g28758235-1focal [29.5 MB] 2022-09-09T22:23:25.832 INFO:teuthology.orchestra.run.smithi005.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.0.0-14894-g28758235-1focal [376 kB] 2022-09-09T22:23:25.896 INFO:teuthology.orchestra.run.smithi005.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.0.0-14894-g28758235-1focal [22.4 kB] 2022-09-09T22:23:25.897 INFO:teuthology.orchestra.run.smithi005.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.0.0-14894-g28758235-1focal [184 kB] 2022-09-09T22:23:25.901 INFO:teuthology.orchestra.run.smithi005.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.0.0-14894-g28758235-1focal [43.7 kB] 2022-09-09T22:23:25.902 INFO:teuthology.orchestra.run.smithi005.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.0.0-14894-g28758235-1focal [346 kB] 2022-09-09T22:23:25.912 INFO:teuthology.orchestra.run.smithi005.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.0.0-14894-g28758235-1focal [6,387 kB] 2022-09-09T22:23:26.491 INFO:teuthology.orchestra.run.smithi005.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.0.0-14894-g28758235-1focal [104 kB] 2022-09-09T22:23:26.494 INFO:teuthology.orchestra.run.smithi005.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.0.0-14894-g28758235-1focal [466 kB] 2022-09-09T22:23:26.544 INFO:teuthology.orchestra.run.smithi005.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.0.0-14894-g28758235-1focal [29.5 MB] 2022-09-09T22:23:27.562 INFO:teuthology.orchestra.run.smithi080.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.0.0-14894-g28758235-1focal [10.5 MB] 2022-09-09T22:23:27.927 INFO:teuthology.orchestra.run.smithi080.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.0.0-14894-g28758235-1focal [218 kB] 2022-09-09T22:23:27.932 INFO:teuthology.orchestra.run.smithi080.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.0.0-14894-g28758235-1focal [132 kB] 2022-09-09T22:23:27.934 INFO:teuthology.orchestra.run.smithi080.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.0.0-14894-g28758235-1focal [1,531 kB] 2022-09-09T22:23:27.963 INFO:teuthology.orchestra.run.smithi080.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.0.0-14894-g28758235-1focal [7,819 kB] 2022-09-09T22:23:28.298 INFO:teuthology.orchestra.run.smithi080.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.0.0-14894-g28758235-1focal [28.7 MB] 2022-09-09T22:23:29.697 INFO:teuthology.orchestra.run.smithi080.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.0.0-14894-g28758235-1focal [3,812 B] 2022-09-09T22:23:29.698 INFO:teuthology.orchestra.run.smithi080.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.0.0-14894-g28758235-1focal [828 kB] 2022-09-09T22:23:29.711 INFO:teuthology.orchestra.run.smithi080.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.0.0-14894-g28758235-1focal [2,270 kB] 2022-09-09T22:23:29.727 INFO:teuthology.orchestra.run.smithi080.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.0.0-14894-g28758235-1focal [79.7 kB] 2022-09-09T22:23:29.728 INFO:teuthology.orchestra.run.smithi080.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.0.0-14894-g28758235-1focal [110 kB] 2022-09-09T22:23:29.729 INFO:teuthology.orchestra.run.smithi080.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.0.0-14894-g28758235-1focal [1,708 kB] 2022-09-09T22:23:29.776 INFO:teuthology.orchestra.run.smithi080.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.0.0-14894-g28758235-1focal [7,877 kB] 2022-09-09T22:23:30.107 INFO:teuthology.orchestra.run.smithi080.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.0.0-14894-g28758235-1focal [3,940 B] 2022-09-09T22:23:30.107 INFO:teuthology.orchestra.run.smithi080.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.0.0-14894-g28758235-1focal [59.0 MB] 2022-09-09T22:23:30.142 INFO:teuthology.orchestra.run.smithi005.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.0.0-14894-g28758235-1focal [10.5 MB] 2022-09-09T22:23:31.463 INFO:teuthology.orchestra.run.smithi005.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.0.0-14894-g28758235-1focal [218 kB] 2022-09-09T22:23:31.481 INFO:teuthology.orchestra.run.smithi005.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.0.0-14894-g28758235-1focal [132 kB] 2022-09-09T22:23:31.509 INFO:teuthology.orchestra.run.smithi005.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.0.0-14894-g28758235-1focal [1,531 kB] 2022-09-09T22:23:31.686 INFO:teuthology.orchestra.run.smithi005.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.0.0-14894-g28758235-1focal [7,819 kB] 2022-09-09T22:23:32.678 INFO:teuthology.orchestra.run.smithi005.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.0.0-14894-g28758235-1focal [28.7 MB] 2022-09-09T22:23:33.019 INFO:teuthology.orchestra.run.smithi080.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.0.0-14894-g28758235-1focal [225 kB] 2022-09-09T22:23:33.024 INFO:teuthology.orchestra.run.smithi080.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.0.0-14894-g28758235-1focal [27.4 kB] 2022-09-09T22:23:33.025 INFO:teuthology.orchestra.run.smithi080.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.0.0-14894-g28758235-1focal [18.4 MB] 2022-09-09T22:23:33.742 INFO:teuthology.orchestra.run.smithi080.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.0.0-14894-g28758235-1focal [97.1 kB] 2022-09-09T22:23:34.397 INFO:teuthology.orchestra.run.smithi080.stdout:Fetched 221 MB in 9s (24.6 MB/s) 2022-09-09T22:23:34.566 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-09-09T22:23:34.608 INFO:teuthology.orchestra.run.smithi080.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102374 files and directories currently installed.) 2022-09-09T22:23:34.610 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-09-09T22:23:34.706 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-09-09T22:23:34.831 INFO:teuthology.orchestra.run.smithi005.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.0.0-14894-g28758235-1focal [3,812 B] 2022-09-09T22:23:34.904 INFO:teuthology.orchestra.run.smithi005.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.0.0-14894-g28758235-1focal [828 kB] 2022-09-09T22:23:34.918 INFO:teuthology.orchestra.run.smithi005.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.0.0-14894-g28758235-1focal [2,270 kB] 2022-09-09T22:23:35.013 INFO:teuthology.orchestra.run.smithi005.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.0.0-14894-g28758235-1focal [79.7 kB] 2022-09-09T22:23:35.015 INFO:teuthology.orchestra.run.smithi005.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.0.0-14894-g28758235-1focal [110 kB] 2022-09-09T22:23:35.016 INFO:teuthology.orchestra.run.smithi005.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.0.0-14894-g28758235-1focal [1,708 kB] 2022-09-09T22:23:35.068 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2022-09-09T22:23:35.082 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2022-09-09T22:23:35.118 INFO:teuthology.orchestra.run.smithi005.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.0.0-14894-g28758235-1focal [7,877 kB] 2022-09-09T22:23:35.119 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2022-09-09T22:23:35.421 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2022-09-09T22:23:35.434 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../002-libpcre2-16-0_10.34-7_amd64.deb ... 2022-09-09T22:23:35.471 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7) ... 2022-09-09T22:23:35.530 INFO:teuthology.orchestra.run.smithi005.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.0.0-14894-g28758235-1focal [3,940 B] 2022-09-09T22:23:35.531 INFO:teuthology.orchestra.run.smithi005.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.0.0-14894-g28758235-1focal [59.0 MB] 2022-09-09T22:23:35.933 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package libqt5core5a:amd64. 2022-09-09T22:23:35.946 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../003-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2022-09-09T22:23:35.993 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2022-09-09T22:23:36.494 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2022-09-09T22:23:36.507 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../004-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2022-09-09T22:23:36.545 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2022-09-09T22:23:36.872 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package libqt5network5:amd64. 2022-09-09T22:23:36.886 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../005-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2022-09-09T22:23:36.930 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2022-09-09T22:23:37.300 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package libthrift-0.13.0. 2022-09-09T22:23:37.313 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../006-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2022-09-09T22:23:37.358 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2022-09-09T22:23:37.741 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../007-librbd1_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:23:37.870 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking librbd1 (17.0.0-14894-g28758235-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-09-09T22:23:38.554 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../008-librados2_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:23:38.613 INFO:teuthology.orchestra.run.smithi005.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.0.0-14894-g28758235-1focal [225 kB] 2022-09-09T22:23:38.619 INFO:teuthology.orchestra.run.smithi005.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.0.0-14894-g28758235-1focal [27.4 kB] 2022-09-09T22:23:38.619 INFO:teuthology.orchestra.run.smithi005.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.0.0-14894-g28758235-1focal [18.4 MB] 2022-09-09T22:23:38.683 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking librados2 (17.0.0-14894-g28758235-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-09-09T22:23:39.386 INFO:teuthology.orchestra.run.smithi005.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-librgw-frontends/28758235ba2ab1c1d8afa0f88d00f07b5687a51e/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.0.0-14894-g28758235-1focal [97.1 kB] 2022-09-09T22:23:39.404 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package libcephfs2. 2022-09-09T22:23:39.419 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../009-libcephfs2_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:23:39.461 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking libcephfs2 (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:23:39.813 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-rados. 2022-09-09T22:23:39.826 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../010-python3-rados_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:23:39.863 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-rados (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:23:39.989 INFO:teuthology.orchestra.run.smithi005.stdout:Fetched 221 MB in 15s (15.2 MB/s) 2022-09-09T22:23:40.160 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-09-09T22:23:40.202 INFO:teuthology.orchestra.run.smithi005.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102374 files and directories currently installed.) 2022-09-09T22:23:40.204 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-09-09T22:23:40.216 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-09-09T22:23:40.230 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../011-python3-ceph-argparse_17.0.0-14894-g28758235-1focal_all.deb ... 2022-09-09T22:23:40.274 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-ceph-argparse (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:23:40.308 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-09-09T22:23:40.585 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-cephfs. 2022-09-09T22:23:40.603 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../012-python3-cephfs_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:23:40.626 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-cephfs (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:23:40.648 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2022-09-09T22:23:40.668 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2022-09-09T22:23:40.704 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2022-09-09T22:23:40.946 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-ceph-common. 2022-09-09T22:23:40.958 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../013-python3-ceph-common_17.0.0-14894-g28758235-1focal_all.deb ... 2022-09-09T22:23:40.996 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-ceph-common (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:23:41.075 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2022-09-09T22:23:41.091 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../002-libpcre2-16-0_10.34-7_amd64.deb ... 2022-09-09T22:23:41.125 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7) ... 2022-09-09T22:23:41.323 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-prettytable. 2022-09-09T22:23:41.338 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../014-python3-prettytable_0.7.2-5_all.deb ... 2022-09-09T22:23:41.381 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-09-09T22:23:41.634 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-rbd. 2022-09-09T22:23:41.648 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../015-python3-rbd_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:23:41.684 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-rbd (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:23:41.704 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libqt5core5a:amd64. 2022-09-09T22:23:41.720 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../003-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2022-09-09T22:23:41.781 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2022-09-09T22:23:42.037 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2022-09-09T22:23:42.050 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../016-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2022-09-09T22:23:42.095 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2022-09-09T22:23:42.349 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2022-09-09T22:23:42.363 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../004-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2022-09-09T22:23:42.412 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2022-09-09T22:23:42.473 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-09-09T22:23:42.487 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../017-librabbitmq4_0.10.0-1_amd64.deb ... 2022-09-09T22:23:42.522 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-09-09T22:23:42.785 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libqt5network5:amd64. 2022-09-09T22:23:42.799 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../005-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2022-09-09T22:23:42.843 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2022-09-09T22:23:42.859 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-09-09T22:23:42.873 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../018-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-09-09T22:23:42.917 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-09-09T22:23:43.231 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libthrift-0.13.0. 2022-09-09T22:23:43.244 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../006-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2022-09-09T22:23:43.286 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package librgw2. 2022-09-09T22:23:43.288 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2022-09-09T22:23:43.300 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../019-librgw2_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:23:43.344 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking librgw2 (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:23:43.672 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../007-librbd1_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:23:43.817 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking librbd1 (17.0.0-14894-g28758235-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-09-09T22:23:44.200 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-rgw. 2022-09-09T22:23:44.212 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../020-python3-rgw_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:23:44.247 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-rgw (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:23:44.526 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../008-librados2_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:23:44.612 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package liboath0. 2022-09-09T22:23:44.627 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2022-09-09T22:23:44.646 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking librados2 (17.0.0-14894-g28758235-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-09-09T22:23:44.676 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-09-09T22:23:44.985 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package libradosstriper1. 2022-09-09T22:23:44.998 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../022-libradosstriper1_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:23:45.035 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking libradosstriper1 (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:23:45.425 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libcephfs2. 2022-09-09T22:23:45.465 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../009-libcephfs2_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:23:45.498 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package ceph-common. 2022-09-09T22:23:45.508 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libcephfs2 (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:23:45.510 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../023-ceph-common_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:23:45.557 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking ceph-common (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:23:45.844 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-rados. 2022-09-09T22:23:45.858 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../010-python3-rados_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:23:45.894 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-rados (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:23:46.238 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-09-09T22:23:46.255 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../011-python3-ceph-argparse_17.0.0-14894-g28758235-1focal_all.deb ... 2022-09-09T22:23:46.288 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-ceph-argparse (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:23:46.591 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-cephfs. 2022-09-09T22:23:46.604 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../012-python3-cephfs_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:23:46.666 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-cephfs (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:23:47.019 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-ceph-common. 2022-09-09T22:23:47.032 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../013-python3-ceph-common_17.0.0-14894-g28758235-1focal_all.deb ... 2022-09-09T22:23:47.069 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-ceph-common (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:23:47.405 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-prettytable. 2022-09-09T22:23:47.419 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../014-python3-prettytable_0.7.2-5_all.deb ... 2022-09-09T22:23:47.464 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-09-09T22:23:47.799 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-rbd. 2022-09-09T22:23:47.814 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../015-python3-rbd_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:23:47.849 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-rbd (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:23:48.202 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2022-09-09T22:23:48.216 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../016-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2022-09-09T22:23:48.262 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2022-09-09T22:23:48.594 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package ceph-base. 2022-09-09T22:23:48.607 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../024-ceph-base_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:23:48.653 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking ceph-base (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:23:48.664 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-09-09T22:23:48.680 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../017-librabbitmq4_0.10.0-1_amd64.deb ... 2022-09-09T22:23:48.721 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-09-09T22:23:49.079 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-09-09T22:23:49.087 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../018-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-09-09T22:23:49.116 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-09-09T22:23:49.535 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package librgw2. 2022-09-09T22:23:49.542 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../019-librgw2_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:23:49.579 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking librgw2 (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:23:49.989 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-cherrypy3. 2022-09-09T22:23:50.005 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2022-09-09T22:23:50.046 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-09-09T22:23:50.426 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-natsort. 2022-09-09T22:23:50.439 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../026-python3-natsort_7.0.1-1_all.deb ... 2022-09-09T22:23:50.476 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-natsort (7.0.1-1) ... 2022-09-09T22:23:50.480 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-rgw. 2022-09-09T22:23:50.495 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../020-python3-rgw_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:23:50.543 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-rgw (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:23:50.786 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-logutils. 2022-09-09T22:23:50.801 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2022-09-09T22:23:50.837 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-09-09T22:23:50.917 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package liboath0. 2022-09-09T22:23:50.929 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2022-09-09T22:23:50.990 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-09-09T22:23:51.147 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-markupsafe. 2022-09-09T22:23:51.161 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-09-09T22:23:51.205 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-09-09T22:23:51.318 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libradosstriper1. 2022-09-09T22:23:51.332 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../022-libradosstriper1_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:23:51.376 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libradosstriper1 (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:23:51.516 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-mako. 2022-09-09T22:23:51.530 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-09-09T22:23:51.566 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-09-09T22:23:51.863 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-common. 2022-09-09T22:23:51.878 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../023-ceph-common_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:23:51.921 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-common (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:23:51.929 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-simplegeneric. 2022-09-09T22:23:51.940 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2022-09-09T22:23:51.977 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-09-09T22:23:52.263 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-singledispatch. 2022-09-09T22:23:52.279 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-09-09T22:23:52.321 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-09-09T22:23:52.582 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-webob. 2022-09-09T22:23:52.595 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2_all.deb ... 2022-09-09T22:23:52.624 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-09-09T22:23:52.952 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-soupsieve. 2022-09-09T22:23:52.966 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-09-09T22:23:53.027 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-09-09T22:23:53.321 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-bs4. 2022-09-09T22:23:53.336 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2022-09-09T22:23:53.371 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-09-09T22:23:53.690 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-waitress. 2022-09-09T22:23:53.705 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2022-09-09T22:23:53.751 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2022-09-09T22:23:54.069 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-tempita. 2022-09-09T22:23:54.083 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2022-09-09T22:23:54.118 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-09-09T22:23:54.422 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-paste. 2022-09-09T22:23:54.437 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-09-09T22:23:54.471 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-09-09T22:23:54.816 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-09-09T22:23:54.827 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-base. 2022-09-09T22:23:54.831 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-09-09T22:23:54.840 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../024-ceph-base_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:23:54.874 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-09-09T22:23:54.887 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-base (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:23:55.144 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-pastedeploy. 2022-09-09T22:23:55.159 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2022-09-09T22:23:55.202 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-09-09T22:23:55.521 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-webtest. 2022-09-09T22:23:55.534 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2022-09-09T22:23:55.572 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-09-09T22:23:56.231 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-cherrypy3. 2022-09-09T22:23:56.244 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2022-09-09T22:23:56.281 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-09-09T22:23:56.624 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-pecan. 2022-09-09T22:23:56.638 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-09-09T22:23:56.643 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-natsort. 2022-09-09T22:23:56.658 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../026-python3-natsort_7.0.1-1_all.deb ... 2022-09-09T22:23:56.691 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-09-09T22:23:56.734 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-natsort (7.0.1-1) ... 2022-09-09T22:23:57.054 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-werkzeug. 2022-09-09T22:23:57.069 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-09-09T22:23:57.071 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-logutils. 2022-09-09T22:23:57.084 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2022-09-09T22:23:57.112 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-09-09T22:23:57.379 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-09-09T22:23:57.394 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-09-09T22:23:57.404 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../043-ceph-mgr-modules-core_17.0.0-14894-g28758235-1focal_all.deb ... 2022-09-09T22:23:57.464 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking ceph-mgr-modules-core (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:23:57.827 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2022-09-09T22:23:57.843 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../044-libsqlite3-mod-ceph_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:23:57.885 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking libsqlite3-mod-ceph (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:23:58.074 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-markupsafe. 2022-09-09T22:23:58.090 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-09-09T22:23:58.124 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-09-09T22:23:58.230 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-cryptography. 2022-09-09T22:23:58.243 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../045-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-09-09T22:23:58.280 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-09-09T22:23:58.485 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-mako. 2022-09-09T22:23:58.499 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-09-09T22:23:58.543 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-09-09T22:23:58.659 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-openssl. 2022-09-09T22:23:58.671 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../046-python3-openssl_19.0.0-1build1_all.deb ... 2022-09-09T22:23:58.708 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-09-09T22:23:58.897 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-simplegeneric. 2022-09-09T22:23:58.911 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2022-09-09T22:23:58.946 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-09-09T22:23:59.028 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package ceph-mgr. 2022-09-09T22:23:59.050 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../047-ceph-mgr_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:23:59.078 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking ceph-mgr (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:23:59.274 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-singledispatch. 2022-09-09T22:23:59.288 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-09-09T22:23:59.349 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-09-09T22:23:59.572 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package ceph-mon. 2022-09-09T22:23:59.585 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../048-ceph-mon_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:23:59.630 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking ceph-mon (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:23:59.645 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-webob. 2022-09-09T22:23:59.659 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2_all.deb ... 2022-09-09T22:23:59.701 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-09-09T22:24:00.013 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-soupsieve. 2022-09-09T22:24:00.027 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-09-09T22:24:00.071 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-09-09T22:24:00.399 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-bs4. 2022-09-09T22:24:00.413 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2022-09-09T22:24:00.449 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-09-09T22:24:00.777 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package ceph-osd. 2022-09-09T22:24:00.789 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../049-ceph-osd_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:24:00.819 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-waitress. 2022-09-09T22:24:00.832 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2022-09-09T22:24:00.833 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking ceph-osd (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:00.872 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2022-09-09T22:24:01.188 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-tempita. 2022-09-09T22:24:01.206 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2022-09-09T22:24:01.250 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-09-09T22:24:01.583 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-paste. 2022-09-09T22:24:01.597 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-09-09T22:24:01.666 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-09-09T22:24:01.970 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-09-09T22:24:01.984 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-09-09T22:24:02.029 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-09-09T22:24:02.356 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pastedeploy. 2022-09-09T22:24:02.374 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2022-09-09T22:24:02.414 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-09-09T22:24:02.742 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-webtest. 2022-09-09T22:24:02.758 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2022-09-09T22:24:02.817 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-09-09T22:24:03.153 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pecan. 2022-09-09T22:24:03.169 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-09-09T22:24:03.211 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-09-09T22:24:03.567 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-werkzeug. 2022-09-09T22:24:03.584 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-09-09T22:24:03.637 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package ceph. 2022-09-09T22:24:03.650 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-09-09T22:24:03.651 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../050-ceph_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:24:03.694 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking ceph (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:03.952 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-09-09T22:24:03.969 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../043-ceph-mgr-modules-core_17.0.0-14894-g28758235-1focal_all.deb ... 2022-09-09T22:24:03.980 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package ceph-fuse. 2022-09-09T22:24:03.994 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../051-ceph-fuse_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:24:04.011 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr-modules-core (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:04.040 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking ceph-fuse (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:04.370 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2022-09-09T22:24:04.381 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../044-libsqlite3-mod-ceph_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:24:04.432 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libsqlite3-mod-ceph (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:04.452 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package ceph-mds. 2022-09-09T22:24:04.466 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../052-ceph-mds_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:24:04.517 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking ceph-mds (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:04.819 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-cryptography. 2022-09-09T22:24:04.834 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../045-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-09-09T22:24:04.877 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-09-09T22:24:05.016 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package cephadm. 2022-09-09T22:24:05.027 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../053-cephadm_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:24:05.072 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking cephadm (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:05.285 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-openssl. 2022-09-09T22:24:05.298 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../046-python3-openssl_19.0.0-1build1_all.deb ... 2022-09-09T22:24:05.330 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-09-09T22:24:05.364 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-asyncssh. 2022-09-09T22:24:05.383 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../054-python3-asyncssh_1.12.2-1_all.deb ... 2022-09-09T22:24:05.415 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-asyncssh (1.12.2-1) ... 2022-09-09T22:24:05.708 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr. 2022-09-09T22:24:05.724 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../047-ceph-mgr_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:24:05.727 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-jinja2. 2022-09-09T22:24:05.742 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../055-python3-jinja2_2.10.1-2_all.deb ... 2022-09-09T22:24:05.767 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:05.777 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2022-09-09T22:24:06.082 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2022-09-09T22:24:06.095 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../056-ceph-mgr-cephadm_17.0.0-14894-g28758235-1focal_all.deb ... 2022-09-09T22:24:06.138 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking ceph-mgr-cephadm (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:06.322 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mon. 2022-09-09T22:24:06.337 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../048-ceph-mon_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:24:06.379 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mon (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:06.500 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-bcrypt. 2022-09-09T22:24:06.517 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../057-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-09-09T22:24:06.558 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-09-09T22:24:06.920 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-jwt. 2022-09-09T22:24:06.934 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../058-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2022-09-09T22:24:06.986 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2022-09-09T22:24:07.326 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-attr. 2022-09-09T22:24:07.338 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../059-python3-attr_19.3.0-2_all.deb ... 2022-09-09T22:24:07.382 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-09-09T22:24:07.532 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-osd. 2022-09-09T22:24:07.548 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../049-ceph-osd_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:24:07.585 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-osd (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:07.722 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-pluggy. 2022-09-09T22:24:07.733 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../060-python3-pluggy_0.13.0-2_all.deb ... 2022-09-09T22:24:07.777 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-09-09T22:24:08.096 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-py. 2022-09-09T22:24:08.113 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../061-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-09-09T22:24:08.162 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-09-09T22:24:08.482 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-atomicwrites. 2022-09-09T22:24:08.496 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../062-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-09-09T22:24:08.543 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-09-09T22:24:08.854 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-packaging. 2022-09-09T22:24:08.874 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../063-python3-packaging_20.3-1_all.deb ... 2022-09-09T22:24:08.911 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-packaging (20.3-1) ... 2022-09-09T22:24:09.213 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-wcwidth. 2022-09-09T22:24:09.226 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../064-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-09-09T22:24:09.262 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-09-09T22:24:09.582 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-pytest. 2022-09-09T22:24:09.597 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../065-python3-pytest_4.6.9-1_all.deb ... 2022-09-09T22:24:09.641 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-09-09T22:24:09.978 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-repoze.lru. 2022-09-09T22:24:09.993 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../066-python3-repoze.lru_0.7-2_all.deb ... 2022-09-09T22:24:10.031 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-09-09T22:24:10.305 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package libjs-underscore. 2022-09-09T22:24:10.320 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../067-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-09-09T22:24:10.374 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-09-09T22:24:10.470 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph. 2022-09-09T22:24:10.483 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../050-ceph_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:24:10.528 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:10.645 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-09-09T22:24:10.660 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../068-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-09-09T22:24:10.702 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-09-09T22:24:10.855 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-fuse. 2022-09-09T22:24:10.870 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../051-ceph-fuse_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:24:10.914 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-fuse (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:11.056 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-routes. 2022-09-09T22:24:11.075 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../069-python3-routes_2.4.1-2_all.deb ... 2022-09-09T22:24:11.114 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-09-09T22:24:11.335 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mds. 2022-09-09T22:24:11.353 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../052-ceph-mds_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:24:11.383 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mds (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:11.451 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-09-09T22:24:11.466 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../070-ceph-mgr-dashboard_17.0.0-14894-g28758235-1focal_all.deb ... 2022-09-09T22:24:11.526 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking ceph-mgr-dashboard (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:11.904 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package cephadm. 2022-09-09T22:24:11.919 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../053-cephadm_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:24:11.964 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking cephadm (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:11.998 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-decorator. 2022-09-09T22:24:12.009 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../071-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-09-09T22:24:12.058 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-09-09T22:24:12.313 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-asyncssh. 2022-09-09T22:24:12.326 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../054-python3-asyncssh_1.12.2-1_all.deb ... 2022-09-09T22:24:12.385 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-asyncssh (1.12.2-1) ... 2022-09-09T22:24:12.390 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-09-09T22:24:12.403 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../072-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-09-09T22:24:12.442 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-09-09T22:24:12.752 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-scipy. 2022-09-09T22:24:12.765 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../073-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-09-09T22:24:12.791 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-jinja2. 2022-09-09T22:24:12.818 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-09-09T22:24:12.819 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../055-python3-jinja2_2.10.1-2_all.deb ... 2022-09-09T22:24:12.850 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2022-09-09T22:24:13.219 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2022-09-09T22:24:13.236 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../056-ceph-mgr-cephadm_17.0.0-14894-g28758235-1focal_all.deb ... 2022-09-09T22:24:13.328 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr-cephadm (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:13.704 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-bcrypt. 2022-09-09T22:24:13.714 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../057-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-09-09T22:24:13.755 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-09-09T22:24:14.065 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-09-09T22:24:14.080 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-09-09T22:24:14.083 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-jwt. 2022-09-09T22:24:14.096 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../058-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2022-09-09T22:24:14.115 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-09-09T22:24:14.159 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2022-09-09T22:24:14.461 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-attr. 2022-09-09T22:24:14.478 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../059-python3-attr_19.3.0-2_all.deb ... 2022-09-09T22:24:14.510 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-09-09T22:24:14.661 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-joblib. 2022-09-09T22:24:14.672 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../075-python3-joblib_0.14.0-3_all.deb ... 2022-09-09T22:24:14.709 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-09-09T22:24:14.856 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pluggy. 2022-09-09T22:24:14.868 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../060-python3-pluggy_0.13.0-2_all.deb ... 2022-09-09T22:24:14.906 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-09-09T22:24:15.096 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-sklearn. 2022-09-09T22:24:15.117 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../076-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-09-09T22:24:15.154 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-09-09T22:24:15.299 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-py. 2022-09-09T22:24:15.312 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../061-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-09-09T22:24:15.354 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-09-09T22:24:15.690 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-atomicwrites. 2022-09-09T22:24:15.706 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../062-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-09-09T22:24:15.748 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-09-09T22:24:15.864 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-09-09T22:24:15.880 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../077-ceph-mgr-diskprediction-local_17.0.0-14894-g28758235-1focal_all.deb ... 2022-09-09T22:24:15.922 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking ceph-mgr-diskprediction-local (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:16.074 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-packaging. 2022-09-09T22:24:16.086 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../063-python3-packaging_20.3-1_all.deb ... 2022-09-09T22:24:16.148 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-packaging (20.3-1) ... 2022-09-09T22:24:16.450 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-wcwidth. 2022-09-09T22:24:16.461 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../064-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-09-09T22:24:16.498 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-09-09T22:24:16.826 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pytest. 2022-09-09T22:24:16.841 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../065-python3-pytest_4.6.9-1_all.deb ... 2022-09-09T22:24:16.885 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-09-09T22:24:17.044 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-cachetools. 2022-09-09T22:24:17.057 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../078-python3-cachetools_4.0.0-1_all.deb ... 2022-09-09T22:24:17.092 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-09-09T22:24:17.241 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-repoze.lru. 2022-09-09T22:24:17.242 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../066-python3-repoze.lru_0.7-2_all.deb ... 2022-09-09T22:24:17.313 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-09-09T22:24:17.412 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-pyasn1. 2022-09-09T22:24:17.430 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../079-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-09-09T22:24:17.474 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-09-09T22:24:17.594 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libjs-underscore. 2022-09-09T22:24:17.615 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../067-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-09-09T22:24:17.651 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-09-09T22:24:17.783 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-09-09T22:24:17.798 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../080-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-09-09T22:24:17.840 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-09-09T22:24:17.985 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-09-09T22:24:18.001 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../068-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-09-09T22:24:18.029 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-09-09T22:24:18.258 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-rsa. 2022-09-09T22:24:18.267 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../081-python3-rsa_4.0-3_all.deb ... 2022-09-09T22:24:18.303 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-rsa (4.0-3) ... 2022-09-09T22:24:18.324 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-routes. 2022-09-09T22:24:18.343 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../069-python3-routes_2.4.1-2_all.deb ... 2022-09-09T22:24:18.374 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-09-09T22:24:18.641 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-google-auth. 2022-09-09T22:24:18.656 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../082-python3-google-auth_1.5.1-2_all.deb ... 2022-09-09T22:24:18.699 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-09-09T22:24:18.743 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-09-09T22:24:18.758 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../070-ceph-mgr-dashboard_17.0.0-14894-g28758235-1focal_all.deb ... 2022-09-09T22:24:18.803 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr-dashboard (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:19.019 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-blinker. 2022-09-09T22:24:19.040 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../083-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-09-09T22:24:19.068 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-09-09T22:24:19.356 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-decorator. 2022-09-09T22:24:19.370 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../071-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-09-09T22:24:19.391 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-oauthlib. 2022-09-09T22:24:19.410 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../084-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-09-09T22:24:19.424 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-09-09T22:24:19.444 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-09-09T22:24:19.725 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-09-09T22:24:19.741 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-09-09T22:24:19.742 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../085-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-09-09T22:24:19.762 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../072-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-09-09T22:24:19.776 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-09-09T22:24:19.799 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-09-09T22:24:20.103 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-websocket. 2022-09-09T22:24:20.121 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-scipy. 2022-09-09T22:24:20.122 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../086-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-09-09T22:24:20.132 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../073-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-09-09T22:24:20.161 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-09-09T22:24:20.192 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-09-09T22:24:20.477 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-kubernetes. 2022-09-09T22:24:20.486 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../087-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-09-09T22:24:20.523 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-09-09T22:24:21.141 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-09-09T22:24:21.160 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../088-ceph-mgr-k8sevents_17.0.0-14894-g28758235-1focal_all.deb ... 2022-09-09T22:24:21.190 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking ceph-mgr-k8sevents (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:21.531 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package libonig5:amd64. 2022-09-09T22:24:21.550 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-09-09T22:24:21.551 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../089-libonig5_6.9.4-1_amd64.deb ... 2022-09-09T22:24:21.574 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-09-09T22:24:21.580 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-09-09T22:24:21.608 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-09-09T22:24:21.892 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package libjq1:amd64. 2022-09-09T22:24:21.914 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../090-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-09-09T22:24:21.935 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-09-09T22:24:22.232 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-joblib. 2022-09-09T22:24:22.240 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package jq. 2022-09-09T22:24:22.241 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../075-python3-joblib_0.14.0-3_all.deb ... 2022-09-09T22:24:22.266 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../091-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-09-09T22:24:22.291 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-09-09T22:24:22.291 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-09-09T22:24:22.541 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package socat. 2022-09-09T22:24:22.565 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../092-socat_1.7.3.3-2_amd64.deb ... 2022-09-09T22:24:22.599 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking socat (1.7.3.3-2) ... 2022-09-09T22:24:22.668 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-sklearn. 2022-09-09T22:24:22.678 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../076-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-09-09T22:24:22.720 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-09-09T22:24:23.027 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-09-09T22:24:23.041 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../093-libxslt1.1_1.1.34-4ubuntu0.20.04.1_amd64.deb ... 2022-09-09T22:24:23.076 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4ubuntu0.20.04.1) ... 2022-09-09T22:24:23.321 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package xmlstarlet. 2022-09-09T22:24:23.337 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../094-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-09-09T22:24:23.379 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-09-09T22:24:23.596 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-09-09T22:24:23.608 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../077-ceph-mgr-diskprediction-local_17.0.0-14894-g28758235-1focal_all.deb ... 2022-09-09T22:24:23.648 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr-diskprediction-local (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:23.734 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package ceph-test. 2022-09-09T22:24:23.748 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../095-ceph-test_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:24:23.784 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking ceph-test (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:24.804 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-cachetools. 2022-09-09T22:24:24.823 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../078-python3-cachetools_4.0.0-1_all.deb ... 2022-09-09T22:24:24.884 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-09-09T22:24:25.188 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pyasn1. 2022-09-09T22:24:25.212 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../079-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-09-09T22:24:25.230 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-09-09T22:24:25.574 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-09-09T22:24:25.588 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../080-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-09-09T22:24:25.642 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-09-09T22:24:26.066 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-rsa. 2022-09-09T22:24:26.076 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../081-python3-rsa_4.0-3_all.deb ... 2022-09-09T22:24:26.125 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-rsa (4.0-3) ... 2022-09-09T22:24:26.467 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-google-auth. 2022-09-09T22:24:26.482 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../082-python3-google-auth_1.5.1-2_all.deb ... 2022-09-09T22:24:26.539 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-09-09T22:24:26.902 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-blinker. 2022-09-09T22:24:26.920 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../083-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-09-09T22:24:26.961 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-09-09T22:24:27.229 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-oauthlib. 2022-09-09T22:24:27.243 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../084-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-09-09T22:24:27.287 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-09-09T22:24:27.608 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-09-09T22:24:27.627 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../085-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-09-09T22:24:27.658 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-09-09T22:24:27.985 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-websocket. 2022-09-09T22:24:28.001 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../086-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-09-09T22:24:28.060 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-09-09T22:24:28.397 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-kubernetes. 2022-09-09T22:24:28.413 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../087-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-09-09T22:24:28.455 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-09-09T22:24:28.993 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-09-09T22:24:29.010 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../088-ceph-mgr-k8sevents_17.0.0-14894-g28758235-1focal_all.deb ... 2022-09-09T22:24:29.012 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package ceph-volume. 2022-09-09T22:24:29.026 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../096-ceph-volume_17.0.0-14894-g28758235-1focal_all.deb ... 2022-09-09T22:24:29.043 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr-k8sevents (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:29.070 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking ceph-volume (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:29.387 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libonig5:amd64. 2022-09-09T22:24:29.404 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../089-libonig5_6.9.4-1_amd64.deb ... 2022-09-09T22:24:29.420 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package fonts-mathjax. 2022-09-09T22:24:29.433 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../097-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2022-09-09T22:24:29.437 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-09-09T22:24:29.475 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2022-09-09T22:24:29.825 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libjq1:amd64. 2022-09-09T22:24:29.840 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../090-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-09-09T22:24:29.882 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-09-09T22:24:29.964 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package libcephfs-dev. 2022-09-09T22:24:29.977 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../098-libcephfs-dev_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:24:30.027 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking libcephfs-dev (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:30.177 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package jq. 2022-09-09T22:24:30.191 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../091-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-09-09T22:24:30.234 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-09-09T22:24:30.382 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package nvme-cli. 2022-09-09T22:24:30.399 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../099-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-09-09T22:24:30.449 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-09-09T22:24:30.520 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package socat. 2022-09-09T22:24:30.535 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../092-socat_1.7.3.3-2_amd64.deb ... 2022-09-09T22:24:30.581 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking socat (1.7.3.3-2) ... 2022-09-09T22:24:30.829 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python-asyncssh-doc. 2022-09-09T22:24:30.845 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../100-python-asyncssh-doc_1.12.2-1_all.deb ... 2022-09-09T22:24:30.890 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python-asyncssh-doc (1.12.2-1) ... 2022-09-09T22:24:31.007 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-09-09T22:24:31.024 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../093-libxslt1.1_1.1.34-4ubuntu0.20.04.1_amd64.deb ... 2022-09-09T22:24:31.065 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4ubuntu0.20.04.1) ... 2022-09-09T22:24:31.241 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package libjs-mathjax. 2022-09-09T22:24:31.255 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../101-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2022-09-09T22:24:31.299 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2022-09-09T22:24:31.327 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package xmlstarlet. 2022-09-09T22:24:31.344 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../094-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-09-09T22:24:31.376 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-09-09T22:24:31.732 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-test. 2022-09-09T22:24:31.764 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../095-ceph-test_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:24:31.829 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-test (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:33.018 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python-natsort-doc. 2022-09-09T22:24:33.030 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../102-python-natsort-doc_7.0.1-1_all.deb ... 2022-09-09T22:24:33.098 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2022-09-09T22:24:33.452 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-webencodings. 2022-09-09T22:24:33.466 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../103-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-09-09T22:24:33.510 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-09-09T22:24:33.838 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-html5lib. 2022-09-09T22:24:33.856 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../104-python3-html5lib_1.0.1-2_all.deb ... 2022-09-09T22:24:33.897 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-09-09T22:24:34.233 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-09-09T22:24:34.248 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../105-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-09-09T22:24:34.291 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-09-09T22:24:34.812 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-pastescript. 2022-09-09T22:24:34.826 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2022-09-09T22:24:34.870 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-09-09T22:24:35.232 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-psutil. 2022-09-09T22:24:35.246 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../107-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-09-09T22:24:35.290 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-09-09T22:24:35.619 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-pyinotify. 2022-09-09T22:24:35.635 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../108-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-09-09T22:24:35.676 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-09-09T22:24:36.021 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-simplejson. 2022-09-09T22:24:36.034 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../109-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-09-09T22:24:36.079 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-09-09T22:24:36.332 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package qttranslations5-l10n. 2022-09-09T22:24:36.353 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../110-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2022-09-09T22:24:36.382 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2022-09-09T22:24:37.001 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-volume. 2022-09-09T22:24:37.014 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../096-ceph-volume_17.0.0-14894-g28758235-1focal_all.deb ... 2022-09-09T22:24:37.050 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-volume (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:37.103 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package radosgw. 2022-09-09T22:24:37.125 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../111-radosgw_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:24:37.170 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking radosgw (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:37.390 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package fonts-mathjax. 2022-09-09T22:24:37.402 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../097-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2022-09-09T22:24:37.474 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2022-09-09T22:24:37.934 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libcephfs-dev. 2022-09-09T22:24:37.951 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../098-libcephfs-dev_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:24:37.976 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libcephfs-dev (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:38.420 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package nvme-cli. 2022-09-09T22:24:38.436 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../099-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-09-09T22:24:38.496 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-09-09T22:24:38.882 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python-asyncssh-doc. 2022-09-09T22:24:38.898 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../100-python-asyncssh-doc_1.12.2-1_all.deb ... 2022-09-09T22:24:38.941 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python-asyncssh-doc (1.12.2-1) ... 2022-09-09T22:24:39.016 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package rbd-fuse. 2022-09-09T22:24:39.030 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../112-rbd-fuse_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:24:39.073 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking rbd-fuse (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:39.344 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libjs-mathjax. 2022-09-09T22:24:39.361 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../101-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2022-09-09T22:24:39.375 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-attr (19.3.0-2) ... 2022-09-09T22:24:39.403 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2022-09-09T22:24:39.713 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-09-09T22:24:39.838 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2022-09-09T22:24:39.964 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-09-09T22:24:40.166 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up cephadm (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:40.819 INFO:teuthology.orchestra.run.smithi080.stdout:Adding system user cephadm....done 2022-09-09T22:24:40.964 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2022-09-09T22:24:41.090 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2022-09-09T22:24:41.221 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python-natsort-doc. 2022-09-09T22:24:41.237 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../102-python-natsort-doc_7.0.1-1_all.deb ... 2022-09-09T22:24:41.278 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2022-09-09T22:24:41.442 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-09-09T22:24:41.665 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-webencodings. 2022-09-09T22:24:41.680 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../103-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-09-09T22:24:41.723 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-09-09T22:24:41.735 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2022-09-09T22:24:41.870 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-09-09T22:24:41.995 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-09-09T22:24:42.052 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-html5lib. 2022-09-09T22:24:42.065 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../104-python3-html5lib_1.0.1-2_all.deb ... 2022-09-09T22:24:42.101 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-09-09T22:24:42.356 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-09-09T22:24:42.413 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-09-09T22:24:42.427 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../105-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-09-09T22:24:42.493 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-09-09T22:24:42.770 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2022-09-09T22:24:42.941 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pastescript. 2022-09-09T22:24:42.956 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2022-09-09T22:24:43.002 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-09-09T22:24:43.067 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-09-09T22:24:43.344 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-09-09T22:24:43.361 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-psutil. 2022-09-09T22:24:43.375 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../107-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-09-09T22:24:43.436 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-09-09T22:24:43.624 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-09-09T22:24:43.806 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pyinotify. 2022-09-09T22:24:43.821 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../108-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-09-09T22:24:43.870 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-09-09T22:24:43.913 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-ceph-argparse (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:44.206 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-09-09T22:24:44.332 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-09-09T22:24:44.575 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-simplejson. 2022-09-09T22:24:44.589 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../109-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-09-09T22:24:44.625 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-09-09T22:24:44.666 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-09-09T22:24:44.879 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package qttranslations5-l10n. 2022-09-09T22:24:44.892 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../110-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2022-09-09T22:24:44.935 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2022-09-09T22:24:44.960 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up libpcre2-16-0:amd64 (10.34-7) ... 2022-09-09T22:24:45.083 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-09-09T22:24:45.470 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-natsort (7.0.1-1) ... 2022-09-09T22:24:45.648 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package radosgw. 2022-09-09T22:24:45.661 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../111-radosgw_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:24:45.722 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking radosgw (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:45.755 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-09-09T22:24:46.065 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-09-09T22:24:46.410 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-09-09T22:24:46.701 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2022-09-09T22:24:47.019 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-09-09T22:24:47.312 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-packaging (20.3-1) ... 2022-09-09T22:24:47.604 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-09-09T22:24:47.651 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package rbd-fuse. 2022-09-09T22:24:47.667 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../112-rbd-fuse_17.0.0-14894-g28758235-1focal_amd64.deb ... 2022-09-09T22:24:47.709 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking rbd-fuse (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:47.856 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2022-09-09T22:24:48.015 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-09-09T22:24:48.067 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-attr (19.3.0-2) ... 2022-09-09T22:24:48.308 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-09-09T22:24:48.355 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-09-09T22:24:48.498 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2022-09-09T22:24:48.624 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-09-09T22:24:48.702 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-09-09T22:24:48.869 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up cephadm (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:49.078 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-09-09T22:24:49.448 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up socat (1.7.3.3-2) ... 2022-09-09T22:24:49.550 INFO:teuthology.orchestra.run.smithi005.stdout:Adding system user cephadm....done 2022-09-09T22:24:49.573 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-ceph-common (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:49.715 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2022-09-09T22:24:49.866 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2022-09-09T22:24:49.900 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2022-09-09T22:24:50.026 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-09-09T22:24:50.202 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-09-09T22:24:50.370 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-09-09T22:24:50.485 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2022-09-09T22:24:50.495 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-09-09T22:24:50.616 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-09-09T22:24:50.754 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-09-09T22:24:50.813 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2022-09-09T22:24:50.940 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2022-09-09T22:24:51.064 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-09-09T22:24:51.065 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up libxslt1.1:amd64 (1.1.34-4ubuntu0.20.04.1) ... 2022-09-09T22:24:51.192 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-09-09T22:24:51.473 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2022-09-09T22:24:51.519 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-09-09T22:24:51.792 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-09-09T22:24:51.803 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-09-09T22:24:52.105 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-09-09T22:24:52.132 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-09-09T22:24:52.264 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-09-09T22:24:52.435 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-09-09T22:24:52.540 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-09-09T22:24:52.736 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-ceph-argparse (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:52.833 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-09-09T22:24:53.029 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-09-09T22:24:53.139 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-09-09T22:24:53.163 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-09-09T22:24:53.423 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-09-09T22:24:53.598 INFO:teuthology.orchestra.run.smithi080.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-09-09T22:24:53.716 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libpcre2-16-0:amd64 (10.34-7) ... 2022-09-09T22:24:53.862 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-09-09T22:24:53.869 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-09-09T22:24:53.988 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-09-09T22:24:54.252 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-natsort (7.0.1-1) ... 2022-09-09T22:24:54.349 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-09-09T22:24:54.553 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-09-09T22:24:54.708 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-09-09T22:24:54.835 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-09-09T22:24:54.837 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-09-09T22:24:54.960 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-09-09T22:24:55.165 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-09-09T22:24:55.295 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-09-09T22:24:55.448 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2022-09-09T22:24:55.579 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-09-09T22:24:55.782 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-09-09T22:24:56.119 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-packaging (20.3-1) ... 2022-09-09T22:24:56.468 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-09-09T22:24:56.594 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2022-09-09T22:24:56.754 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-09-09T22:24:57.036 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-09-09T22:24:57.046 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-09-09T22:24:57.196 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-09-09T22:24:57.432 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-09-09T22:24:57.564 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-09-09T22:24:57.815 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-09-09T22:24:57.865 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-09-09T22:24:58.187 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2022-09-09T22:24:58.227 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up socat (1.7.3.3-2) ... 2022-09-09T22:24:58.311 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-09-09T22:24:58.355 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-ceph-common (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:24:58.636 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-09-09T22:24:58.670 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2022-09-09T22:24:58.821 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-09-09T22:24:58.964 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-asyncssh (1.12.2-1) ... 2022-09-09T22:24:59.156 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-09-09T22:24:59.281 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-09-09T22:24:59.339 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-09-09T22:24:59.638 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2022-09-09T22:24:59.666 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-09-09T22:24:59.759 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2022-09-09T22:24:59.907 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libxslt1.1:amd64 (1.1.34-4ubuntu0.20.04.1) ... 2022-09-09T22:25:00.036 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-09-09T22:25:00.370 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-09-09T22:25:00.655 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-09-09T22:25:00.713 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-09-09T22:25:00.856 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-09-09T22:25:00.931 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-09-09T22:25:00.982 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2022-09-09T22:25:01.090 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-09-09T22:25:01.117 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-09-09T22:25:01.400 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-09-09T22:25:01.444 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-09-09T22:25:01.561 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-rsa (4.0-3) ... 2022-09-09T22:25:01.701 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-09-09T22:25:01.855 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2022-09-09T22:25:01.981 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-09-09T22:25:02.025 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-09-09T22:25:02.307 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-09-09T22:25:02.348 INFO:teuthology.orchestra.run.smithi005.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-09-09T22:25:02.608 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-09-09T22:25:02.638 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-09-09T22:25:02.787 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-09-09T22:25:02.918 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-09-09T22:25:03.107 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-09-09T22:25:03.256 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up librados2 (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:03.381 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python-asyncssh-doc (1.12.2-1) ... 2022-09-09T22:25:03.470 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-09-09T22:25:03.540 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up librgw2 (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:03.598 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-09-09T22:25:03.747 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-09-09T22:25:03.753 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up libsqlite3-mod-ceph (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:03.864 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up libcephfs2 (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:03.978 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-routes (2.4.1-2) ... 2022-09-09T22:25:04.089 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-09-09T22:25:04.288 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up libradosstriper1 (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:04.373 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-09-09T22:25:04.414 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-09-09T22:25:04.716 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up librbd1 (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:04.860 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up ceph-mgr-modules-core (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:04.993 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up ceph-fuse (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:05.211 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-09-09T22:25:05.212 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-09-09T22:25:05.670 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up libcephfs-dev (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:05.796 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-rados (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:05.887 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-09-09T22:25:05.923 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-09-09T22:25:06.038 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-09-09T22:25:06.415 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-09-09T22:25:06.724 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-09-09T22:25:06.902 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-rbd (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:07.051 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2022-09-09T22:25:07.061 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up rbd-fuse (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:07.188 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-rgw (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:07.210 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-09-09T22:25:07.313 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-cephfs (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:07.440 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up ceph-common (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:07.553 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-09-09T22:25:07.872 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-asyncssh (1.12.2-1) ... 2022-09-09T22:25:07.993 INFO:teuthology.orchestra.run.smithi080.stdout:Adding group ceph....done 2022-09-09T22:25:08.298 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-09-09T22:25:08.378 INFO:teuthology.orchestra.run.smithi080.stdout:Adding system user ceph....done 2022-09-09T22:25:08.580 INFO:teuthology.orchestra.run.smithi080.stdout:Setting system user ceph properties....done 2022-09-09T22:25:08.596 INFO:teuthology.orchestra.run.smithi080.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-09-09T22:25:08.625 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-09-09T22:25:08.740 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-09-09T22:25:09.051 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-09-09T22:25:09.501 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up ceph-test (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:09.627 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up radosgw (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:09.713 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-09-09T22:25:09.839 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-09-09T22:25:09.905 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-09-09T22:25:09.905 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-09-09T22:25:09.988 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2022-09-09T22:25:10.117 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-09-09T22:25:10.443 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-09-09T22:25:10.571 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up ceph-base (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:10.594 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-rsa (4.0-3) ... 2022-09-09T22:25:10.904 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2022-09-09T22:25:10.945 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-09-09T22:25:11.030 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-09-09T22:25:11.348 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-09-09T22:25:11.381 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up ceph-mds (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:11.611 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-09-09T22:25:11.611 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-09-09T22:25:11.658 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-09-09T22:25:12.017 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-09-09T22:25:12.041 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up ceph-mgr (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:12.268 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-09-09T22:25:12.269 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-09-09T22:25:12.378 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up librados2 (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:12.504 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python-asyncssh-doc (1.12.2-1) ... 2022-09-09T22:25:12.655 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up librgw2 (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:12.701 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up ceph-osd (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:12.891 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libsqlite3-mod-ceph (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:13.008 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libcephfs2 (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:13.168 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-routes (2.4.1-2) ... 2022-09-09T22:25:13.191 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-09-09T22:25:13.191 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-09-09T22:25:13.486 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libradosstriper1 (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:13.587 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-09-09T22:25:13.906 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up librbd1 (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:13.945 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up ceph-mgr-k8sevents (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:14.056 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr-modules-core (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:14.113 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up ceph-mgr-diskprediction-local (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:14.186 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-fuse (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:14.247 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up ceph-mon (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:14.408 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-09-09T22:25:14.409 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-09-09T22:25:14.458 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-09-09T22:25:14.459 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-09-09T22:25:14.850 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libcephfs-dev (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:14.899 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up ceph-mgr-cephadm (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:15.042 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up ceph (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:15.168 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up ceph-mgr-dashboard (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:15.311 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up ceph-volume (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:15.315 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-rados (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:15.457 INFO:teuthology.orchestra.run.smithi080.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-09-09T22:25:15.486 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-09-09T22:25:15.632 INFO:teuthology.orchestra.run.smithi080.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2022-09-09T22:25:15.916 INFO:teuthology.orchestra.run.smithi080.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-09-09T22:25:16.464 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-rbd (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:16.565 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up rbd-fuse (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:16.716 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-rgw (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:16.842 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-cephfs (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:17.003 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-common (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:17.596 INFO:teuthology.orchestra.run.smithi005.stdout:Adding group ceph....done 2022-09-09T22:25:17.981 INFO:teuthology.orchestra.run.smithi005.stdout:Adding system user ceph....done 2022-09-09T22:25:18.183 INFO:teuthology.orchestra.run.smithi005.stdout:Setting system user ceph properties....done 2022-09-09T22:25:18.198 INFO:teuthology.orchestra.run.smithi005.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-09-09T22:25:18.353 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-09-09T22:25:18.627 INFO:teuthology.orchestra.run.smithi080.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2022-09-09T22:25:18.673 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-09-09T22:25:19.195 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-test (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:19.380 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up radosgw (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:19.457 INFO:teuthology.orchestra.run.smithi080.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-09-09T22:25:19.460 DEBUG:teuthology.parallel:result is None 2022-09-09T22:25:19.672 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-09-09T22:25:19.672 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-09-09T22:25:20.316 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-base (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:20.715 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-09-09T22:25:21.194 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mds (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:21.441 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-09-09T22:25:21.442 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-09-09T22:25:21.895 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:22.126 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-09-09T22:25:22.127 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-09-09T22:25:22.571 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-osd (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:23.090 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-09-09T22:25:23.090 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-09-09T22:25:23.531 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr-k8sevents (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:23.674 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr-diskprediction-local (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:23.820 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mon (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:24.071 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-09-09T22:25:24.071 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-09-09T22:25:24.510 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr-cephadm (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:24.654 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:24.804 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr-dashboard (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:24.955 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-volume (17.0.0-14894-g28758235-1focal) ... 2022-09-09T22:25:25.107 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-09-09T22:25:25.259 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2022-09-09T22:25:25.602 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-09-09T22:25:28.419 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2022-09-09T22:25:29.186 INFO:teuthology.orchestra.run.smithi005.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-09-09T22:25:29.194 DEBUG:teuthology.parallel:result is None 2022-09-09T22:25:29.195 DEBUG:teuthology.orchestra.run.smithi005:> sudo lsb_release -is 2022-09-09T22:25:29.241 INFO:teuthology.orchestra.run.smithi005.stdout:Ubuntu 2022-09-09T22:25:29.246 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-09-09T22:25:29.247 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=28758235ba2ab1c1d8afa0f88d00f07b5687a51e 2022-09-09T22:25:29.404 DEBUG:teuthology.orchestra.run.smithi005:> dpkg-query -W -f '${Version}' ceph 2022-09-09T22:25:29.430 INFO:teuthology.orchestra.run.smithi005.stdout:17.0.0-14894-g28758235-1focal 2022-09-09T22:25:29.431 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-14894-g28758235-1focal 2022-09-09T22:25:29.431 INFO:teuthology.task.install:The correct ceph version 17.0.0-14894-g28758235-1focal is installed. 2022-09-09T22:25:29.432 DEBUG:teuthology.orchestra.run.smithi080:> sudo lsb_release -is 2022-09-09T22:25:29.484 INFO:teuthology.orchestra.run.smithi080.stdout:Ubuntu 2022-09-09T22:25:29.490 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-09-09T22:25:29.491 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=28758235ba2ab1c1d8afa0f88d00f07b5687a51e 2022-09-09T22:25:29.653 DEBUG:teuthology.orchestra.run.smithi080:> dpkg-query -W -f '${Version}' ceph 2022-09-09T22:25:29.679 INFO:teuthology.orchestra.run.smithi080.stdout:17.0.0-14894-g28758235-1focal 2022-09-09T22:25:29.680 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-14894-g28758235-1focal 2022-09-09T22:25:29.681 INFO:teuthology.task.install:The correct ceph version 17.0.0-14894-g28758235-1focal is installed. 2022-09-09T22:25:29.689 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-09-09T22:25:29.690 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-09-09T22:25:29.691 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-09-09T22:25:29.709 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2022-09-09T22:25:29.709 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-09-09T22:25:29.727 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-09-09T22:25:29.728 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-09-09T22:25:29.728 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/daemon-helper 2022-09-09T22:25:29.771 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-09-09T22:25:29.787 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2022-09-09T22:25:29.788 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/usr/bin/daemon-helper 2022-09-09T22:25:29.807 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-09-09T22:25:29.826 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-09-09T22:25:29.827 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-09-09T22:25:29.827 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/adjust-ulimits 2022-09-09T22:25:29.848 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-09-09T22:25:29.859 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2022-09-09T22:25:29.859 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/usr/bin/adjust-ulimits 2022-09-09T22:25:29.876 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-09-09T22:25:29.934 INFO:teuthology.run_tasks:Running task ceph... 2022-09-09T22:25:30.019 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2022-09-09T22:25:30.020 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 777 /var/log/ceph 2022-09-09T22:25:30.023 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod 777 /var/log/ceph 2022-09-09T22:25:30.040 INFO:tasks.ceph:Disabling ceph logrotate... 2022-09-09T22:25:30.040 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /etc/logrotate.d/ceph 2022-09-09T22:25:30.043 DEBUG:teuthology.orchestra.run.smithi080:> sudo rm -f -- /etc/logrotate.d/ceph 2022-09-09T22:25:30.097 INFO:tasks.ceph:Creating extra log directories... 2022-09-09T22:25:30.097 DEBUG:teuthology.orchestra.run.smithi005:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-09-09T22:25:30.101 DEBUG:teuthology.orchestra.run.smithi080:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-09-09T22:25:30.137 INFO:tasks.ceph:Creating ceph cluster c1... 2022-09-09T22:25:30.138 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2022-09-09T22:25:30.139 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/mbenjamin-2022-09-09_22:09:46-rgw-wip-librgw-frontends-distro-default-smithi/7023840', 'branch': 'wip-librgw-frontends', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{ubuntu_latest} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '7023840', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'mbenjamin-2022-09-09_22:09:46-rgw-wip-librgw-frontends-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-librgw-frontends'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '28758235ba2ab1c1d8afa0f88d00f07b5687a51e', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '28758235ba2ab1c1d8afa0f88d00f07b5687a51e'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py', 'tests_ps.py']}, 'workunit': {'branch': 'wip-librgw-frontends', 'sha1': '28758235ba2ab1c1d8afa0f88d00f07b5687a51e'}}, 'owner': 'scheduled_mbenjamin@teuthology', 'priority': 70, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '28758235ba2ab1c1d8afa0f88d00f07b5687a51e', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-librgw-frontends', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_28758235ba2ab1c1d8afa0f88d00f07b5687a51e/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '28758235ba2ab1c1d8afa0f88d00f07b5687a51e', 'targets': {'smithi005.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzFKnoOt2BoS/UBn9KunV/1GYUVB2AcY7cWUKACXEAWiSig9PUvOMwoMCkSvUbgs5stNzMZn58M3TydDGcq1zQwW7asez3uAzAi387ckCknkDmTFJdbrHLGna2mHGF6eL74cRn+9grca/oUUwKrTEQsW8bsgiHtL0gTV/f1rTK/3pyDBRSyy6gfPbIU3LK63P4/e8rRTNm57Jr/RWGt1TwV7dTiHSbBrAJpxD/6qY/4q4fsVM7HJZHB7rmV8tdeoB85+15PDy8JjoyxBdlp1UOE7NIrvBQzu+9V0nR2UvH8N4ryDBG8N1XfHsD9R+3Zg2lCYHenXcKJupnY6xNoSlJfHfkwsard+2GIy+4aJ57s6c3K+UKjnqdxib40X9RdJzNs2Qc+ARp+zbO+sAamr4wWWZFb0nhZUO89u3INoFbpT5gC3mnCI+p2YTeHZG/kJ8U14RnQ37Mai06jQeqFIHivFr8JX/3/pfWiBXg/9bl3y/37pU24aQ/dtBg/K87W68=', 'smithi080.front.sepia.ceph.com': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIC0XOORBR/2Qj/b6x+ngeEme2+rTgZ2w3aDFr4sjpDmu'}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '28758235ba2ab1c1d8afa0f88d00f07b5687a51e', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'b58f2c18636eb10faa77ed3614abd00cb85dfc2c', 'timestamp': '2022-09-09_22:09:46', 'tube': 'smithi', 'user': 'mbenjamin', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.1713'} 2022-09-09T22:25:30.141 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2022-09-09T22:25:30.146 DEBUG:teuthology.orchestra.run.smithi080:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2022-09-09T22:25:30.155 DEBUG:teuthology.orchestra.run.smithi005:> sudo install -d -m0777 -- /var/run/ceph 2022-09-09T22:25:30.196 DEBUG:teuthology.orchestra.run.smithi080:> sudo install -d -m0777 -- /var/run/ceph 2022-09-09T22:25:30.215 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-09-09T22:25:30.215 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/scratch_devs of=/dev/stdout 2022-09-09T22:25:30.223 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-09-09T22:25:30.224 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-09-09T22:25:30.272 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-09-09T22:25:30.272 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-09T22:25:30.272 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 6h/6d Inode: 360 Links: 1 2022-09-09T22:25:30.273 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-09T22:25:30.273 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2022-09-09 22:22:34.892006328 +0000 2022-09-09T22:25:30.273 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2022-09-09 22:22:34.520006316 +0000 2022-09-09T22:25:30.273 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2022-09-09 22:22:34.520006316 +0000 2022-09-09T22:25:30.274 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2022-09-09T22:25:30.282 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2022-09-09T22:25:30.283 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2022-09-09T22:25:30.283 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000303711 s, 1.7 MB/s 2022-09-09T22:25:30.287 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-09-09T22:25:30.294 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-09-09T22:25:30.294 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-09T22:25:30.295 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 6h/6d Inode: 368 Links: 1 2022-09-09T22:25:30.295 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-09T22:25:30.295 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2022-09-09 22:22:35.456006346 +0000 2022-09-09T22:25:30.295 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2022-09-09 22:22:35.104006335 +0000 2022-09-09T22:25:30.296 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2022-09-09 22:22:35.104006335 +0000 2022-09-09T22:25:30.296 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2022-09-09T22:25:30.304 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2022-09-09T22:25:30.305 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2022-09-09T22:25:30.305 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000199074 s, 2.6 MB/s 2022-09-09T22:25:30.309 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-09-09T22:25:30.316 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-09-09T22:25:30.317 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-09T22:25:30.317 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 6h/6d Inode: 374 Links: 1 2022-09-09T22:25:30.317 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-09T22:25:30.317 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2022-09-09 22:22:36.020006364 +0000 2022-09-09T22:25:30.318 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2022-09-09 22:22:35.668006353 +0000 2022-09-09T22:25:30.318 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2022-09-09 22:22:35.668006353 +0000 2022-09-09T22:25:30.318 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2022-09-09T22:25:30.326 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2022-09-09T22:25:30.326 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2022-09-09T22:25:30.327 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.00025056 s, 2.0 MB/s 2022-09-09T22:25:30.330 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-09-09T22:25:30.380 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-09-09T22:25:30.381 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-09T22:25:30.381 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 6h/6d Inode: 379 Links: 1 2022-09-09T22:25:30.382 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-09T22:25:30.382 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2022-09-09 22:22:36.560006382 +0000 2022-09-09T22:25:30.383 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2022-09-09 22:22:36.224006371 +0000 2022-09-09T22:25:30.383 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2022-09-09 22:22:36.224006371 +0000 2022-09-09T22:25:30.383 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2022-09-09T22:25:30.391 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2022-09-09T22:25:30.392 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2022-09-09T22:25:30.392 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.00029745 s, 1.7 MB/s 2022-09-09T22:25:30.395 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-09T22:25:30.396 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi005.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2022-09-09T22:25:30.396 INFO:tasks.ceph:Generating config... 2022-09-09T22:25:30.398 INFO:tasks.ceph:[client] debug rgw = 20 2022-09-09T22:25:30.398 INFO:tasks.ceph:[client] debug rgw sync = 20 2022-09-09T22:25:30.399 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2022-09-09T22:25:30.399 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2022-09-09T22:25:30.399 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-09-09T22:25:30.400 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2022-09-09T22:25:30.400 INFO:tasks.ceph:[client] rgw data log num shards = 4 2022-09-09T22:25:30.400 INFO:tasks.ceph:[client] rgw md log max shards = 4 2022-09-09T22:25:30.401 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2022-09-09T22:25:30.401 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2022-09-09T22:25:30.401 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2022-09-09T22:25:30.402 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2022-09-09T22:25:30.402 INFO:tasks.ceph:[client] setgroup = ceph 2022-09-09T22:25:30.402 INFO:tasks.ceph:[client] setuser = ceph 2022-09-09T22:25:30.403 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2022-09-09T22:25:30.403 INFO:tasks.ceph:[mgr] debug mgr = 20 2022-09-09T22:25:30.403 INFO:tasks.ceph:[mgr] debug ms = 1 2022-09-09T22:25:30.404 INFO:tasks.ceph:[mon] debug mon = 20 2022-09-09T22:25:30.404 INFO:tasks.ceph:[mon] debug ms = 1 2022-09-09T22:25:30.404 INFO:tasks.ceph:[mon] debug paxos = 20 2022-09-09T22:25:30.405 INFO:tasks.ceph:[osd] debug ms = 1 2022-09-09T22:25:30.405 INFO:tasks.ceph:[osd] debug osd = 20 2022-09-09T22:25:30.405 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2022-09-09T22:25:30.405 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2022-09-09T22:25:30.406 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2022-09-09T22:25:30.406 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2022-09-09T22:25:30.407 INFO:tasks.ceph:Setting up c1.mon.a... 2022-09-09T22:25:30.407 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2022-09-09T22:25:30.449 INFO:teuthology.orchestra.run.smithi005.stdout:creating /etc/ceph/c1.keyring 2022-09-09T22:25:30.459 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2022-09-09T22:25:30.545 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0644 /etc/ceph/c1.keyring 2022-09-09T22:25:30.562 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.5')] 2022-09-09T22:25:30.563 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.5'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2022-09-09T22:25:30.565 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-09-09T22:25:30.565 DEBUG:teuthology.orchestra.run.smithi005:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2022-09-09T22:25:30.572 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.5 --print /home/ubuntu/cephtest/c1.monmap 2022-09-09T22:25:30.652 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:25:30.653 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:25:30.655 INFO:teuthology.orchestra.run.smithi005.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2022-09-09T22:25:30.656 INFO:teuthology.orchestra.run.smithi005.stdout:monmaptool: generated fsid 5b208e58-c41f-477d-87e6-917d46e40055 2022-09-09T22:25:30.656 INFO:teuthology.orchestra.run.smithi005.stdout:setting min_mon_release = octopus 2022-09-09T22:25:30.656 INFO:teuthology.orchestra.run.smithi005.stdout:epoch 0 2022-09-09T22:25:30.657 INFO:teuthology.orchestra.run.smithi005.stdout:fsid 5b208e58-c41f-477d-87e6-917d46e40055 2022-09-09T22:25:30.657 INFO:teuthology.orchestra.run.smithi005.stdout:last_changed 2022-09-09T22:25:30.650665+0000 2022-09-09T22:25:30.658 INFO:teuthology.orchestra.run.smithi005.stdout:created 2022-09-09T22:25:30.650665+0000 2022-09-09T22:25:30.658 INFO:teuthology.orchestra.run.smithi005.stdout:min_mon_release 15 (octopus) 2022-09-09T22:25:30.658 INFO:teuthology.orchestra.run.smithi005.stdout:election_strategy: 1 2022-09-09T22:25:30.659 INFO:teuthology.orchestra.run.smithi005.stdout:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2022-09-09T22:25:30.659 INFO:teuthology.orchestra.run.smithi005.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2022-09-09T22:25:30.660 DEBUG:teuthology.orchestra.run.smithi005:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2022-09-09T22:25:30.671 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 5b208e58-c41f-477d-87e6-917d46e40055... 2022-09-09T22:25:30.673 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2022-09-09T22:25:30.715 DEBUG:teuthology.orchestra.run.smithi080:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2022-09-09T22:25:30.751 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2022-09-09T22:25:30.752 INFO:teuthology.orchestra.run.smithi005.stdout: chdir = "" 2022-09-09T22:25:30.752 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-09-09T22:25:30.753 INFO:teuthology.orchestra.run.smithi005.stdout: auth supported = cephx 2022-09-09T22:25:30.753 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:30.754 INFO:teuthology.orchestra.run.smithi005.stdout: filestore xattr use omap = true 2022-09-09T22:25:30.754 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:30.754 INFO:teuthology.orchestra.run.smithi005.stdout: mon clock drift allowed = 1.000 2022-09-09T22:25:30.755 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:30.755 INFO:teuthology.orchestra.run.smithi005.stdout: osd crush chooseleaf type = 0 2022-09-09T22:25:30.756 INFO:teuthology.orchestra.run.smithi005.stdout: auth debug = true 2022-09-09T22:25:30.756 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:30.756 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on old message = true 2022-09-09T22:25:30.757 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on bug = true 2022-09-09T22:25:30.757 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:30.757 INFO:teuthology.orchestra.run.smithi005.stdout: mon max pg per osd = 10000 # >= luminous 2022-09-09T22:25:30.757 INFO:teuthology.orchestra.run.smithi005.stdout: mon pg warn max object skew = 0 2022-09-09T22:25:30.757 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:30.758 INFO:teuthology.orchestra.run.smithi005.stdout: # disable pg_autoscaler by default for new pools 2022-09-09T22:25:30.758 INFO:teuthology.orchestra.run.smithi005.stdout: osd_pool_default_pg_autoscale_mode = off 2022-09-09T22:25:30.758 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:30.758 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default size = 2 2022-09-09T22:25:30.759 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:30.759 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow primary affinity = true 2022-09-09T22:25:30.760 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow pg remap = true 2022-09-09T22:25:30.760 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on legacy crush tunables = false 2022-09-09T22:25:30.760 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on crush straw calc version zero = false 2022-09-09T22:25:30.760 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on no sortbitwise = false 2022-09-09T22:25:30.761 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on osd down out interval zero = false 2022-09-09T22:25:30.761 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on too few osds = false 2022-09-09T22:25:30.761 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-09-09T22:25:30.761 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_no_redundancy = false 2022-09-09T22:25:30.761 INFO:teuthology.orchestra.run.smithi005.stdout: mon_allow_pool_size_one = true 2022-09-09T22:25:30.762 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:30.762 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-09-09T22:25:30.762 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:30.762 INFO:teuthology.orchestra.run.smithi005.stdout: osd default data pool replay window = 5 2022-09-09T22:25:30.763 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:30.763 INFO:teuthology.orchestra.run.smithi005.stdout: mon allow pool delete = true 2022-09-09T22:25:30.763 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:30.763 INFO:teuthology.orchestra.run.smithi005.stdout: mon cluster log file level = debug 2022-09-09T22:25:30.764 INFO:teuthology.orchestra.run.smithi005.stdout: debug asserts on shutdown = true 2022-09-09T22:25:30.764 INFO:teuthology.orchestra.run.smithi005.stdout: mon health detail to clog = false 2022-09-09T22:25:30.764 INFO:teuthology.orchestra.run.smithi005.stdout: mon host = 172.21.15.5 2022-09-09T22:25:30.765 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = 5b208e58-c41f-477d-87e6-917d46e40055 2022-09-09T22:25:30.765 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:30.766 INFO:teuthology.orchestra.run.smithi005.stdout:[osd] 2022-09-09T22:25:30.766 INFO:teuthology.orchestra.run.smithi005.stdout: osd journal size = 100 2022-09-09T22:25:30.766 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:30.766 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub load threshold = 5.0 2022-09-09T22:25:30.767 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub max interval = 600 2022-09-09T22:25:30.767 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:30.767 INFO:teuthology.orchestra.run.smithi005.stdout: osd recover clone overlap = true 2022-09-09T22:25:30.767 INFO:teuthology.orchestra.run.smithi005.stdout: osd recovery max chunk = 1048576 2022-09-09T22:25:30.768 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:30.768 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug shutdown = true 2022-09-09T22:25:30.768 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug op order = true 2022-09-09T22:25:30.768 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify stray on activate = true 2022-09-09T22:25:30.769 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:30.769 INFO:teuthology.orchestra.run.smithi005.stdout: osd open classes on start = true 2022-09-09T22:25:30.770 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug pg log writeout = true 2022-09-09T22:25:30.770 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:30.770 INFO:teuthology.orchestra.run.smithi005.stdout: osd deep scrub update digest min age = 30 2022-09-09T22:25:30.771 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:30.771 INFO:teuthology.orchestra.run.smithi005.stdout: osd map max advance = 10 2022-09-09T22:25:30.771 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:30.771 INFO:teuthology.orchestra.run.smithi005.stdout: journal zero on create = true 2022-09-09T22:25:30.772 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:30.772 INFO:teuthology.orchestra.run.smithi005.stdout: filestore ondisk finisher threads = 3 2022-09-09T22:25:30.772 INFO:teuthology.orchestra.run.smithi005.stdout: filestore apply finisher threads = 3 2022-09-09T22:25:30.772 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:30.773 INFO:teuthology.orchestra.run.smithi005.stdout: bdev debug aio = true 2022-09-09T22:25:30.773 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug misdirected ops = true 2022-09-09T22:25:30.773 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2022-09-09T22:25:30.773 INFO:teuthology.orchestra.run.smithi005.stdout: debug osd = 20 2022-09-09T22:25:30.773 INFO:teuthology.orchestra.run.smithi005.stdout: rocksdb delete range threshold = 0 2022-09-09T22:25:30.774 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:30.774 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr] 2022-09-09T22:25:30.774 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2022-09-09T22:25:30.774 INFO:teuthology.orchestra.run.smithi005.stdout: debug mgr = 20 2022-09-09T22:25:30.775 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2022-09-09T22:25:30.775 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2022-09-09T22:25:30.775 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2022-09-09T22:25:30.775 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2022-09-09T22:25:30.776 INFO:teuthology.orchestra.run.smithi005.stdout: mgr/telemetry/nag = false 2022-09-09T22:25:30.776 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:30.776 INFO:teuthology.orchestra.run.smithi005.stdout:[mon] 2022-09-09T22:25:30.776 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2022-09-09T22:25:30.777 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2022-09-09T22:25:30.777 INFO:teuthology.orchestra.run.smithi005.stdout: debug paxos = 20 2022-09-09T22:25:30.777 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2022-09-09T22:25:30.777 INFO:teuthology.orchestra.run.smithi005.stdout: mon data avail warn = 5 2022-09-09T22:25:30.778 INFO:teuthology.orchestra.run.smithi005.stdout: mon mgr mkfs grace = 240 2022-09-09T22:25:30.778 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2022-09-09T22:25:30.778 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd reporter subtree level = osd 2022-09-09T22:25:30.778 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd prime pg temp = true 2022-09-09T22:25:30.779 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2022-09-09T22:25:30.779 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:30.779 INFO:teuthology.orchestra.run.smithi005.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-09-09T22:25:30.779 INFO:teuthology.orchestra.run.smithi005.stdout: auth mon ticket ttl = 660 # 11m 2022-09-09T22:25:30.779 INFO:teuthology.orchestra.run.smithi005.stdout: auth service ticket ttl = 240 # 4m 2022-09-09T22:25:30.780 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:30.780 INFO:teuthology.orchestra.run.smithi005.stdout: # don't complain about insecure global_id in the test suite 2022-09-09T22:25:30.780 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-09-09T22:25:30.780 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-09-09T22:25:30.781 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:30.781 INFO:teuthology.orchestra.run.smithi005.stdout: # 1m isn't quite enough 2022-09-09T22:25:30.781 INFO:teuthology.orchestra.run.smithi005.stdout: mon_down_mkfs_grace = 2m 2022-09-09T22:25:30.781 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:30.782 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_filestore_osds = false 2022-09-09T22:25:30.782 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:30.782 INFO:teuthology.orchestra.run.smithi005.stdout:[client] 2022-09-09T22:25:30.782 INFO:teuthology.orchestra.run.smithi005.stdout: rgw cache enabled = true 2022-09-09T22:25:30.783 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable ops log = true 2022-09-09T22:25:30.783 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable usage log = true 2022-09-09T22:25:30.783 INFO:teuthology.orchestra.run.smithi005.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-09-09T22:25:30.783 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-09-09T22:25:30.784 INFO:teuthology.orchestra.run.smithi005.stdout: debug rgw = 20 2022-09-09T22:25:30.784 INFO:teuthology.orchestra.run.smithi005.stdout: debug rgw sync = 20 2022-09-09T22:25:30.784 INFO:teuthology.orchestra.run.smithi005.stdout: rgw crypt require ssl = False 2022-09-09T22:25:30.784 INFO:teuthology.orchestra.run.smithi005.stdout: rgw crypt s3 kms backend = testing 2022-09-09T22:25:30.784 INFO:teuthology.orchestra.run.smithi005.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-09-09T22:25:30.785 INFO:teuthology.orchestra.run.smithi005.stdout: rgw curl low speed time = 300 2022-09-09T22:25:30.785 INFO:teuthology.orchestra.run.smithi005.stdout: rgw data log num shards = 4 2022-09-09T22:25:30.785 INFO:teuthology.orchestra.run.smithi005.stdout: rgw md log max shards = 4 2022-09-09T22:25:30.785 INFO:teuthology.orchestra.run.smithi005.stdout: rgw sync data inject err probability = 0.1 2022-09-09T22:25:30.786 INFO:teuthology.orchestra.run.smithi005.stdout: rgw sync log trim interval = 0 2022-09-09T22:25:30.786 INFO:teuthology.orchestra.run.smithi005.stdout: rgw sync meta inject err probability = 0.1 2022-09-09T22:25:30.786 INFO:teuthology.orchestra.run.smithi005.stdout: rgw sync obj etag verify = True 2022-09-09T22:25:30.786 INFO:teuthology.orchestra.run.smithi005.stdout: setgroup = ceph 2022-09-09T22:25:30.787 INFO:teuthology.orchestra.run.smithi005.stdout: setuser = ceph 2022-09-09T22:25:30.787 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.a] 2022-09-09T22:25:30.787 INFO:teuthology.orchestra.run.smithi005.stdout:[client.0] 2022-09-09T22:25:30.787 INFO:teuthology.orchestra.run.smithi005.stdout: rgw data notify interval msec = 0 2022-09-09T22:25:30.788 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.0] 2022-09-09T22:25:30.788 INFO:teuthology.orchestra.run.smithi005.stdout: osd_max_omap_entries_per_request = 10 2022-09-09T22:25:30.788 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.1] 2022-09-09T22:25:30.788 INFO:teuthology.orchestra.run.smithi005.stdout: osd_max_omap_entries_per_request = 1000 2022-09-09T22:25:30.788 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.2] 2022-09-09T22:25:30.789 INFO:teuthology.orchestra.run.smithi005.stdout: osd_max_omap_entries_per_request = 10000 2022-09-09T22:25:30.789 INFO:teuthology.orchestra.run.smithi080.stdout:[global] 2022-09-09T22:25:30.789 INFO:teuthology.orchestra.run.smithi080.stdout: chdir = "" 2022-09-09T22:25:30.790 INFO:teuthology.orchestra.run.smithi080.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-09-09T22:25:30.790 INFO:teuthology.orchestra.run.smithi080.stdout: auth supported = cephx 2022-09-09T22:25:30.790 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:25:30.790 INFO:teuthology.orchestra.run.smithi080.stdout: filestore xattr use omap = true 2022-09-09T22:25:30.791 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:25:30.791 INFO:teuthology.orchestra.run.smithi080.stdout: mon clock drift allowed = 1.000 2022-09-09T22:25:30.791 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:25:30.791 INFO:teuthology.orchestra.run.smithi080.stdout: osd crush chooseleaf type = 0 2022-09-09T22:25:30.791 INFO:teuthology.orchestra.run.smithi080.stdout: auth debug = true 2022-09-09T22:25:30.792 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:25:30.792 INFO:teuthology.orchestra.run.smithi080.stdout: ms die on old message = true 2022-09-09T22:25:30.792 INFO:teuthology.orchestra.run.smithi080.stdout: ms die on bug = true 2022-09-09T22:25:30.792 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:25:30.793 INFO:teuthology.orchestra.run.smithi080.stdout: mon max pg per osd = 10000 # >= luminous 2022-09-09T22:25:30.793 INFO:teuthology.orchestra.run.smithi080.stdout: mon pg warn max object skew = 0 2022-09-09T22:25:30.793 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:25:30.793 INFO:teuthology.orchestra.run.smithi080.stdout: # disable pg_autoscaler by default for new pools 2022-09-09T22:25:30.794 INFO:teuthology.orchestra.run.smithi080.stdout: osd_pool_default_pg_autoscale_mode = off 2022-09-09T22:25:30.794 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:25:30.794 INFO:teuthology.orchestra.run.smithi080.stdout: osd pool default size = 2 2022-09-09T22:25:30.794 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:25:30.795 INFO:teuthology.orchestra.run.smithi080.stdout: mon osd allow primary affinity = true 2022-09-09T22:25:30.795 INFO:teuthology.orchestra.run.smithi080.stdout: mon osd allow pg remap = true 2022-09-09T22:25:30.795 INFO:teuthology.orchestra.run.smithi080.stdout: mon warn on legacy crush tunables = false 2022-09-09T22:25:30.795 INFO:teuthology.orchestra.run.smithi080.stdout: mon warn on crush straw calc version zero = false 2022-09-09T22:25:30.795 INFO:teuthology.orchestra.run.smithi080.stdout: mon warn on no sortbitwise = false 2022-09-09T22:25:30.796 INFO:teuthology.orchestra.run.smithi080.stdout: mon warn on osd down out interval zero = false 2022-09-09T22:25:30.796 INFO:teuthology.orchestra.run.smithi080.stdout: mon warn on too few osds = false 2022-09-09T22:25:30.796 INFO:teuthology.orchestra.run.smithi080.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-09-09T22:25:30.796 INFO:teuthology.orchestra.run.smithi080.stdout: mon_warn_on_pool_no_redundancy = false 2022-09-09T22:25:30.797 INFO:teuthology.orchestra.run.smithi080.stdout: mon_allow_pool_size_one = true 2022-09-09T22:25:30.797 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:25:30.797 INFO:teuthology.orchestra.run.smithi080.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-09-09T22:25:30.797 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:25:30.798 INFO:teuthology.orchestra.run.smithi080.stdout: osd default data pool replay window = 5 2022-09-09T22:25:30.798 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:25:30.798 INFO:teuthology.orchestra.run.smithi080.stdout: mon allow pool delete = true 2022-09-09T22:25:30.798 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:25:30.799 INFO:teuthology.orchestra.run.smithi080.stdout: mon cluster log file level = debug 2022-09-09T22:25:30.799 INFO:teuthology.orchestra.run.smithi080.stdout: debug asserts on shutdown = true 2022-09-09T22:25:30.799 INFO:teuthology.orchestra.run.smithi080.stdout: mon health detail to clog = false 2022-09-09T22:25:30.799 INFO:teuthology.orchestra.run.smithi080.stdout: mon host = 172.21.15.5 2022-09-09T22:25:30.799 INFO:teuthology.orchestra.run.smithi080.stdout: fsid = 5b208e58-c41f-477d-87e6-917d46e40055 2022-09-09T22:25:30.800 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:25:30.800 INFO:teuthology.orchestra.run.smithi080.stdout:[osd] 2022-09-09T22:25:30.800 INFO:teuthology.orchestra.run.smithi080.stdout: osd journal size = 100 2022-09-09T22:25:30.800 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:25:30.801 INFO:teuthology.orchestra.run.smithi080.stdout: osd scrub load threshold = 5.0 2022-09-09T22:25:30.801 INFO:teuthology.orchestra.run.smithi080.stdout: osd scrub max interval = 600 2022-09-09T22:25:30.801 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:25:30.801 INFO:teuthology.orchestra.run.smithi080.stdout: osd recover clone overlap = true 2022-09-09T22:25:30.802 INFO:teuthology.orchestra.run.smithi080.stdout: osd recovery max chunk = 1048576 2022-09-09T22:25:30.802 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:25:30.802 INFO:teuthology.orchestra.run.smithi080.stdout: osd debug shutdown = true 2022-09-09T22:25:30.802 INFO:teuthology.orchestra.run.smithi080.stdout: osd debug op order = true 2022-09-09T22:25:30.803 INFO:teuthology.orchestra.run.smithi080.stdout: osd debug verify stray on activate = true 2022-09-09T22:25:30.803 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:25:30.803 INFO:teuthology.orchestra.run.smithi080.stdout: osd open classes on start = true 2022-09-09T22:25:30.803 INFO:teuthology.orchestra.run.smithi080.stdout: osd debug pg log writeout = true 2022-09-09T22:25:30.803 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:25:30.804 INFO:teuthology.orchestra.run.smithi080.stdout: osd deep scrub update digest min age = 30 2022-09-09T22:25:30.804 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:25:30.804 INFO:teuthology.orchestra.run.smithi080.stdout: osd map max advance = 10 2022-09-09T22:25:30.804 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:25:30.805 INFO:teuthology.orchestra.run.smithi080.stdout: journal zero on create = true 2022-09-09T22:25:30.805 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:25:30.805 INFO:teuthology.orchestra.run.smithi080.stdout: filestore ondisk finisher threads = 3 2022-09-09T22:25:30.805 INFO:teuthology.orchestra.run.smithi080.stdout: filestore apply finisher threads = 3 2022-09-09T22:25:30.806 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:25:30.806 INFO:teuthology.orchestra.run.smithi080.stdout: bdev debug aio = true 2022-09-09T22:25:30.806 INFO:teuthology.orchestra.run.smithi080.stdout: osd debug misdirected ops = true 2022-09-09T22:25:30.806 INFO:teuthology.orchestra.run.smithi080.stdout: debug ms = 1 2022-09-09T22:25:30.807 INFO:teuthology.orchestra.run.smithi080.stdout: debug osd = 20 2022-09-09T22:25:30.807 INFO:teuthology.orchestra.run.smithi080.stdout: rocksdb delete range threshold = 0 2022-09-09T22:25:30.807 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:25:30.807 INFO:teuthology.orchestra.run.smithi080.stdout:[mgr] 2022-09-09T22:25:30.807 INFO:teuthology.orchestra.run.smithi080.stdout: debug ms = 1 2022-09-09T22:25:30.808 INFO:teuthology.orchestra.run.smithi080.stdout: debug mgr = 20 2022-09-09T22:25:30.808 INFO:teuthology.orchestra.run.smithi080.stdout: debug mon = 20 2022-09-09T22:25:30.808 INFO:teuthology.orchestra.run.smithi080.stdout: debug auth = 20 2022-09-09T22:25:30.808 INFO:teuthology.orchestra.run.smithi080.stdout: mon reweight min pgs per osd = 4 2022-09-09T22:25:30.809 INFO:teuthology.orchestra.run.smithi080.stdout: mon reweight min bytes per osd = 10 2022-09-09T22:25:30.809 INFO:teuthology.orchestra.run.smithi080.stdout: mgr/telemetry/nag = false 2022-09-09T22:25:30.809 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:25:30.809 INFO:teuthology.orchestra.run.smithi080.stdout:[mon] 2022-09-09T22:25:30.809 INFO:teuthology.orchestra.run.smithi080.stdout: debug ms = 1 2022-09-09T22:25:30.810 INFO:teuthology.orchestra.run.smithi080.stdout: debug mon = 20 2022-09-09T22:25:30.810 INFO:teuthology.orchestra.run.smithi080.stdout: debug paxos = 20 2022-09-09T22:25:30.810 INFO:teuthology.orchestra.run.smithi080.stdout: debug auth = 20 2022-09-09T22:25:30.811 INFO:teuthology.orchestra.run.smithi080.stdout: mon data avail warn = 5 2022-09-09T22:25:30.811 INFO:teuthology.orchestra.run.smithi080.stdout: mon mgr mkfs grace = 240 2022-09-09T22:25:30.811 INFO:teuthology.orchestra.run.smithi080.stdout: mon reweight min pgs per osd = 4 2022-09-09T22:25:30.811 INFO:teuthology.orchestra.run.smithi080.stdout: mon osd reporter subtree level = osd 2022-09-09T22:25:30.811 INFO:teuthology.orchestra.run.smithi080.stdout: mon osd prime pg temp = true 2022-09-09T22:25:30.812 INFO:teuthology.orchestra.run.smithi080.stdout: mon reweight min bytes per osd = 10 2022-09-09T22:25:30.812 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:25:30.812 INFO:teuthology.orchestra.run.smithi080.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-09-09T22:25:30.812 INFO:teuthology.orchestra.run.smithi080.stdout: auth mon ticket ttl = 660 # 11m 2022-09-09T22:25:30.813 INFO:teuthology.orchestra.run.smithi080.stdout: auth service ticket ttl = 240 # 4m 2022-09-09T22:25:30.813 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:25:30.813 INFO:teuthology.orchestra.run.smithi080.stdout: # don't complain about insecure global_id in the test suite 2022-09-09T22:25:30.813 INFO:teuthology.orchestra.run.smithi080.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-09-09T22:25:30.813 INFO:teuthology.orchestra.run.smithi080.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-09-09T22:25:30.814 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:25:30.814 INFO:teuthology.orchestra.run.smithi080.stdout: # 1m isn't quite enough 2022-09-09T22:25:30.814 INFO:teuthology.orchestra.run.smithi080.stdout: mon_down_mkfs_grace = 2m 2022-09-09T22:25:30.814 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:25:30.815 INFO:teuthology.orchestra.run.smithi080.stdout: mon_warn_on_filestore_osds = false 2022-09-09T22:25:30.815 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:25:30.815 INFO:teuthology.orchestra.run.smithi080.stdout:[client] 2022-09-09T22:25:30.815 INFO:teuthology.orchestra.run.smithi080.stdout: rgw cache enabled = true 2022-09-09T22:25:30.816 INFO:teuthology.orchestra.run.smithi080.stdout: rgw enable ops log = true 2022-09-09T22:25:30.816 INFO:teuthology.orchestra.run.smithi080.stdout: rgw enable usage log = true 2022-09-09T22:25:30.816 INFO:teuthology.orchestra.run.smithi080.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-09-09T22:25:30.816 INFO:teuthology.orchestra.run.smithi080.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-09-09T22:25:30.817 INFO:teuthology.orchestra.run.smithi080.stdout: debug rgw = 20 2022-09-09T22:25:30.817 INFO:teuthology.orchestra.run.smithi080.stdout: debug rgw sync = 20 2022-09-09T22:25:30.817 INFO:teuthology.orchestra.run.smithi080.stdout: rgw crypt require ssl = False 2022-09-09T22:25:30.817 INFO:teuthology.orchestra.run.smithi080.stdout: rgw crypt s3 kms backend = testing 2022-09-09T22:25:30.817 INFO:teuthology.orchestra.run.smithi080.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-09-09T22:25:30.818 INFO:teuthology.orchestra.run.smithi080.stdout: rgw curl low speed time = 300 2022-09-09T22:25:30.818 INFO:teuthology.orchestra.run.smithi080.stdout: rgw data log num shards = 4 2022-09-09T22:25:30.818 INFO:teuthology.orchestra.run.smithi080.stdout: rgw md log max shards = 4 2022-09-09T22:25:30.818 INFO:teuthology.orchestra.run.smithi080.stdout: rgw sync data inject err probability = 0.1 2022-09-09T22:25:30.819 INFO:teuthology.orchestra.run.smithi080.stdout: rgw sync log trim interval = 0 2022-09-09T22:25:30.819 INFO:teuthology.orchestra.run.smithi080.stdout: rgw sync meta inject err probability = 0.1 2022-09-09T22:25:30.819 INFO:teuthology.orchestra.run.smithi080.stdout: rgw sync obj etag verify = True 2022-09-09T22:25:30.819 INFO:teuthology.orchestra.run.smithi080.stdout: setgroup = ceph 2022-09-09T22:25:30.820 INFO:teuthology.orchestra.run.smithi080.stdout: setuser = ceph 2022-09-09T22:25:30.820 INFO:teuthology.orchestra.run.smithi080.stdout:[mon.a] 2022-09-09T22:25:30.820 INFO:teuthology.orchestra.run.smithi080.stdout:[client.0] 2022-09-09T22:25:30.820 INFO:teuthology.orchestra.run.smithi080.stdout: rgw data notify interval msec = 0 2022-09-09T22:25:30.820 INFO:teuthology.orchestra.run.smithi080.stdout:[osd.0] 2022-09-09T22:25:30.821 INFO:teuthology.orchestra.run.smithi080.stdout: osd_max_omap_entries_per_request = 10 2022-09-09T22:25:30.821 INFO:teuthology.orchestra.run.smithi080.stdout:[osd.1] 2022-09-09T22:25:30.821 INFO:teuthology.orchestra.run.smithi080.stdout: osd_max_omap_entries_per_request = 1000 2022-09-09T22:25:30.821 INFO:teuthology.orchestra.run.smithi080.stdout:[osd.2] 2022-09-09T22:25:30.822 INFO:teuthology.orchestra.run.smithi080.stdout: osd_max_omap_entries_per_request = 10000 2022-09-09T22:25:30.823 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2022-09-09T22:25:30.824 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2022-09-09T22:25:30.882 INFO:tasks.ceph:Copying monmap to all nodes... 2022-09-09T22:25:30.883 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-09-09T22:25:30.883 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2022-09-09T22:25:30.892 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-09-09T22:25:30.892 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2022-09-09T22:25:30.905 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi005.front.sepia.ceph.com 2022-09-09T22:25:30.906 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-09-09T22:25:30.906 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/ceph/c1.keyring 2022-09-09T22:25:30.907 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0644 /etc/ceph/c1.keyring 2022-09-09T22:25:30.935 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-09-09T22:25:30.935 DEBUG:teuthology.orchestra.run.smithi005:> dd of=/home/ubuntu/cephtest/c1.monmap 2022-09-09T22:25:30.984 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi080.front.sepia.ceph.com 2022-09-09T22:25:30.985 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2022-09-09T22:25:30.985 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/ceph/c1.keyring 2022-09-09T22:25:30.985 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod 0644 /etc/ceph/c1.keyring 2022-09-09T22:25:31.013 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2022-09-09T22:25:31.014 DEBUG:teuthology.orchestra.run.smithi080:> dd of=/home/ubuntu/cephtest/c1.monmap 2022-09-09T22:25:31.022 INFO:tasks.ceph:Setting up mon nodes... 2022-09-09T22:25:31.023 INFO:tasks.ceph:Setting up mgr nodes... 2022-09-09T22:25:31.024 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2022-09-09T22:25:31.075 INFO:teuthology.orchestra.run.smithi005.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2022-09-09T22:25:31.078 INFO:tasks.ceph:Setting up mds nodes... 2022-09-09T22:25:31.079 INFO:tasks.ceph_client:Setting up client nodes... 2022-09-09T22:25:31.079 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2022-09-09T22:25:31.115 INFO:teuthology.orchestra.run.smithi005.stdout:creating /etc/ceph/c1.client.0.keyring 2022-09-09T22:25:31.128 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2022-09-09T22:25:31.211 INFO:teuthology.orchestra.run.smithi005.stdout:creating /etc/ceph/c1.client.1.keyring 2022-09-09T22:25:31.223 INFO:tasks.ceph:Running mkfs on osd nodes... 2022-09-09T22:25:31.224 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi005.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2022-09-09T22:25:31.224 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2022-09-09T22:25:31.237 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-09T22:25:31.238 INFO:tasks.ceph:role: c1.osd.0 2022-09-09T22:25:31.238 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi005.front.sepia.ceph.com 2022-09-09T22:25:31.239 DEBUG:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-09-09T22:25:31.279 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-09-09T22:25:31.279 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2022-09-09T22:25:31.279 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-09-09T22:25:31.280 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 2022-09-09T22:25:31.280 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-09-09T22:25:31.280 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2022-09-09T22:25:31.280 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-09-09T22:25:31.281 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-09-09T22:25:31.281 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-09-09T22:25:31.281 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-09-09T22:25:31.282 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2022-09-09T22:25:31.283 DEBUG:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2022-09-09T22:25:31.348 DEBUG:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2022-09-09T22:25:31.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-09T22:25:31.362 INFO:teuthology.orchestra.run.smithi005.stderr:sudo: /sbin/restorecon: command not found 2022-09-09T22:25:31.363 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2022-09-09T22:25:31.373 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-09T22:25:31.374 INFO:tasks.ceph:role: c1.osd.1 2022-09-09T22:25:31.374 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi005.front.sepia.ceph.com 2022-09-09T22:25:31.374 DEBUG:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-09-09T22:25:31.416 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-09-09T22:25:31.417 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2022-09-09T22:25:31.417 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-09-09T22:25:31.417 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 2022-09-09T22:25:31.417 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-09-09T22:25:31.418 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2022-09-09T22:25:31.418 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-09-09T22:25:31.418 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-09-09T22:25:31.418 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-09-09T22:25:31.419 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-09-09T22:25:31.420 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2022-09-09T22:25:31.420 DEBUG:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2022-09-09T22:25:31.483 DEBUG:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2022-09-09T22:25:31.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-09T22:25:31.497 INFO:teuthology.orchestra.run.smithi005.stderr:sudo: /sbin/restorecon: command not found 2022-09-09T22:25:31.497 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2022-09-09T22:25:31.509 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-09T22:25:31.509 INFO:tasks.ceph:role: c1.osd.2 2022-09-09T22:25:31.509 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi005.front.sepia.ceph.com 2022-09-09T22:25:31.510 DEBUG:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-09-09T22:25:31.601 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-09-09T22:25:31.601 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2022-09-09T22:25:31.602 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-09-09T22:25:31.602 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 2022-09-09T22:25:31.603 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-09-09T22:25:31.603 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2022-09-09T22:25:31.603 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-09-09T22:25:31.604 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-09-09T22:25:31.604 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-09-09T22:25:31.605 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-09-09T22:25:31.605 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2022-09-09T22:25:31.606 DEBUG:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2022-09-09T22:25:31.628 DEBUG:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2022-09-09T22:25:31.637 INFO:teuthology.orchestra.run.smithi005.stderr:sudo: /sbin/restorecon: command not found 2022-09-09T22:25:31.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-09T22:25:31.639 DEBUG:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2022-09-09T22:25:31.753 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:25:31.746+0000 7fafa8dcc4c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2022-09-09T22:25:31.754 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:25:31.746+0000 7fafa8dcc4c0 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2022-09-09T22:25:31.754 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:25:31.746+0000 7fafa8dcc4c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2022-09-09T22:25:31.754 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:25:31.746+0000 7fafa8dcc4c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2022-09-09T22:25:31.755 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:25:31.746+0000 7fafa8dcc4c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2022-09-09T22:25:31.755 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:25:31.750+0000 7fafa8dcc4c0 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2022-09-09T22:25:33.945 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2022-09-09T22:25:33.960 DEBUG:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2022-09-09T22:25:34.048 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:25:34.042+0000 7f2112f424c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2022-09-09T22:25:34.048 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:25:34.042+0000 7f2112f424c0 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2022-09-09T22:25:34.049 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:25:34.042+0000 7f2112f424c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2022-09-09T22:25:34.049 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:25:34.042+0000 7f2112f424c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2022-09-09T22:25:34.049 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:25:34.042+0000 7f2112f424c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2022-09-09T22:25:34.050 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:25:34.042+0000 7f2112f424c0 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2022-09-09T22:25:36.256 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2022-09-09T22:25:36.274 DEBUG:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2022-09-09T22:25:36.368 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:25:36.362+0000 7f362a90a4c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2022-09-09T22:25:36.369 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:25:36.362+0000 7f362a90a4c0 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2022-09-09T22:25:36.369 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:25:36.362+0000 7f362a90a4c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2022-09-09T22:25:36.370 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:25:36.362+0000 7f362a90a4c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2022-09-09T22:25:36.370 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:25:36.362+0000 7f362a90a4c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2022-09-09T22:25:36.370 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:25:36.362+0000 7f362a90a4c0 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2022-09-09T22:25:38.560 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2022-09-09T22:25:38.578 INFO:tasks.ceph:Reading keys from all nodes... 2022-09-09T22:25:38.579 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-09-09T22:25:38.579 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2022-09-09T22:25:38.637 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-09-09T22:25:38.637 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2022-09-09T22:25:38.648 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-09-09T22:25:38.648 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2022-09-09T22:25:38.662 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-09-09T22:25:38.663 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2022-09-09T22:25:38.720 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-09-09T22:25:38.721 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2022-09-09T22:25:38.729 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-09-09T22:25:38.729 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2022-09-09T22:25:38.739 INFO:tasks.ceph:Adding keys to all mons... 2022-09-09T22:25:38.739 DEBUG:teuthology.orchestra.run.smithi005:> sudo tee -a /etc/ceph/c1.keyring 2022-09-09T22:25:38.794 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr.x] 2022-09-09T22:25:38.794 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQBbvRtjrjOGBBAAKEjtp+FKO+S0GQL/NPkrUQ== 2022-09-09T22:25:38.795 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.0] 2022-09-09T22:25:38.795 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQBbvRtjrh3hLBAAAtUKnhMNxBTy9Iv8lb+dpA== 2022-09-09T22:25:38.795 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.1] 2022-09-09T22:25:38.795 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQBevRtjZLzaAhAAyg0h/BCtYUseQWPPi5BvMQ== 2022-09-09T22:25:38.796 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.2] 2022-09-09T22:25:38.796 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQBgvRtj01vbFRAA/UX+uynZ3PO5/wuoPXltmA== 2022-09-09T22:25:38.796 INFO:teuthology.orchestra.run.smithi005.stdout:[client.0] 2022-09-09T22:25:38.796 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQBbvRtjldzpBhAAx3ed+wkMycFhFm2Pf7xAPw== 2022-09-09T22:25:38.797 INFO:teuthology.orchestra.run.smithi005.stdout:[client.1] 2022-09-09T22:25:38.797 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQBbvRtjAlybDBAAf27JxKCg+FQh9drGakW0uA== 2022-09-09T22:25:38.798 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-09-09T22:25:38.865 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-09-09T22:25:38.924 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-09-09T22:25:38.970 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-09-09T22:25:39.061 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-09-09T22:25:39.116 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-09-09T22:25:39.167 INFO:tasks.ceph:Running mkfs on mon nodes... 2022-09-09T22:25:39.167 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2022-09-09T22:25:39.183 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2022-09-09T22:25:39.289 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2022-09-09T22:25:39.303 DEBUG:teuthology.orchestra.run.smithi005:> rm -- /home/ubuntu/cephtest/c1.monmap 2022-09-09T22:25:39.355 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2022-09-09T22:25:39.356 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2022-09-09T22:25:39.357 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2022-09-09T22:25:39.360 INFO:tasks.ceph.c1.mon.a:Started 2022-09-09T22:25:39.361 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2022-09-09T22:25:39.362 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2022-09-09T22:25:39.363 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2022-09-09T22:25:39.367 INFO:tasks.ceph.c1.mgr.x:Started 2022-09-09T22:25:39.368 INFO:tasks.ceph:Setting crush tunables to default 2022-09-09T22:25:39.369 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster c1 osd crush tunables default 2022-09-09T22:25:39.620 INFO:teuthology.orchestra.run.smithi005.stderr:adjusted tunables profile to default 2022-09-09T22:25:39.632 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2022-09-09T22:25:39.633 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-09-09T22:25:39.633 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2022-09-09T22:25:39.644 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-09-09T22:25:39.645 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2022-09-09T22:25:39.655 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-09-09T22:25:39.656 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2022-09-09T22:25:39.724 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster c1 osd new 0bb9dc41-555d-4aff-8b41-9cdc2ad5a531 0 2022-09-09T22:25:39.937 INFO:teuthology.orchestra.run.smithi005.stdout:0 2022-09-09T22:25:39.952 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster c1 osd new cfccd431-020e-401d-9857-7dcb485564f1 1 2022-09-09T22:25:40.053 INFO:tasks.ceph.c1.mgr.x.smithi005.stderr:2022-09-09T22:25:40.050+0000 7eff2572b140 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-09-09T22:25:40.137 INFO:tasks.ceph.c1.mgr.x.smithi005.stderr:2022-09-09T22:25:40.130+0000 7eff2572b140 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-09-09T22:25:40.163 INFO:teuthology.orchestra.run.smithi005.stdout:1 2022-09-09T22:25:40.183 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster c1 osd new 3c8716cf-e7e5-44bf-89b8-bd600b77faca 2 2022-09-09T22:25:40.389 INFO:teuthology.orchestra.run.smithi005.stdout:2 2022-09-09T22:25:40.404 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2022-09-09T22:25:40.404 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2022-09-09T22:25:40.408 INFO:tasks.ceph.c1.osd.0:Started 2022-09-09T22:25:40.409 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2022-09-09T22:25:40.409 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2022-09-09T22:25:40.417 INFO:tasks.ceph.c1.osd.1:Started 2022-09-09T22:25:40.417 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2022-09-09T22:25:40.418 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2022-09-09T22:25:40.421 INFO:tasks.ceph.c1.osd.2:Started 2022-09-09T22:25:40.423 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2022-09-09T22:25:40.601 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:40.601 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":5,"fsid":"5b208e58-c41f-477d-87e6-917d46e40055","created":"2022-09-09T22:25:39.512569+0000","modified":"2022-09-09T22:25:40.385756+0000","last_up_change":"0.000000","last_in_change":"2022-09-09T22:25:40.385756+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"0bb9dc41-555d-4aff-8b41-9cdc2ad5a531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cfccd431-020e-401d-9857-7dcb485564f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3c8716cf-e7e5-44bf-89b8-bd600b77faca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-09T22:25:40.604 INFO:tasks.ceph.c1.mgr.x.smithi005.stderr:2022-09-09T22:25:40.598+0000 7eff2572b140 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-09-09T22:25:40.617 INFO:tasks.ceph.ceph_manager.c1:[] 2022-09-09T22:25:40.618 INFO:tasks.ceph:Waiting for OSDs to come up 2022-09-09T22:25:40.664 INFO:tasks.ceph.c1.mgr.x.smithi005.stderr:2022-09-09T22:25:40.658+0000 7eff2572b140 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-09-09T22:25:40.778 INFO:tasks.ceph.c1.mgr.x.smithi005.stderr:2022-09-09T22:25:40.766+0000 7eff2572b140 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-09-09T22:25:40.910 INFO:tasks.ceph.c1.mgr.x.smithi005.stderr:2022-09-09T22:25:40.906+0000 7eff2572b140 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-09-09T22:25:40.929 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2022-09-09T22:25:40.974 INFO:tasks.ceph.c1.mgr.x.smithi005.stderr:2022-09-09T22:25:40.966+0000 7eff2572b140 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-09-09T22:25:41.043 INFO:tasks.ceph.c1.osd.2.smithi005.stderr:2022-09-09T22:25:41.038+0000 7f1003a614c0 -1 Falling back to public interface 2022-09-09T22:25:41.045 INFO:tasks.ceph.c1.osd.1.smithi005.stderr:2022-09-09T22:25:41.042+0000 7fc257c984c0 -1 Falling back to public interface 2022-09-09T22:25:41.058 INFO:tasks.ceph.c1.osd.0.smithi005.stderr:2022-09-09T22:25:41.054+0000 7f1da1ab64c0 -1 Falling back to public interface 2022-09-09T22:25:41.114 INFO:teuthology.misc.health.smithi005.stdout: 2022-09-09T22:25:41.115 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":5,"fsid":"5b208e58-c41f-477d-87e6-917d46e40055","created":"2022-09-09T22:25:39.512569+0000","modified":"2022-09-09T22:25:40.385756+0000","last_up_change":"0.000000","last_in_change":"2022-09-09T22:25:40.385756+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"0bb9dc41-555d-4aff-8b41-9cdc2ad5a531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cfccd431-020e-401d-9857-7dcb485564f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3c8716cf-e7e5-44bf-89b8-bd600b77faca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-09T22:25:41.127 DEBUG:teuthology.misc:0 of 3 OSDs are up 2022-09-09T22:25:41.237 INFO:tasks.ceph.c1.mgr.x.smithi005.stderr:2022-09-09T22:25:41.230+0000 7eff2572b140 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-09-09T22:25:41.404 INFO:tasks.ceph.c1.mgr.x.smithi005.stderr:2022-09-09T22:25:41.398+0000 7eff2572b140 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-09-09T22:25:41.461 INFO:tasks.ceph.c1.mgr.x.smithi005.stderr:2022-09-09T22:25:41.454+0000 7eff2572b140 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-09-09T22:25:42.102 INFO:tasks.ceph.c1.mgr.x.smithi005.stderr:2022-09-09T22:25:42.094+0000 7eff2572b140 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-09-09T22:25:42.299 INFO:tasks.ceph.c1.mgr.x.smithi005.stderr:2022-09-09T22:25:42.294+0000 7eff2572b140 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-09-09T22:25:42.948 INFO:tasks.ceph.c1.mgr.x.smithi005.stderr:2022-09-09T22:25:42.942+0000 7eff2572b140 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-09-09T22:25:43.006 INFO:tasks.ceph.c1.mgr.x.smithi005.stderr:2022-09-09T22:25:43.002+0000 7eff2572b140 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-09-09T22:25:43.084 INFO:tasks.ceph.c1.mgr.x.smithi005.stderr:2022-09-09T22:25:43.078+0000 7eff2572b140 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-09-09T22:25:43.817 INFO:tasks.ceph.c1.mgr.x.smithi005.stderr:2022-09-09T22:25:43.814+0000 7eff2572b140 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-09-09T22:25:43.882 INFO:tasks.ceph.c1.mgr.x.smithi005.stderr:2022-09-09T22:25:43.874+0000 7eff2572b140 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-09-09T22:25:43.945 INFO:tasks.ceph.c1.mgr.x.smithi005.stderr:2022-09-09T22:25:43.938+0000 7eff2572b140 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-09-09T22:25:44.104 INFO:tasks.ceph.c1.osd.1.smithi005.stderr:2022-09-09T22:25:44.098+0000 7fc257c984c0 -1 osd.1 0 log_to_monitors true 2022-09-09T22:25:44.146 INFO:tasks.ceph.c1.osd.0.smithi005.stderr:2022-09-09T22:25:44.138+0000 7f1da1ab64c0 -1 osd.0 0 log_to_monitors true 2022-09-09T22:25:44.158 INFO:tasks.ceph.c1.osd.2.smithi005.stderr:2022-09-09T22:25:44.150+0000 7f1003a614c0 -1 osd.2 0 log_to_monitors true 2022-09-09T22:25:44.249 INFO:tasks.ceph.c1.mgr.x.smithi005.stderr:2022-09-09T22:25:44.242+0000 7eff2572b140 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-09-09T22:25:44.343 INFO:tasks.ceph.c1.mgr.x.smithi005.stderr:2022-09-09T22:25:44.310+0000 7eff2572b140 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-09-09T22:25:44.601 INFO:tasks.ceph.c1.mgr.x.smithi005.stderr:2022-09-09T22:25:44.598+0000 7eff2572b140 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-09-09T22:25:44.721 INFO:tasks.ceph.c1.mgr.x.smithi005.stderr:2022-09-09T22:25:44.714+0000 7eff2572b140 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-09-09T22:25:44.786 INFO:tasks.ceph.c1.mgr.x.smithi005.stderr:2022-09-09T22:25:44.778+0000 7eff2572b140 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-09-09T22:25:45.814 INFO:tasks.ceph.c1.mgr.x.smithi005.stderr:2022-09-09T22:25:45.810+0000 7eff0ce89700 -1 mgr.server handle_report got status from non-daemon mon.a 2022-09-09T22:25:45.879 INFO:tasks.ceph.c1.osd.1.smithi005.stderr:2022-09-09T22:25:45.874+0000 7fc254c5e700 -1 osd.1 0 waiting for initial osdmap 2022-09-09T22:25:45.882 INFO:tasks.ceph.c1.osd.0.smithi005.stderr:2022-09-09T22:25:45.878+0000 7f1d9ea7c700 -1 osd.0 0 waiting for initial osdmap 2022-09-09T22:25:45.883 INFO:tasks.ceph.c1.osd.1.smithi005.stderr:2022-09-09T22:25:45.878+0000 7fc24b70e700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-09T22:25:45.885 INFO:tasks.ceph.c1.osd.0.smithi005.stderr:2022-09-09T22:25:45.882+0000 7f1d9552c700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-09T22:25:46.786 INFO:tasks.ceph.c1.osd.2.smithi005.stderr:2022-09-09T22:25:46.782+0000 7f0fff202700 -1 osd.2 0 waiting for initial osdmap 2022-09-09T22:25:46.793 INFO:tasks.ceph.c1.osd.2.smithi005.stderr:2022-09-09T22:25:46.790+0000 7f0ff74d7700 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-09T22:25:47.434 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2022-09-09T22:25:47.721 INFO:teuthology.misc.health.smithi005.stdout: 2022-09-09T22:25:47.721 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":8,"fsid":"5b208e58-c41f-477d-87e6-917d46e40055","created":"2022-09-09T22:25:39.512569+0000","modified":"2022-09-09T22:25:46.167760+0000","last_up_change":"2022-09-09T22:25:46.167760+0000","last_in_change":"2022-09-09T22:25:40.385756+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"0bb9dc41-555d-4aff-8b41-9cdc2ad5a531","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":14744},{"type":"v1","addr":"172.21.15.5:6817","nonce":14744}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":14744},{"type":"v1","addr":"172.21.15.5:6819","nonce":14744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":14744},{"type":"v1","addr":"172.21.15.5:6823","nonce":14744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":14744},{"type":"v1","addr":"172.21.15.5:6821","nonce":14744}]},"public_addr":"172.21.15.5:6817/14744","cluster_addr":"172.21.15.5:6819/14744","heartbeat_back_addr":"172.21.15.5:6823/14744","heartbeat_front_addr":"172.21.15.5:6821/14744","state":["exists","up"]},{"osd":1,"uuid":"cfccd431-020e-401d-9857-7dcb485564f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":14745},{"type":"v1","addr":"172.21.15.5:6809","nonce":14745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":14745},{"type":"v1","addr":"172.21.15.5:6811","nonce":14745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":14745},{"type":"v1","addr":"172.21.15.5:6815","nonce":14745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":14745},{"type":"v1","addr":"172.21.15.5:6813","nonce":14745}]},"public_addr":"172.21.15.5:6809/14745","cluster_addr":"172.21.15.5:6811/14745","heartbeat_back_addr":"172.21.15.5:6815/14745","heartbeat_front_addr":"172.21.15.5:6813/14745","state":["exists","up"]},{"osd":2,"uuid":"3c8716cf-e7e5-44bf-89b8-bd600b77faca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-09T22:25:47.735 DEBUG:teuthology.misc:2 of 3 OSDs are up 2022-09-09T22:25:54.039 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2022-09-09T22:25:54.329 INFO:teuthology.misc.health.smithi005.stdout: 2022-09-09T22:25:54.330 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":11,"fsid":"5b208e58-c41f-477d-87e6-917d46e40055","created":"2022-09-09T22:25:39.512569+0000","modified":"2022-09-09T22:25:49.799421+0000","last_up_change":"2022-09-09T22:25:47.788726+0000","last_in_change":"2022-09-09T22:25:40.385756+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-09T22:25:46.888928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"0bb9dc41-555d-4aff-8b41-9cdc2ad5a531","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":14744},{"type":"v1","addr":"172.21.15.5:6817","nonce":14744}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":14744},{"type":"v1","addr":"172.21.15.5:6819","nonce":14744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":14744},{"type":"v1","addr":"172.21.15.5:6823","nonce":14744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":14744},{"type":"v1","addr":"172.21.15.5:6821","nonce":14744}]},"public_addr":"172.21.15.5:6817/14744","cluster_addr":"172.21.15.5:6819/14744","heartbeat_back_addr":"172.21.15.5:6823/14744","heartbeat_front_addr":"172.21.15.5:6821/14744","state":["exists","up"]},{"osd":1,"uuid":"cfccd431-020e-401d-9857-7dcb485564f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":14745},{"type":"v1","addr":"172.21.15.5:6809","nonce":14745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":14745},{"type":"v1","addr":"172.21.15.5:6811","nonce":14745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":14745},{"type":"v1","addr":"172.21.15.5:6815","nonce":14745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":14745},{"type":"v1","addr":"172.21.15.5:6813","nonce":14745}]},"public_addr":"172.21.15.5:6809/14745","cluster_addr":"172.21.15.5:6811/14745","heartbeat_back_addr":"172.21.15.5:6815/14745","heartbeat_front_addr":"172.21.15.5:6813/14745","state":["exists","up"]},{"osd":2,"uuid":"3c8716cf-e7e5-44bf-89b8-bd600b77faca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":14743},{"type":"v1","addr":"172.21.15.5:6801","nonce":14743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":14743},{"type":"v1","addr":"172.21.15.5:6803","nonce":14743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":14743},{"type":"v1","addr":"172.21.15.5:6807","nonce":14743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":14743},{"type":"v1","addr":"172.21.15.5:6805","nonce":14743}]},"public_addr":"172.21.15.5:6801/14743","cluster_addr":"172.21.15.5:6803/14743","heartbeat_back_addr":"172.21.15.5:6807/14743","heartbeat_front_addr":"172.21.15.5:6805/14743","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-09T22:25:45.132616+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-09T22:25:45.072484+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-09T22:25:45.139526+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-09T22:25:54.342 DEBUG:teuthology.misc:3 of 3 OSDs are up 2022-09-09T22:25:54.342 INFO:tasks.ceph:Creating RBD pool 2022-09-09T22:25:54.343 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster c1 osd pool create rbd 8 2022-09-09T22:25:55.516 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'rbd' created 2022-09-09T22:25:55.527 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2022-09-09T22:25:56.518 INFO:teuthology.orchestra.run.smithi005.stderr:enabled application 'rbd' on pool 'rbd' 2022-09-09T22:25:56.531 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2022-09-09T22:25:56.532 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2022-09-09T22:25:56.532 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2022-09-09T22:25:56.532 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2022-09-09T22:25:56.533 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2022-09-09T22:25:56.868 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:56.882 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":5,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":14558},{"type":"v1","addr":"172.21.15.5:6825","nonce":14558}]},"active_addr":"172.21.15.5:6825/14558","active_change":"2022-09-09T22:25:44.803469+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":944481897}]},{"addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":2760249198}]},{"addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":2369326667}]},{"addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":3947098912}]}]}} 2022-09-09T22:25:56.892 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2022-09-09T22:25:56.893 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2022-09-09T22:25:56.894 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2022-09-09T22:25:57.186 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:57.187 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":13,"fsid":"5b208e58-c41f-477d-87e6-917d46e40055","created":"2022-09-09T22:25:39.512569+0000","modified":"2022-09-09T22:25:56.501678+0000","last_up_change":"2022-09-09T22:25:47.788726+0000","last_in_change":"2022-09-09T22:25:40.385756+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-09T22:25:46.888928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-09-09T22:25:54.675151+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"0bb9dc41-555d-4aff-8b41-9cdc2ad5a531","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":14744},{"type":"v1","addr":"172.21.15.5:6817","nonce":14744}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":14744},{"type":"v1","addr":"172.21.15.5:6819","nonce":14744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":14744},{"type":"v1","addr":"172.21.15.5:6823","nonce":14744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":14744},{"type":"v1","addr":"172.21.15.5:6821","nonce":14744}]},"public_addr":"172.21.15.5:6817/14744","cluster_addr":"172.21.15.5:6819/14744","heartbeat_back_addr":"172.21.15.5:6823/14744","heartbeat_front_addr":"172.21.15.5:6821/14744","state":["exists","up"]},{"osd":1,"uuid":"cfccd431-020e-401d-9857-7dcb485564f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":14745},{"type":"v1","addr":"172.21.15.5:6809","nonce":14745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":14745},{"type":"v1","addr":"172.21.15.5:6811","nonce":14745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":14745},{"type":"v1","addr":"172.21.15.5:6815","nonce":14745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":14745},{"type":"v1","addr":"172.21.15.5:6813","nonce":14745}]},"public_addr":"172.21.15.5:6809/14745","cluster_addr":"172.21.15.5:6811/14745","heartbeat_back_addr":"172.21.15.5:6815/14745","heartbeat_front_addr":"172.21.15.5:6813/14745","state":["exists","up"]},{"osd":2,"uuid":"3c8716cf-e7e5-44bf-89b8-bd600b77faca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":14743},{"type":"v1","addr":"172.21.15.5:6801","nonce":14743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":14743},{"type":"v1","addr":"172.21.15.5:6803","nonce":14743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":14743},{"type":"v1","addr":"172.21.15.5:6807","nonce":14743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":14743},{"type":"v1","addr":"172.21.15.5:6805","nonce":14743}]},"public_addr":"172.21.15.5:6801/14743","cluster_addr":"172.21.15.5:6803/14743","heartbeat_back_addr":"172.21.15.5:6807/14743","heartbeat_front_addr":"172.21.15.5:6805/14743","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-09T22:25:45.132616+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-09T22:25:45.072484+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-09T22:25:45.139526+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-09T22:25:57.202 INFO:tasks.ceph.ceph_manager.c1:all up! 2022-09-09T22:25:57.203 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2022-09-09T22:25:57.494 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:25:57.495 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":13,"fsid":"5b208e58-c41f-477d-87e6-917d46e40055","created":"2022-09-09T22:25:39.512569+0000","modified":"2022-09-09T22:25:56.501678+0000","last_up_change":"2022-09-09T22:25:47.788726+0000","last_in_change":"2022-09-09T22:25:40.385756+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-09T22:25:46.888928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-09-09T22:25:54.675151+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"0bb9dc41-555d-4aff-8b41-9cdc2ad5a531","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":14744},{"type":"v1","addr":"172.21.15.5:6817","nonce":14744}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":14744},{"type":"v1","addr":"172.21.15.5:6819","nonce":14744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":14744},{"type":"v1","addr":"172.21.15.5:6823","nonce":14744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":14744},{"type":"v1","addr":"172.21.15.5:6821","nonce":14744}]},"public_addr":"172.21.15.5:6817/14744","cluster_addr":"172.21.15.5:6819/14744","heartbeat_back_addr":"172.21.15.5:6823/14744","heartbeat_front_addr":"172.21.15.5:6821/14744","state":["exists","up"]},{"osd":1,"uuid":"cfccd431-020e-401d-9857-7dcb485564f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":14745},{"type":"v1","addr":"172.21.15.5:6809","nonce":14745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":14745},{"type":"v1","addr":"172.21.15.5:6811","nonce":14745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":14745},{"type":"v1","addr":"172.21.15.5:6815","nonce":14745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":14745},{"type":"v1","addr":"172.21.15.5:6813","nonce":14745}]},"public_addr":"172.21.15.5:6809/14745","cluster_addr":"172.21.15.5:6811/14745","heartbeat_back_addr":"172.21.15.5:6815/14745","heartbeat_front_addr":"172.21.15.5:6813/14745","state":["exists","up"]},{"osd":2,"uuid":"3c8716cf-e7e5-44bf-89b8-bd600b77faca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":14743},{"type":"v1","addr":"172.21.15.5:6801","nonce":14743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":14743},{"type":"v1","addr":"172.21.15.5:6803","nonce":14743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":14743},{"type":"v1","addr":"172.21.15.5:6807","nonce":14743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":14743},{"type":"v1","addr":"172.21.15.5:6805","nonce":14743}]},"public_addr":"172.21.15.5:6801/14743","cluster_addr":"172.21.15.5:6803/14743","heartbeat_back_addr":"172.21.15.5:6807/14743","heartbeat_front_addr":"172.21.15.5:6805/14743","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-09T22:25:45.132616+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-09T22:25:45.072484+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-09T22:25:45.139526+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-09T22:25:57.509 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2022-09-09T22:25:57.510 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2022-09-09T22:25:57.511 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2022-09-09T22:25:57.658 INFO:teuthology.orchestra.run.smithi005.stdout:34359738372 2022-09-09T22:25:57.659 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2022-09-09T22:25:57.661 INFO:teuthology.orchestra.run.smithi005.stdout:38654705669 2022-09-09T22:25:57.662 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2022-09-09T22:25:57.664 INFO:teuthology.orchestra.run.smithi005.stdout:34359738372 2022-09-09T22:25:57.665 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2022-09-09T22:25:57.949 INFO:teuthology.orchestra.run.smithi005.stdout:34359738371 2022-09-09T22:25:57.962 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738371 for osd.1 2022-09-09T22:25:57.964 INFO:teuthology.orchestra.run.smithi005.stdout:38654705668 2022-09-09T22:25:57.971 INFO:teuthology.orchestra.run.smithi005.stdout:34359738371 2022-09-09T22:25:57.979 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705669 got 38654705668 for osd.2 2022-09-09T22:25:57.987 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738371 for osd.0 2022-09-09T22:25:58.963 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2022-09-09T22:25:58.980 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2022-09-09T22:25:58.988 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2022-09-09T22:25:59.267 INFO:teuthology.orchestra.run.smithi005.stdout:34359738371 2022-09-09T22:25:59.282 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738371 for osd.1 2022-09-09T22:25:59.296 INFO:teuthology.orchestra.run.smithi005.stdout:34359738371 2022-09-09T22:25:59.306 INFO:teuthology.orchestra.run.smithi005.stdout:38654705668 2022-09-09T22:25:59.309 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738371 for osd.0 2022-09-09T22:25:59.320 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705669 got 38654705668 for osd.2 2022-09-09T22:26:00.283 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2022-09-09T22:26:00.310 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2022-09-09T22:26:00.323 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2022-09-09T22:26:00.609 INFO:teuthology.orchestra.run.smithi005.stdout:34359738372 2022-09-09T22:26:00.621 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.1 2022-09-09T22:26:00.621 DEBUG:teuthology.parallel:result is None 2022-09-09T22:26:00.625 INFO:teuthology.orchestra.run.smithi005.stdout:38654705670 2022-09-09T22:26:00.633 INFO:teuthology.orchestra.run.smithi005.stdout:34359738372 2022-09-09T22:26:00.640 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705669 got 38654705670 for osd.2 2022-09-09T22:26:00.641 DEBUG:teuthology.parallel:result is None 2022-09-09T22:26:00.648 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.0 2022-09-09T22:26:00.649 DEBUG:teuthology.parallel:result is None 2022-09-09T22:26:00.650 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2022-09-09T22:26:00.650 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2022-09-09T22:26:00.984 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:00.987 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2022-09-09T22:26:00.999 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2022-09-09T22:26:00.815611+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":24944,"kb_used_data":1496,"kb_used_omap":0,"kb_used_meta":23424,"kb_avail":314547856,"statfs":{"total":322122547200,"available":322097004544,"internally_reserved":0,"allocated":1531904,"data_stored":1275068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23986176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001495"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-09T22:25:56.814063+0000","last_change":"2022-09-09T22:25:56.515799+0000","last_active":"2022-09-09T22:25:56.814063+0000","last_peered":"2022-09-09T22:25:56.814063+0000","last_clean":"2022-09-09T22:25:56.814063+0000","last_became_active":"2022-09-09T22:25:56.515460+0000","last_became_peered":"2022-09-09T22:25:56.515460+0000","last_unstale":"2022-09-09T22:25:56.814063+0000","last_undegraded":"2022-09-09T22:25:56.814063+0000","last_fullsized":"2022-09-09T22:25:56.814063+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_clean_scrub_stamp":"2022-09-09T22:25:55.496702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-10T23:26:04.015554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-09T22:25:56.814003+0000","last_change":"2022-09-09T22:25:56.518044+0000","last_active":"2022-09-09T22:25:56.814003+0000","last_peered":"2022-09-09T22:25:56.814003+0000","last_clean":"2022-09-09T22:25:56.814003+0000","last_became_active":"2022-09-09T22:25:56.517790+0000","last_became_peered":"2022-09-09T22:25:56.517790+0000","last_unstale":"2022-09-09T22:25:56.814003+0000","last_undegraded":"2022-09-09T22:25:56.814003+0000","last_fullsized":"2022-09-09T22:25:56.814003+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_clean_scrub_stamp":"2022-09-09T22:25:55.496702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:45:41.862047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-09T22:25:56.813965+0000","last_change":"2022-09-09T22:25:56.517849+0000","last_active":"2022-09-09T22:25:56.813965+0000","last_peered":"2022-09-09T22:25:56.813965+0000","last_clean":"2022-09-09T22:25:56.813965+0000","last_became_active":"2022-09-09T22:25:56.517231+0000","last_became_peered":"2022-09-09T22:25:56.517231+0000","last_unstale":"2022-09-09T22:25:56.813965+0000","last_undegraded":"2022-09-09T22:25:56.813965+0000","last_fullsized":"2022-09-09T22:25:56.813965+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_clean_scrub_stamp":"2022-09-09T22:25:55.496702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:10:35.226138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-09T22:25:56.813943+0000","last_change":"2022-09-09T22:25:56.516804+0000","last_active":"2022-09-09T22:25:56.813943+0000","last_peered":"2022-09-09T22:25:56.813943+0000","last_clean":"2022-09-09T22:25:56.813943+0000","last_became_active":"2022-09-09T22:25:56.516476+0000","last_became_peered":"2022-09-09T22:25:56.516476+0000","last_unstale":"2022-09-09T22:25:56.813943+0000","last_undegraded":"2022-09-09T22:25:56.813943+0000","last_fullsized":"2022-09-09T22:25:56.813943+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_clean_scrub_stamp":"2022-09-09T22:25:55.496702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T00:37:30.330991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-09T22:25:56.518701+0000","last_change":"2022-09-09T22:25:56.517369+0000","last_active":"2022-09-09T22:25:56.518701+0000","last_peered":"2022-09-09T22:25:56.518701+0000","last_clean":"2022-09-09T22:25:56.518701+0000","last_became_active":"2022-09-09T22:25:56.517025+0000","last_became_peered":"2022-09-09T22:25:56.517025+0000","last_unstale":"2022-09-09T22:25:56.518701+0000","last_undegraded":"2022-09-09T22:25:56.518701+0000","last_fullsized":"2022-09-09T22:25:56.518701+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_clean_scrub_stamp":"2022-09-09T22:25:55.496702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:09:48.746200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-09T22:25:56.519505+0000","last_change":"2022-09-09T22:25:56.518462+0000","last_active":"2022-09-09T22:25:56.519505+0000","last_peered":"2022-09-09T22:25:56.519505+0000","last_clean":"2022-09-09T22:25:56.519505+0000","last_became_active":"2022-09-09T22:25:56.518195+0000","last_became_peered":"2022-09-09T22:25:56.518195+0000","last_unstale":"2022-09-09T22:25:56.519505+0000","last_undegraded":"2022-09-09T22:25:56.519505+0000","last_fullsized":"2022-09-09T22:25:56.519505+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_clean_scrub_stamp":"2022-09-09T22:25:55.496702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:11:31.179613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-09T22:25:56.517995+0000","last_change":"2022-09-09T22:25:56.516924+0000","last_active":"2022-09-09T22:25:56.517995+0000","last_peered":"2022-09-09T22:25:56.517995+0000","last_clean":"2022-09-09T22:25:56.517995+0000","last_became_active":"2022-09-09T22:25:56.516529+0000","last_became_peered":"2022-09-09T22:25:56.516529+0000","last_unstale":"2022-09-09T22:25:56.517995+0000","last_undegraded":"2022-09-09T22:25:56.517995+0000","last_fullsized":"2022-09-09T22:25:56.517995+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_clean_scrub_stamp":"2022-09-09T22:25:55.496702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-10T23:40:34.010071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-09T22:25:56.813904+0000","last_change":"2022-09-09T22:25:56.515708+0000","last_active":"2022-09-09T22:25:56.813904+0000","last_peered":"2022-09-09T22:25:56.813904+0000","last_clean":"2022-09-09T22:25:56.813904+0000","last_became_active":"2022-09-09T22:25:56.515432+0000","last_became_peered":"2022-09-09T22:25:56.515432+0000","last_unstale":"2022-09-09T22:25:56.813904+0000","last_undegraded":"2022-09-09T22:25:56.813904+0000","last_fullsized":"2022-09-09T22:25:56.813904+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_clean_scrub_stamp":"2022-09-09T22:25:55.496702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-10T23:58:07.435719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":140,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-09T22:25:56.506907+0000","last_change":"2022-09-09T22:25:48.811263+0000","last_active":"2022-09-09T22:25:56.506907+0000","last_peered":"2022-09-09T22:25:56.506907+0000","last_clean":"2022-09-09T22:25:56.506907+0000","last_became_active":"2022-09-09T22:25:48.809299+0000","last_became_peered":"2022-09-09T22:25:48.809299+0000","last_unstale":"2022-09-09T22:25:56.506907+0000","last_undegraded":"2022-09-09T22:25:56.506907+0000","last_fullsized":"2022-09-09T22:25:56.506907+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:25:47.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:25:47.788726+0000","last_clean_scrub_stamp":"2022-09-09T22:25:47.788726+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:38:18.508882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7672,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849928,"statfs":{"total":107374182400,"available":107366326272,"internally_reserved":0,"allocated":114688,"data_stored":31444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8636,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848964,"statfs":{"total":107374182400,"available":107365339136,"internally_reserved":0,"allocated":708608,"data_stored":621812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8636,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848964,"statfs":{"total":107374182400,"available":107365339136,"internally_reserved":0,"allocated":708608,"data_stored":621812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-09-09T22:26:01.001 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2022-09-09T22:26:01.290 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:01.292 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2022-09-09T22:26:01.303 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2022-09-09T22:26:00.815611+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":24944,"kb_used_data":1496,"kb_used_omap":0,"kb_used_meta":23424,"kb_avail":314547856,"statfs":{"total":322122547200,"available":322097004544,"internally_reserved":0,"allocated":1531904,"data_stored":1275068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23986176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001495"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-09T22:25:56.814063+0000","last_change":"2022-09-09T22:25:56.515799+0000","last_active":"2022-09-09T22:25:56.814063+0000","last_peered":"2022-09-09T22:25:56.814063+0000","last_clean":"2022-09-09T22:25:56.814063+0000","last_became_active":"2022-09-09T22:25:56.515460+0000","last_became_peered":"2022-09-09T22:25:56.515460+0000","last_unstale":"2022-09-09T22:25:56.814063+0000","last_undegraded":"2022-09-09T22:25:56.814063+0000","last_fullsized":"2022-09-09T22:25:56.814063+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_clean_scrub_stamp":"2022-09-09T22:25:55.496702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-10T23:26:04.015554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-09T22:25:56.814003+0000","last_change":"2022-09-09T22:25:56.518044+0000","last_active":"2022-09-09T22:25:56.814003+0000","last_peered":"2022-09-09T22:25:56.814003+0000","last_clean":"2022-09-09T22:25:56.814003+0000","last_became_active":"2022-09-09T22:25:56.517790+0000","last_became_peered":"2022-09-09T22:25:56.517790+0000","last_unstale":"2022-09-09T22:25:56.814003+0000","last_undegraded":"2022-09-09T22:25:56.814003+0000","last_fullsized":"2022-09-09T22:25:56.814003+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_clean_scrub_stamp":"2022-09-09T22:25:55.496702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:45:41.862047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-09T22:25:56.813965+0000","last_change":"2022-09-09T22:25:56.517849+0000","last_active":"2022-09-09T22:25:56.813965+0000","last_peered":"2022-09-09T22:25:56.813965+0000","last_clean":"2022-09-09T22:25:56.813965+0000","last_became_active":"2022-09-09T22:25:56.517231+0000","last_became_peered":"2022-09-09T22:25:56.517231+0000","last_unstale":"2022-09-09T22:25:56.813965+0000","last_undegraded":"2022-09-09T22:25:56.813965+0000","last_fullsized":"2022-09-09T22:25:56.813965+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_clean_scrub_stamp":"2022-09-09T22:25:55.496702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:10:35.226138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-09T22:25:56.813943+0000","last_change":"2022-09-09T22:25:56.516804+0000","last_active":"2022-09-09T22:25:56.813943+0000","last_peered":"2022-09-09T22:25:56.813943+0000","last_clean":"2022-09-09T22:25:56.813943+0000","last_became_active":"2022-09-09T22:25:56.516476+0000","last_became_peered":"2022-09-09T22:25:56.516476+0000","last_unstale":"2022-09-09T22:25:56.813943+0000","last_undegraded":"2022-09-09T22:25:56.813943+0000","last_fullsized":"2022-09-09T22:25:56.813943+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_clean_scrub_stamp":"2022-09-09T22:25:55.496702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T00:37:30.330991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-09T22:25:56.518701+0000","last_change":"2022-09-09T22:25:56.517369+0000","last_active":"2022-09-09T22:25:56.518701+0000","last_peered":"2022-09-09T22:25:56.518701+0000","last_clean":"2022-09-09T22:25:56.518701+0000","last_became_active":"2022-09-09T22:25:56.517025+0000","last_became_peered":"2022-09-09T22:25:56.517025+0000","last_unstale":"2022-09-09T22:25:56.518701+0000","last_undegraded":"2022-09-09T22:25:56.518701+0000","last_fullsized":"2022-09-09T22:25:56.518701+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_clean_scrub_stamp":"2022-09-09T22:25:55.496702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:09:48.746200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-09T22:25:56.519505+0000","last_change":"2022-09-09T22:25:56.518462+0000","last_active":"2022-09-09T22:25:56.519505+0000","last_peered":"2022-09-09T22:25:56.519505+0000","last_clean":"2022-09-09T22:25:56.519505+0000","last_became_active":"2022-09-09T22:25:56.518195+0000","last_became_peered":"2022-09-09T22:25:56.518195+0000","last_unstale":"2022-09-09T22:25:56.519505+0000","last_undegraded":"2022-09-09T22:25:56.519505+0000","last_fullsized":"2022-09-09T22:25:56.519505+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_clean_scrub_stamp":"2022-09-09T22:25:55.496702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:11:31.179613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-09T22:25:56.517995+0000","last_change":"2022-09-09T22:25:56.516924+0000","last_active":"2022-09-09T22:25:56.517995+0000","last_peered":"2022-09-09T22:25:56.517995+0000","last_clean":"2022-09-09T22:25:56.517995+0000","last_became_active":"2022-09-09T22:25:56.516529+0000","last_became_peered":"2022-09-09T22:25:56.516529+0000","last_unstale":"2022-09-09T22:25:56.517995+0000","last_undegraded":"2022-09-09T22:25:56.517995+0000","last_fullsized":"2022-09-09T22:25:56.517995+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_clean_scrub_stamp":"2022-09-09T22:25:55.496702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-10T23:40:34.010071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-09T22:25:56.813904+0000","last_change":"2022-09-09T22:25:56.515708+0000","last_active":"2022-09-09T22:25:56.813904+0000","last_peered":"2022-09-09T22:25:56.813904+0000","last_clean":"2022-09-09T22:25:56.813904+0000","last_became_active":"2022-09-09T22:25:56.515432+0000","last_became_peered":"2022-09-09T22:25:56.515432+0000","last_unstale":"2022-09-09T22:25:56.813904+0000","last_undegraded":"2022-09-09T22:25:56.813904+0000","last_fullsized":"2022-09-09T22:25:56.813904+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:25:55.496702+0000","last_clean_scrub_stamp":"2022-09-09T22:25:55.496702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-10T23:58:07.435719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":140,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-09T22:25:56.506907+0000","last_change":"2022-09-09T22:25:48.811263+0000","last_active":"2022-09-09T22:25:56.506907+0000","last_peered":"2022-09-09T22:25:56.506907+0000","last_clean":"2022-09-09T22:25:56.506907+0000","last_became_active":"2022-09-09T22:25:48.809299+0000","last_became_peered":"2022-09-09T22:25:48.809299+0000","last_unstale":"2022-09-09T22:25:56.506907+0000","last_undegraded":"2022-09-09T22:25:56.506907+0000","last_fullsized":"2022-09-09T22:25:56.506907+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:25:47.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:25:47.788726+0000","last_clean_scrub_stamp":"2022-09-09T22:25:47.788726+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:38:18.508882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7672,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849928,"statfs":{"total":107374182400,"available":107366326272,"internally_reserved":0,"allocated":114688,"data_stored":31444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8636,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848964,"statfs":{"total":107374182400,"available":107365339136,"internally_reserved":0,"allocated":708608,"data_stored":621812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8636,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848964,"statfs":{"total":107374182400,"available":107365339136,"internally_reserved":0,"allocated":708608,"data_stored":621812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-09-09T22:26:01.304 INFO:tasks.ceph.ceph_manager.c1:clean! 2022-09-09T22:26:01.305 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2022-09-09T22:26:01.305 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2022-09-09T22:26:01.305 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2022-09-09T22:26:01.628 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:01.628 INFO:teuthology.orchestra.run.smithi005.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-09-09T22:26:01.641 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2022-09-09T22:26:01.642 INFO:teuthology.run_tasks:Running task ceph... 2022-09-09T22:26:01.652 INFO:tasks.ceph:Creating ceph cluster c2... 2022-09-09T22:26:01.652 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2022-09-09T22:26:01.652 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/mbenjamin-2022-09-09_22:09:46-rgw-wip-librgw-frontends-distro-default-smithi/7023840', 'branch': 'wip-librgw-frontends', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{ubuntu_latest} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '7023840', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'mbenjamin-2022-09-09_22:09:46-rgw-wip-librgw-frontends-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-librgw-frontends'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '28758235ba2ab1c1d8afa0f88d00f07b5687a51e', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '28758235ba2ab1c1d8afa0f88d00f07b5687a51e'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py', 'tests_ps.py']}, 'workunit': {'branch': 'wip-librgw-frontends', 'sha1': '28758235ba2ab1c1d8afa0f88d00f07b5687a51e'}}, 'owner': 'scheduled_mbenjamin@teuthology', 'priority': 70, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '28758235ba2ab1c1d8afa0f88d00f07b5687a51e', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-librgw-frontends', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_28758235ba2ab1c1d8afa0f88d00f07b5687a51e/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '28758235ba2ab1c1d8afa0f88d00f07b5687a51e', 'targets': {'smithi005.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzFKnoOt2BoS/UBn9KunV/1GYUVB2AcY7cWUKACXEAWiSig9PUvOMwoMCkSvUbgs5stNzMZn58M3TydDGcq1zQwW7asez3uAzAi387ckCknkDmTFJdbrHLGna2mHGF6eL74cRn+9grca/oUUwKrTEQsW8bsgiHtL0gTV/f1rTK/3pyDBRSyy6gfPbIU3LK63P4/e8rRTNm57Jr/RWGt1TwV7dTiHSbBrAJpxD/6qY/4q4fsVM7HJZHB7rmV8tdeoB85+15PDy8JjoyxBdlp1UOE7NIrvBQzu+9V0nR2UvH8N4ryDBG8N1XfHsD9R+3Zg2lCYHenXcKJupnY6xNoSlJfHfkwsard+2GIy+4aJ57s6c3K+UKjnqdxib40X9RdJzNs2Qc+ARp+zbO+sAamr4wWWZFb0nhZUO89u3INoFbpT5gC3mnCI+p2YTeHZG/kJ8U14RnQ37Mai06jQeqFIHivFr8JX/3/pfWiBXg/9bl3y/37pU24aQ/dtBg/K87W68=', 'smithi080.front.sepia.ceph.com': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIC0XOORBR/2Qj/b6x+ngeEme2+rTgZ2w3aDFr4sjpDmu'}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '28758235ba2ab1c1d8afa0f88d00f07b5687a51e', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '28758235ba2ab1c1d8afa0f88d00f07b5687a51e', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'b58f2c18636eb10faa77ed3614abd00cb85dfc2c', 'timestamp': '2022-09-09_22:09:46', 'tube': 'smithi', 'user': 'mbenjamin', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.1713'} 2022-09-09T22:26:01.653 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2022-09-09T22:26:01.657 DEBUG:teuthology.orchestra.run.smithi080:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2022-09-09T22:26:01.665 DEBUG:teuthology.orchestra.run.smithi005:> sudo install -d -m0777 -- /var/run/ceph 2022-09-09T22:26:01.668 DEBUG:teuthology.orchestra.run.smithi080:> sudo install -d -m0777 -- /var/run/ceph 2022-09-09T22:26:01.727 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2022-09-09T22:26:01.727 DEBUG:teuthology.orchestra.run.smithi080:> dd if=/scratch_devs of=/dev/stdout 2022-09-09T22:26:01.735 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-09-09T22:26:01.736 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-09-09T22:26:01.743 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-09-09T22:26:01.744 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-09T22:26:01.745 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 6h/6d Inode: 358 Links: 1 2022-09-09T22:26:01.745 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-09T22:26:01.746 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2022-09-09 22:22:34.727996332 +0000 2022-09-09T22:26:01.746 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2022-09-09 22:22:34.375996340 +0000 2022-09-09T22:26:01.747 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2022-09-09 22:22:34.375996340 +0000 2022-09-09T22:26:01.747 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: - 2022-09-09T22:26:01.753 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2022-09-09T22:26:01.754 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2022-09-09T22:26:01.754 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.00039897 s, 1.3 MB/s 2022-09-09T22:26:01.757 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-09-09T22:26:01.763 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-09-09T22:26:01.763 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-09T22:26:01.764 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 6h/6d Inode: 361 Links: 1 2022-09-09T22:26:01.764 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-09T22:26:01.765 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2022-09-09 22:22:34.951996327 +0000 2022-09-09T22:26:01.765 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2022-09-09 22:22:34.947996328 +0000 2022-09-09T22:26:01.766 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2022-09-09 22:22:34.947996328 +0000 2022-09-09T22:26:01.766 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: - 2022-09-09T22:26:01.772 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2022-09-09T22:26:01.772 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2022-09-09T22:26:01.773 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.000320023 s, 1.6 MB/s 2022-09-09T22:26:01.777 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-09-09T22:26:01.825 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-09-09T22:26:01.825 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-09T22:26:01.825 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 6h/6d Inode: 368 Links: 1 2022-09-09T22:26:01.826 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-09T22:26:01.826 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2022-09-09 22:22:35.891996307 +0000 2022-09-09T22:26:01.826 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2022-09-09 22:22:35.535996314 +0000 2022-09-09T22:26:01.826 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2022-09-09 22:22:35.535996314 +0000 2022-09-09T22:26:01.827 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: - 2022-09-09T22:26:01.835 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2022-09-09T22:26:01.835 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2022-09-09T22:26:01.835 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.000398514 s, 1.3 MB/s 2022-09-09T22:26:01.840 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-09-09T22:26:01.847 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-09-09T22:26:01.848 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-09T22:26:01.849 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 6h/6d Inode: 377 Links: 1 2022-09-09T22:26:01.849 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-09T22:26:01.849 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2022-09-09 22:22:36.463996294 +0000 2022-09-09T22:26:01.850 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2022-09-09 22:22:36.107996302 +0000 2022-09-09T22:26:01.850 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2022-09-09 22:22:36.107996302 +0000 2022-09-09T22:26:01.851 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: - 2022-09-09T22:26:01.853 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2022-09-09T22:26:01.854 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2022-09-09T22:26:01.855 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.00040559 s, 1.3 MB/s 2022-09-09T22:26:01.858 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-09T22:26:01.859 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi080.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2022-09-09T22:26:01.859 INFO:tasks.ceph:Generating config... 2022-09-09T22:26:01.861 INFO:tasks.ceph:[client] debug rgw = 20 2022-09-09T22:26:01.861 INFO:tasks.ceph:[client] debug rgw sync = 20 2022-09-09T22:26:01.862 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2022-09-09T22:26:01.863 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2022-09-09T22:26:01.863 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-09-09T22:26:01.863 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2022-09-09T22:26:01.864 INFO:tasks.ceph:[client] rgw data log num shards = 4 2022-09-09T22:26:01.864 INFO:tasks.ceph:[client] rgw md log max shards = 4 2022-09-09T22:26:01.864 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2022-09-09T22:26:01.865 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2022-09-09T22:26:01.865 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2022-09-09T22:26:01.865 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2022-09-09T22:26:01.865 INFO:tasks.ceph:[client] setgroup = ceph 2022-09-09T22:26:01.866 INFO:tasks.ceph:[client] setuser = ceph 2022-09-09T22:26:01.866 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2022-09-09T22:26:01.866 INFO:tasks.ceph:[mgr] debug mgr = 20 2022-09-09T22:26:01.867 INFO:tasks.ceph:[mgr] debug ms = 1 2022-09-09T22:26:01.867 INFO:tasks.ceph:[mon] debug mon = 20 2022-09-09T22:26:01.867 INFO:tasks.ceph:[mon] debug ms = 1 2022-09-09T22:26:01.868 INFO:tasks.ceph:[mon] debug paxos = 20 2022-09-09T22:26:01.868 INFO:tasks.ceph:[osd] debug ms = 1 2022-09-09T22:26:01.868 INFO:tasks.ceph:[osd] debug osd = 20 2022-09-09T22:26:01.869 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2022-09-09T22:26:01.869 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2022-09-09T22:26:01.869 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2022-09-09T22:26:01.870 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2022-09-09T22:26:01.870 INFO:tasks.ceph:Setting up c2.mon.a... 2022-09-09T22:26:01.872 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2022-09-09T22:26:01.909 INFO:teuthology.orchestra.run.smithi080.stdout:creating /etc/ceph/c2.keyring 2022-09-09T22:26:01.932 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2022-09-09T22:26:02.016 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod 0644 /etc/ceph/c2.keyring 2022-09-09T22:26:02.028 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.80')] 2022-09-09T22:26:02.028 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.80'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2022-09-09T22:26:02.030 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2022-09-09T22:26:02.031 DEBUG:teuthology.orchestra.run.smithi080:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2022-09-09T22:26:02.040 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.80 --print /home/ubuntu/cephtest/c2.monmap 2022-09-09T22:26:02.126 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:26:02.127 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:26:02.129 INFO:teuthology.orchestra.run.smithi080.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2022-09-09T22:26:02.129 INFO:teuthology.orchestra.run.smithi080.stdout:monmaptool: generated fsid f02fb5e7-d126-4010-a4f9-7dd384aa0e34 2022-09-09T22:26:02.130 INFO:teuthology.orchestra.run.smithi080.stdout:setting min_mon_release = octopus 2022-09-09T22:26:02.131 INFO:teuthology.orchestra.run.smithi080.stdout:epoch 0 2022-09-09T22:26:02.131 INFO:teuthology.orchestra.run.smithi080.stdout:fsid f02fb5e7-d126-4010-a4f9-7dd384aa0e34 2022-09-09T22:26:02.132 INFO:teuthology.orchestra.run.smithi080.stdout:last_changed 2022-09-09T22:26:02.126725+0000 2022-09-09T22:26:02.133 INFO:teuthology.orchestra.run.smithi080.stdout:created 2022-09-09T22:26:02.126725+0000 2022-09-09T22:26:02.134 INFO:teuthology.orchestra.run.smithi080.stdout:min_mon_release 15 (octopus) 2022-09-09T22:26:02.134 INFO:teuthology.orchestra.run.smithi080.stdout:election_strategy: 1 2022-09-09T22:26:02.135 INFO:teuthology.orchestra.run.smithi080.stdout:0: [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] mon.a 2022-09-09T22:26:02.135 INFO:teuthology.orchestra.run.smithi080.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2022-09-09T22:26:02.136 DEBUG:teuthology.orchestra.run.smithi080:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2022-09-09T22:26:02.176 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID f02fb5e7-d126-4010-a4f9-7dd384aa0e34... 2022-09-09T22:26:02.178 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2022-09-09T22:26:02.184 DEBUG:teuthology.orchestra.run.smithi080:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2022-09-09T22:26:02.215 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2022-09-09T22:26:02.215 INFO:teuthology.orchestra.run.smithi005.stdout: chdir = "" 2022-09-09T22:26:02.215 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-09-09T22:26:02.216 INFO:teuthology.orchestra.run.smithi005.stdout: auth supported = cephx 2022-09-09T22:26:02.216 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:02.217 INFO:teuthology.orchestra.run.smithi005.stdout: filestore xattr use omap = true 2022-09-09T22:26:02.217 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:02.217 INFO:teuthology.orchestra.run.smithi005.stdout: mon clock drift allowed = 1.000 2022-09-09T22:26:02.218 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:02.218 INFO:teuthology.orchestra.run.smithi005.stdout: osd crush chooseleaf type = 0 2022-09-09T22:26:02.218 INFO:teuthology.orchestra.run.smithi005.stdout: auth debug = true 2022-09-09T22:26:02.218 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:02.219 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on old message = true 2022-09-09T22:26:02.219 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on bug = true 2022-09-09T22:26:02.219 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:02.219 INFO:teuthology.orchestra.run.smithi005.stdout: mon max pg per osd = 10000 # >= luminous 2022-09-09T22:26:02.219 INFO:teuthology.orchestra.run.smithi005.stdout: mon pg warn max object skew = 0 2022-09-09T22:26:02.220 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:02.220 INFO:teuthology.orchestra.run.smithi005.stdout: # disable pg_autoscaler by default for new pools 2022-09-09T22:26:02.220 INFO:teuthology.orchestra.run.smithi005.stdout: osd_pool_default_pg_autoscale_mode = off 2022-09-09T22:26:02.220 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:02.221 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default size = 2 2022-09-09T22:26:02.221 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:02.221 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow primary affinity = true 2022-09-09T22:26:02.222 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow pg remap = true 2022-09-09T22:26:02.222 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on legacy crush tunables = false 2022-09-09T22:26:02.222 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on crush straw calc version zero = false 2022-09-09T22:26:02.222 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on no sortbitwise = false 2022-09-09T22:26:02.224 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on osd down out interval zero = false 2022-09-09T22:26:02.224 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on too few osds = false 2022-09-09T22:26:02.225 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-09-09T22:26:02.225 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_no_redundancy = false 2022-09-09T22:26:02.225 INFO:teuthology.orchestra.run.smithi005.stdout: mon_allow_pool_size_one = true 2022-09-09T22:26:02.225 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:02.226 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-09-09T22:26:02.226 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:02.226 INFO:teuthology.orchestra.run.smithi005.stdout: osd default data pool replay window = 5 2022-09-09T22:26:02.227 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:02.227 INFO:teuthology.orchestra.run.smithi005.stdout: mon allow pool delete = true 2022-09-09T22:26:02.228 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:02.228 INFO:teuthology.orchestra.run.smithi005.stdout: mon cluster log file level = debug 2022-09-09T22:26:02.228 INFO:teuthology.orchestra.run.smithi005.stdout: debug asserts on shutdown = true 2022-09-09T22:26:02.228 INFO:teuthology.orchestra.run.smithi005.stdout: mon health detail to clog = false 2022-09-09T22:26:02.229 INFO:teuthology.orchestra.run.smithi005.stdout: mon host = 172.21.15.80 2022-09-09T22:26:02.229 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = f02fb5e7-d126-4010-a4f9-7dd384aa0e34 2022-09-09T22:26:02.229 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:02.229 INFO:teuthology.orchestra.run.smithi005.stdout:[osd] 2022-09-09T22:26:02.230 INFO:teuthology.orchestra.run.smithi005.stdout: osd journal size = 100 2022-09-09T22:26:02.230 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:02.230 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub load threshold = 5.0 2022-09-09T22:26:02.230 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub max interval = 600 2022-09-09T22:26:02.231 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:02.231 INFO:teuthology.orchestra.run.smithi005.stdout: osd recover clone overlap = true 2022-09-09T22:26:02.231 INFO:teuthology.orchestra.run.smithi005.stdout: osd recovery max chunk = 1048576 2022-09-09T22:26:02.231 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:02.232 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug shutdown = true 2022-09-09T22:26:02.232 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug op order = true 2022-09-09T22:26:02.232 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify stray on activate = true 2022-09-09T22:26:02.232 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:02.232 INFO:teuthology.orchestra.run.smithi005.stdout: osd open classes on start = true 2022-09-09T22:26:02.233 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug pg log writeout = true 2022-09-09T22:26:02.233 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:02.233 INFO:teuthology.orchestra.run.smithi005.stdout: osd deep scrub update digest min age = 30 2022-09-09T22:26:02.233 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:02.234 INFO:teuthology.orchestra.run.smithi005.stdout: osd map max advance = 10 2022-09-09T22:26:02.236 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:02.236 INFO:teuthology.orchestra.run.smithi005.stdout: journal zero on create = true 2022-09-09T22:26:02.236 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:02.236 INFO:teuthology.orchestra.run.smithi005.stdout: filestore ondisk finisher threads = 3 2022-09-09T22:26:02.237 INFO:teuthology.orchestra.run.smithi005.stdout: filestore apply finisher threads = 3 2022-09-09T22:26:02.237 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:02.237 INFO:teuthology.orchestra.run.smithi005.stdout: bdev debug aio = true 2022-09-09T22:26:02.237 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug misdirected ops = true 2022-09-09T22:26:02.238 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2022-09-09T22:26:02.238 INFO:teuthology.orchestra.run.smithi005.stdout: debug osd = 20 2022-09-09T22:26:02.238 INFO:teuthology.orchestra.run.smithi005.stdout: rocksdb delete range threshold = 0 2022-09-09T22:26:02.238 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:02.239 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr] 2022-09-09T22:26:02.239 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2022-09-09T22:26:02.239 INFO:teuthology.orchestra.run.smithi005.stdout: debug mgr = 20 2022-09-09T22:26:02.239 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2022-09-09T22:26:02.239 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2022-09-09T22:26:02.240 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2022-09-09T22:26:02.240 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2022-09-09T22:26:02.240 INFO:teuthology.orchestra.run.smithi005.stdout: mgr/telemetry/nag = false 2022-09-09T22:26:02.240 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:02.241 INFO:teuthology.orchestra.run.smithi005.stdout:[mon] 2022-09-09T22:26:02.241 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2022-09-09T22:26:02.241 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2022-09-09T22:26:02.241 INFO:teuthology.orchestra.run.smithi005.stdout: debug paxos = 20 2022-09-09T22:26:02.242 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2022-09-09T22:26:02.242 INFO:teuthology.orchestra.run.smithi005.stdout: mon data avail warn = 5 2022-09-09T22:26:02.242 INFO:teuthology.orchestra.run.smithi005.stdout: mon mgr mkfs grace = 240 2022-09-09T22:26:02.243 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2022-09-09T22:26:02.243 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd reporter subtree level = osd 2022-09-09T22:26:02.244 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd prime pg temp = true 2022-09-09T22:26:02.244 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2022-09-09T22:26:02.244 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:02.245 INFO:teuthology.orchestra.run.smithi005.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-09-09T22:26:02.245 INFO:teuthology.orchestra.run.smithi005.stdout: auth mon ticket ttl = 660 # 11m 2022-09-09T22:26:02.246 INFO:teuthology.orchestra.run.smithi005.stdout: auth service ticket ttl = 240 # 4m 2022-09-09T22:26:02.246 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:02.246 INFO:teuthology.orchestra.run.smithi005.stdout: # don't complain about insecure global_id in the test suite 2022-09-09T22:26:02.247 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-09-09T22:26:02.247 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-09-09T22:26:02.248 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:02.248 INFO:teuthology.orchestra.run.smithi005.stdout: # 1m isn't quite enough 2022-09-09T22:26:02.249 INFO:teuthology.orchestra.run.smithi005.stdout: mon_down_mkfs_grace = 2m 2022-09-09T22:26:02.249 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:02.249 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_filestore_osds = false 2022-09-09T22:26:02.250 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:26:02.250 INFO:teuthology.orchestra.run.smithi005.stdout:[client] 2022-09-09T22:26:02.250 INFO:teuthology.orchestra.run.smithi005.stdout: rgw cache enabled = true 2022-09-09T22:26:02.251 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable ops log = true 2022-09-09T22:26:02.251 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable usage log = true 2022-09-09T22:26:02.251 INFO:teuthology.orchestra.run.smithi005.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-09-09T22:26:02.252 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-09-09T22:26:02.252 INFO:teuthology.orchestra.run.smithi005.stdout: debug rgw = 20 2022-09-09T22:26:02.253 INFO:teuthology.orchestra.run.smithi005.stdout: debug rgw sync = 20 2022-09-09T22:26:02.253 INFO:teuthology.orchestra.run.smithi005.stdout: rgw crypt require ssl = False 2022-09-09T22:26:02.253 INFO:teuthology.orchestra.run.smithi005.stdout: rgw crypt s3 kms backend = testing 2022-09-09T22:26:02.254 INFO:teuthology.orchestra.run.smithi005.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-09-09T22:26:02.254 INFO:teuthology.orchestra.run.smithi005.stdout: rgw curl low speed time = 300 2022-09-09T22:26:02.255 INFO:teuthology.orchestra.run.smithi005.stdout: rgw data log num shards = 4 2022-09-09T22:26:02.255 INFO:teuthology.orchestra.run.smithi005.stdout: rgw md log max shards = 4 2022-09-09T22:26:02.256 INFO:teuthology.orchestra.run.smithi005.stdout: rgw sync data inject err probability = 0.1 2022-09-09T22:26:02.256 INFO:teuthology.orchestra.run.smithi005.stdout: rgw sync log trim interval = 0 2022-09-09T22:26:02.257 INFO:teuthology.orchestra.run.smithi005.stdout: rgw sync meta inject err probability = 0.1 2022-09-09T22:26:02.257 INFO:teuthology.orchestra.run.smithi005.stdout: rgw sync obj etag verify = True 2022-09-09T22:26:02.257 INFO:teuthology.orchestra.run.smithi005.stdout: setgroup = ceph 2022-09-09T22:26:02.258 INFO:teuthology.orchestra.run.smithi005.stdout: setuser = ceph 2022-09-09T22:26:02.258 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.a] 2022-09-09T22:26:02.259 INFO:teuthology.orchestra.run.smithi005.stdout:[client.0] 2022-09-09T22:26:02.259 INFO:teuthology.orchestra.run.smithi005.stdout: rgw data notify interval msec = 0 2022-09-09T22:26:02.260 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.0] 2022-09-09T22:26:02.260 INFO:teuthology.orchestra.run.smithi005.stdout: osd_max_omap_entries_per_request = 10 2022-09-09T22:26:02.261 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.1] 2022-09-09T22:26:02.261 INFO:teuthology.orchestra.run.smithi005.stdout: osd_max_omap_entries_per_request = 1000 2022-09-09T22:26:02.261 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.2] 2022-09-09T22:26:02.262 INFO:teuthology.orchestra.run.smithi005.stdout: osd_max_omap_entries_per_request = 10000 2022-09-09T22:26:02.265 INFO:teuthology.orchestra.run.smithi080.stdout:[global] 2022-09-09T22:26:02.265 INFO:teuthology.orchestra.run.smithi080.stdout: chdir = "" 2022-09-09T22:26:02.266 INFO:teuthology.orchestra.run.smithi080.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-09-09T22:26:02.266 INFO:teuthology.orchestra.run.smithi080.stdout: auth supported = cephx 2022-09-09T22:26:02.266 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:02.267 INFO:teuthology.orchestra.run.smithi080.stdout: filestore xattr use omap = true 2022-09-09T22:26:02.267 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:02.267 INFO:teuthology.orchestra.run.smithi080.stdout: mon clock drift allowed = 1.000 2022-09-09T22:26:02.268 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:02.268 INFO:teuthology.orchestra.run.smithi080.stdout: osd crush chooseleaf type = 0 2022-09-09T22:26:02.268 INFO:teuthology.orchestra.run.smithi080.stdout: auth debug = true 2022-09-09T22:26:02.269 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:02.269 INFO:teuthology.orchestra.run.smithi080.stdout: ms die on old message = true 2022-09-09T22:26:02.270 INFO:teuthology.orchestra.run.smithi080.stdout: ms die on bug = true 2022-09-09T22:26:02.270 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:02.270 INFO:teuthology.orchestra.run.smithi080.stdout: mon max pg per osd = 10000 # >= luminous 2022-09-09T22:26:02.271 INFO:teuthology.orchestra.run.smithi080.stdout: mon pg warn max object skew = 0 2022-09-09T22:26:02.271 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:02.271 INFO:teuthology.orchestra.run.smithi080.stdout: # disable pg_autoscaler by default for new pools 2022-09-09T22:26:02.272 INFO:teuthology.orchestra.run.smithi080.stdout: osd_pool_default_pg_autoscale_mode = off 2022-09-09T22:26:02.272 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:02.272 INFO:teuthology.orchestra.run.smithi080.stdout: osd pool default size = 2 2022-09-09T22:26:02.273 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:02.273 INFO:teuthology.orchestra.run.smithi080.stdout: mon osd allow primary affinity = true 2022-09-09T22:26:02.273 INFO:teuthology.orchestra.run.smithi080.stdout: mon osd allow pg remap = true 2022-09-09T22:26:02.273 INFO:teuthology.orchestra.run.smithi080.stdout: mon warn on legacy crush tunables = false 2022-09-09T22:26:02.274 INFO:teuthology.orchestra.run.smithi080.stdout: mon warn on crush straw calc version zero = false 2022-09-09T22:26:02.274 INFO:teuthology.orchestra.run.smithi080.stdout: mon warn on no sortbitwise = false 2022-09-09T22:26:02.274 INFO:teuthology.orchestra.run.smithi080.stdout: mon warn on osd down out interval zero = false 2022-09-09T22:26:02.274 INFO:teuthology.orchestra.run.smithi080.stdout: mon warn on too few osds = false 2022-09-09T22:26:02.275 INFO:teuthology.orchestra.run.smithi080.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-09-09T22:26:02.275 INFO:teuthology.orchestra.run.smithi080.stdout: mon_warn_on_pool_no_redundancy = false 2022-09-09T22:26:02.275 INFO:teuthology.orchestra.run.smithi080.stdout: mon_allow_pool_size_one = true 2022-09-09T22:26:02.275 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:02.276 INFO:teuthology.orchestra.run.smithi080.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-09-09T22:26:02.276 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:02.276 INFO:teuthology.orchestra.run.smithi080.stdout: osd default data pool replay window = 5 2022-09-09T22:26:02.276 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:02.277 INFO:teuthology.orchestra.run.smithi080.stdout: mon allow pool delete = true 2022-09-09T22:26:02.277 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:02.277 INFO:teuthology.orchestra.run.smithi080.stdout: mon cluster log file level = debug 2022-09-09T22:26:02.278 INFO:teuthology.orchestra.run.smithi080.stdout: debug asserts on shutdown = true 2022-09-09T22:26:02.278 INFO:teuthology.orchestra.run.smithi080.stdout: mon health detail to clog = false 2022-09-09T22:26:02.278 INFO:teuthology.orchestra.run.smithi080.stdout: mon host = 172.21.15.80 2022-09-09T22:26:02.279 INFO:teuthology.orchestra.run.smithi080.stdout: fsid = f02fb5e7-d126-4010-a4f9-7dd384aa0e34 2022-09-09T22:26:02.279 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:02.279 INFO:teuthology.orchestra.run.smithi080.stdout:[osd] 2022-09-09T22:26:02.279 INFO:teuthology.orchestra.run.smithi080.stdout: osd journal size = 100 2022-09-09T22:26:02.280 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:02.280 INFO:teuthology.orchestra.run.smithi080.stdout: osd scrub load threshold = 5.0 2022-09-09T22:26:02.280 INFO:teuthology.orchestra.run.smithi080.stdout: osd scrub max interval = 600 2022-09-09T22:26:02.280 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:02.280 INFO:teuthology.orchestra.run.smithi080.stdout: osd recover clone overlap = true 2022-09-09T22:26:02.281 INFO:teuthology.orchestra.run.smithi080.stdout: osd recovery max chunk = 1048576 2022-09-09T22:26:02.281 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:02.281 INFO:teuthology.orchestra.run.smithi080.stdout: osd debug shutdown = true 2022-09-09T22:26:02.281 INFO:teuthology.orchestra.run.smithi080.stdout: osd debug op order = true 2022-09-09T22:26:02.281 INFO:teuthology.orchestra.run.smithi080.stdout: osd debug verify stray on activate = true 2022-09-09T22:26:02.282 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:02.282 INFO:teuthology.orchestra.run.smithi080.stdout: osd open classes on start = true 2022-09-09T22:26:02.283 INFO:teuthology.orchestra.run.smithi080.stdout: osd debug pg log writeout = true 2022-09-09T22:26:02.283 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:02.284 INFO:teuthology.orchestra.run.smithi080.stdout: osd deep scrub update digest min age = 30 2022-09-09T22:26:02.284 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:02.284 INFO:teuthology.orchestra.run.smithi080.stdout: osd map max advance = 10 2022-09-09T22:26:02.285 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:02.285 INFO:teuthology.orchestra.run.smithi080.stdout: journal zero on create = true 2022-09-09T22:26:02.286 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:02.286 INFO:teuthology.orchestra.run.smithi080.stdout: filestore ondisk finisher threads = 3 2022-09-09T22:26:02.287 INFO:teuthology.orchestra.run.smithi080.stdout: filestore apply finisher threads = 3 2022-09-09T22:26:02.287 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:02.287 INFO:teuthology.orchestra.run.smithi080.stdout: bdev debug aio = true 2022-09-09T22:26:02.288 INFO:teuthology.orchestra.run.smithi080.stdout: osd debug misdirected ops = true 2022-09-09T22:26:02.288 INFO:teuthology.orchestra.run.smithi080.stdout: debug ms = 1 2022-09-09T22:26:02.289 INFO:teuthology.orchestra.run.smithi080.stdout: debug osd = 20 2022-09-09T22:26:02.290 INFO:teuthology.orchestra.run.smithi080.stdout: rocksdb delete range threshold = 0 2022-09-09T22:26:02.290 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:02.290 INFO:teuthology.orchestra.run.smithi080.stdout:[mgr] 2022-09-09T22:26:02.291 INFO:teuthology.orchestra.run.smithi080.stdout: debug ms = 1 2022-09-09T22:26:02.291 INFO:teuthology.orchestra.run.smithi080.stdout: debug mgr = 20 2022-09-09T22:26:02.292 INFO:teuthology.orchestra.run.smithi080.stdout: debug mon = 20 2022-09-09T22:26:02.292 INFO:teuthology.orchestra.run.smithi080.stdout: debug auth = 20 2022-09-09T22:26:02.292 INFO:teuthology.orchestra.run.smithi080.stdout: mon reweight min pgs per osd = 4 2022-09-09T22:26:02.293 INFO:teuthology.orchestra.run.smithi080.stdout: mon reweight min bytes per osd = 10 2022-09-09T22:26:02.293 INFO:teuthology.orchestra.run.smithi080.stdout: mgr/telemetry/nag = false 2022-09-09T22:26:02.294 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:02.294 INFO:teuthology.orchestra.run.smithi080.stdout:[mon] 2022-09-09T22:26:02.295 INFO:teuthology.orchestra.run.smithi080.stdout: debug ms = 1 2022-09-09T22:26:02.295 INFO:teuthology.orchestra.run.smithi080.stdout: debug mon = 20 2022-09-09T22:26:02.295 INFO:teuthology.orchestra.run.smithi080.stdout: debug paxos = 20 2022-09-09T22:26:02.296 INFO:teuthology.orchestra.run.smithi080.stdout: debug auth = 20 2022-09-09T22:26:02.296 INFO:teuthology.orchestra.run.smithi080.stdout: mon data avail warn = 5 2022-09-09T22:26:02.297 INFO:teuthology.orchestra.run.smithi080.stdout: mon mgr mkfs grace = 240 2022-09-09T22:26:02.297 INFO:teuthology.orchestra.run.smithi080.stdout: mon reweight min pgs per osd = 4 2022-09-09T22:26:02.298 INFO:teuthology.orchestra.run.smithi080.stdout: mon osd reporter subtree level = osd 2022-09-09T22:26:02.298 INFO:teuthology.orchestra.run.smithi080.stdout: mon osd prime pg temp = true 2022-09-09T22:26:02.298 INFO:teuthology.orchestra.run.smithi080.stdout: mon reweight min bytes per osd = 10 2022-09-09T22:26:02.299 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:02.299 INFO:teuthology.orchestra.run.smithi080.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-09-09T22:26:02.300 INFO:teuthology.orchestra.run.smithi080.stdout: auth mon ticket ttl = 660 # 11m 2022-09-09T22:26:02.300 INFO:teuthology.orchestra.run.smithi080.stdout: auth service ticket ttl = 240 # 4m 2022-09-09T22:26:02.300 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:02.301 INFO:teuthology.orchestra.run.smithi080.stdout: # don't complain about insecure global_id in the test suite 2022-09-09T22:26:02.301 INFO:teuthology.orchestra.run.smithi080.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-09-09T22:26:02.301 INFO:teuthology.orchestra.run.smithi080.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-09-09T22:26:02.302 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:02.302 INFO:teuthology.orchestra.run.smithi080.stdout: # 1m isn't quite enough 2022-09-09T22:26:02.302 INFO:teuthology.orchestra.run.smithi080.stdout: mon_down_mkfs_grace = 2m 2022-09-09T22:26:02.302 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:02.303 INFO:teuthology.orchestra.run.smithi080.stdout: mon_warn_on_filestore_osds = false 2022-09-09T22:26:02.303 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:02.303 INFO:teuthology.orchestra.run.smithi080.stdout:[client] 2022-09-09T22:26:02.303 INFO:teuthology.orchestra.run.smithi080.stdout: rgw cache enabled = true 2022-09-09T22:26:02.304 INFO:teuthology.orchestra.run.smithi080.stdout: rgw enable ops log = true 2022-09-09T22:26:02.304 INFO:teuthology.orchestra.run.smithi080.stdout: rgw enable usage log = true 2022-09-09T22:26:02.304 INFO:teuthology.orchestra.run.smithi080.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-09-09T22:26:02.304 INFO:teuthology.orchestra.run.smithi080.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-09-09T22:26:02.305 INFO:teuthology.orchestra.run.smithi080.stdout: debug rgw = 20 2022-09-09T22:26:02.305 INFO:teuthology.orchestra.run.smithi080.stdout: debug rgw sync = 20 2022-09-09T22:26:02.305 INFO:teuthology.orchestra.run.smithi080.stdout: rgw crypt require ssl = False 2022-09-09T22:26:02.305 INFO:teuthology.orchestra.run.smithi080.stdout: rgw crypt s3 kms backend = testing 2022-09-09T22:26:02.305 INFO:teuthology.orchestra.run.smithi080.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-09-09T22:26:02.306 INFO:teuthology.orchestra.run.smithi080.stdout: rgw curl low speed time = 300 2022-09-09T22:26:02.306 INFO:teuthology.orchestra.run.smithi080.stdout: rgw data log num shards = 4 2022-09-09T22:26:02.306 INFO:teuthology.orchestra.run.smithi080.stdout: rgw md log max shards = 4 2022-09-09T22:26:02.306 INFO:teuthology.orchestra.run.smithi080.stdout: rgw sync data inject err probability = 0.1 2022-09-09T22:26:02.307 INFO:teuthology.orchestra.run.smithi080.stdout: rgw sync log trim interval = 0 2022-09-09T22:26:02.307 INFO:teuthology.orchestra.run.smithi080.stdout: rgw sync meta inject err probability = 0.1 2022-09-09T22:26:02.307 INFO:teuthology.orchestra.run.smithi080.stdout: rgw sync obj etag verify = True 2022-09-09T22:26:02.307 INFO:teuthology.orchestra.run.smithi080.stdout: setgroup = ceph 2022-09-09T22:26:02.307 INFO:teuthology.orchestra.run.smithi080.stdout: setuser = ceph 2022-09-09T22:26:02.308 INFO:teuthology.orchestra.run.smithi080.stdout:[mon.a] 2022-09-09T22:26:02.308 INFO:teuthology.orchestra.run.smithi080.stdout:[client.0] 2022-09-09T22:26:02.308 INFO:teuthology.orchestra.run.smithi080.stdout: rgw data notify interval msec = 0 2022-09-09T22:26:02.308 INFO:teuthology.orchestra.run.smithi080.stdout:[osd.0] 2022-09-09T22:26:02.309 INFO:teuthology.orchestra.run.smithi080.stdout: osd_max_omap_entries_per_request = 10 2022-09-09T22:26:02.309 INFO:teuthology.orchestra.run.smithi080.stdout:[osd.1] 2022-09-09T22:26:02.309 INFO:teuthology.orchestra.run.smithi080.stdout: osd_max_omap_entries_per_request = 1000 2022-09-09T22:26:02.309 INFO:teuthology.orchestra.run.smithi080.stdout:[osd.2] 2022-09-09T22:26:02.309 INFO:teuthology.orchestra.run.smithi080.stdout: osd_max_omap_entries_per_request = 10000 2022-09-09T22:26:02.310 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2022-09-09T22:26:02.310 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2022-09-09T22:26:02.365 INFO:tasks.ceph:Copying monmap to all nodes... 2022-09-09T22:26:02.366 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2022-09-09T22:26:02.367 DEBUG:teuthology.orchestra.run.smithi080:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2022-09-09T22:26:02.374 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2022-09-09T22:26:02.375 DEBUG:teuthology.orchestra.run.smithi080:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2022-09-09T22:26:02.426 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi005.front.sepia.ceph.com 2022-09-09T22:26:02.426 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-09-09T22:26:02.427 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/ceph/c2.keyring 2022-09-09T22:26:02.427 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0644 /etc/ceph/c2.keyring 2022-09-09T22:26:02.454 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-09-09T22:26:02.456 DEBUG:teuthology.orchestra.run.smithi005:> dd of=/home/ubuntu/cephtest/c2.monmap 2022-09-09T22:26:02.506 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi080.front.sepia.ceph.com 2022-09-09T22:26:02.506 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2022-09-09T22:26:02.507 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/ceph/c2.keyring 2022-09-09T22:26:02.508 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod 0644 /etc/ceph/c2.keyring 2022-09-09T22:26:02.535 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2022-09-09T22:26:02.536 DEBUG:teuthology.orchestra.run.smithi080:> dd of=/home/ubuntu/cephtest/c2.monmap 2022-09-09T22:26:02.541 INFO:tasks.ceph:Setting up mon nodes... 2022-09-09T22:26:02.543 INFO:tasks.ceph:Setting up mgr nodes... 2022-09-09T22:26:02.545 DEBUG:teuthology.orchestra.run.smithi080:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2022-09-09T22:26:02.636 INFO:teuthology.orchestra.run.smithi080.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2022-09-09T22:26:02.640 INFO:tasks.ceph:Setting up mds nodes... 2022-09-09T22:26:02.641 INFO:tasks.ceph_client:Setting up client nodes... 2022-09-09T22:26:02.642 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2022-09-09T22:26:02.680 INFO:teuthology.orchestra.run.smithi080.stdout:creating /etc/ceph/c2.client.0.keyring 2022-09-09T22:26:02.692 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2022-09-09T22:26:02.775 INFO:teuthology.orchestra.run.smithi080.stdout:creating /etc/ceph/c2.client.1.keyring 2022-09-09T22:26:02.793 INFO:tasks.ceph:Running mkfs on osd nodes... 2022-09-09T22:26:02.794 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi005.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi080.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2022-09-09T22:26:02.794 DEBUG:teuthology.orchestra.run.smithi080:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2022-09-09T22:26:02.808 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-09T22:26:02.809 INFO:tasks.ceph:role: c2.osd.0 2022-09-09T22:26:02.809 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi080.front.sepia.ceph.com 2022-09-09T22:26:02.810 DEBUG:teuthology.orchestra.run.smithi080:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-09-09T22:26:03.318 INFO:teuthology.orchestra.run.smithi080.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-09-09T22:26:03.318 INFO:teuthology.orchestra.run.smithi080.stdout: = sectsz=512 attr=2, projid32bit=1 2022-09-09T22:26:03.319 INFO:teuthology.orchestra.run.smithi080.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-09-09T22:26:03.319 INFO:teuthology.orchestra.run.smithi080.stdout: = reflink=1 2022-09-09T22:26:03.319 INFO:teuthology.orchestra.run.smithi080.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-09-09T22:26:03.320 INFO:teuthology.orchestra.run.smithi080.stdout: = sunit=0 swidth=0 blks 2022-09-09T22:26:03.320 INFO:teuthology.orchestra.run.smithi080.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-09-09T22:26:03.320 INFO:teuthology.orchestra.run.smithi080.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-09-09T22:26:03.320 INFO:teuthology.orchestra.run.smithi080.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-09-09T22:26:03.321 INFO:teuthology.orchestra.run.smithi080.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-09-09T22:26:03.322 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi080.front.sepia.ceph.com -o noatime 2022-09-09T22:26:03.323 DEBUG:teuthology.orchestra.run.smithi080:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2022-09-09T22:26:03.405 DEBUG:teuthology.orchestra.run.smithi080:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2022-09-09T22:26:03.421 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: /sbin/restorecon: command not found 2022-09-09T22:26:03.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-09T22:26:03.424 DEBUG:teuthology.orchestra.run.smithi080:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2022-09-09T22:26:03.482 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-09T22:26:03.483 INFO:tasks.ceph:role: c2.osd.1 2022-09-09T22:26:03.483 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi080.front.sepia.ceph.com 2022-09-09T22:26:03.484 DEBUG:teuthology.orchestra.run.smithi080:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-09-09T22:26:03.993 INFO:teuthology.orchestra.run.smithi080.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-09-09T22:26:03.993 INFO:teuthology.orchestra.run.smithi080.stdout: = sectsz=512 attr=2, projid32bit=1 2022-09-09T22:26:03.994 INFO:teuthology.orchestra.run.smithi080.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-09-09T22:26:03.994 INFO:teuthology.orchestra.run.smithi080.stdout: = reflink=1 2022-09-09T22:26:03.995 INFO:teuthology.orchestra.run.smithi080.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-09-09T22:26:03.995 INFO:teuthology.orchestra.run.smithi080.stdout: = sunit=0 swidth=0 blks 2022-09-09T22:26:03.996 INFO:teuthology.orchestra.run.smithi080.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-09-09T22:26:03.997 INFO:teuthology.orchestra.run.smithi080.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-09-09T22:26:03.997 INFO:teuthology.orchestra.run.smithi080.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-09-09T22:26:03.997 INFO:teuthology.orchestra.run.smithi080.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-09-09T22:26:03.999 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi080.front.sepia.ceph.com -o noatime 2022-09-09T22:26:04.000 DEBUG:teuthology.orchestra.run.smithi080:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2022-09-09T22:26:04.038 DEBUG:teuthology.orchestra.run.smithi080:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2022-09-09T22:26:04.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-09T22:26:04.095 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: /sbin/restorecon: command not found 2022-09-09T22:26:04.096 DEBUG:teuthology.orchestra.run.smithi080:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2022-09-09T22:26:04.109 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-09T22:26:04.109 INFO:tasks.ceph:role: c2.osd.2 2022-09-09T22:26:04.109 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi080.front.sepia.ceph.com 2022-09-09T22:26:04.110 DEBUG:teuthology.orchestra.run.smithi080:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-09-09T22:26:04.610 INFO:teuthology.orchestra.run.smithi080.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-09-09T22:26:04.610 INFO:teuthology.orchestra.run.smithi080.stdout: = sectsz=512 attr=2, projid32bit=1 2022-09-09T22:26:04.611 INFO:teuthology.orchestra.run.smithi080.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-09-09T22:26:04.611 INFO:teuthology.orchestra.run.smithi080.stdout: = reflink=1 2022-09-09T22:26:04.611 INFO:teuthology.orchestra.run.smithi080.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-09-09T22:26:04.611 INFO:teuthology.orchestra.run.smithi080.stdout: = sunit=0 swidth=0 blks 2022-09-09T22:26:04.612 INFO:teuthology.orchestra.run.smithi080.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-09-09T22:26:04.612 INFO:teuthology.orchestra.run.smithi080.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-09-09T22:26:04.612 INFO:teuthology.orchestra.run.smithi080.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-09-09T22:26:04.612 INFO:teuthology.orchestra.run.smithi080.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-09-09T22:26:04.613 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi080.front.sepia.ceph.com -o noatime 2022-09-09T22:26:04.614 DEBUG:teuthology.orchestra.run.smithi080:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2022-09-09T22:26:04.677 DEBUG:teuthology.orchestra.run.smithi080:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2022-09-09T22:26:04.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-09T22:26:04.693 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: /sbin/restorecon: command not found 2022-09-09T22:26:04.694 DEBUG:teuthology.orchestra.run.smithi080:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2022-09-09T22:26:04.823 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:04.819+0000 7f161c4de4c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2022-09-09T22:26:04.828 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:04.823+0000 7f161c4de4c0 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2022-09-09T22:26:04.828 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:04.823+0000 7f161c4de4c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2022-09-09T22:26:04.829 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:04.823+0000 7f161c4de4c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2022-09-09T22:26:04.829 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:04.823+0000 7f161c4de4c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2022-09-09T22:26:04.831 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:04.827+0000 7f161c4de4c0 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2022-09-09T22:26:07.025 DEBUG:teuthology.orchestra.run.smithi080:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2022-09-09T22:26:07.049 DEBUG:teuthology.orchestra.run.smithi080:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2022-09-09T22:26:07.142 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:07.135+0000 7f04f80a84c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2022-09-09T22:26:07.144 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:07.135+0000 7f04f80a84c0 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2022-09-09T22:26:07.144 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:07.135+0000 7f04f80a84c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2022-09-09T22:26:07.145 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:07.135+0000 7f04f80a84c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2022-09-09T22:26:07.145 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:07.135+0000 7f04f80a84c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2022-09-09T22:26:07.146 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:07.139+0000 7f04f80a84c0 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2022-09-09T22:26:09.329 DEBUG:teuthology.orchestra.run.smithi080:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2022-09-09T22:26:09.348 DEBUG:teuthology.orchestra.run.smithi080:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2022-09-09T22:26:09.394 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:09.387+0000 7fb1edaab4c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2022-09-09T22:26:09.395 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:09.387+0000 7fb1edaab4c0 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2022-09-09T22:26:09.395 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:09.387+0000 7fb1edaab4c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2022-09-09T22:26:09.396 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:09.387+0000 7fb1edaab4c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2022-09-09T22:26:09.396 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:09.387+0000 7fb1edaab4c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2022-09-09T22:26:09.397 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:09.387+0000 7fb1edaab4c0 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2022-09-09T22:26:11.581 DEBUG:teuthology.orchestra.run.smithi080:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2022-09-09T22:26:11.638 INFO:tasks.ceph:Reading keys from all nodes... 2022-09-09T22:26:11.639 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2022-09-09T22:26:11.640 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2022-09-09T22:26:11.651 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2022-09-09T22:26:11.652 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2022-09-09T22:26:11.709 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2022-09-09T22:26:11.710 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2022-09-09T22:26:11.721 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2022-09-09T22:26:11.721 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2022-09-09T22:26:11.775 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2022-09-09T22:26:11.776 DEBUG:teuthology.orchestra.run.smithi080:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2022-09-09T22:26:11.784 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2022-09-09T22:26:11.785 DEBUG:teuthology.orchestra.run.smithi080:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2022-09-09T22:26:11.792 INFO:tasks.ceph:Adding keys to all mons... 2022-09-09T22:26:11.793 DEBUG:teuthology.orchestra.run.smithi080:> sudo tee -a /etc/ceph/c2.keyring 2022-09-09T22:26:11.804 INFO:teuthology.orchestra.run.smithi080.stdout:[mgr.x] 2022-09-09T22:26:11.804 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQB6vRtjy1gAJhAA0l4tI8U9xZpOeGIPr2Ri8Q== 2022-09-09T22:26:11.804 INFO:teuthology.orchestra.run.smithi080.stdout:[osd.0] 2022-09-09T22:26:11.805 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQB8vRtjsbQiMRAAmq0Gta9skRLcGwPO2G6zEw== 2022-09-09T22:26:11.805 INFO:teuthology.orchestra.run.smithi080.stdout:[osd.1] 2022-09-09T22:26:11.805 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQB/vRtj4/OFCBAA2Ms6XrVHEWQuMwm/w1mSAQ== 2022-09-09T22:26:11.806 INFO:teuthology.orchestra.run.smithi080.stdout:[osd.2] 2022-09-09T22:26:11.806 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQCBvRtjASd8FxAAUkHbEvfok/EydyvCBECKMQ== 2022-09-09T22:26:11.806 INFO:teuthology.orchestra.run.smithi080.stdout:[client.0] 2022-09-09T22:26:11.807 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQB6vRtjML2ZKBAABsVs34tWGcJAL9uQVbDQVA== 2022-09-09T22:26:11.807 INFO:teuthology.orchestra.run.smithi080.stdout:[client.1] 2022-09-09T22:26:11.807 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQB6vRtjn8hOLhAAvn4SlsbTLbZoh151nkZz+A== 2022-09-09T22:26:11.808 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-09-09T22:26:11.901 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-09-09T22:26:11.951 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-09-09T22:26:12.002 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-09-09T22:26:12.053 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-09-09T22:26:12.104 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-09-09T22:26:12.156 INFO:tasks.ceph:Running mkfs on mon nodes... 2022-09-09T22:26:12.157 DEBUG:teuthology.orchestra.run.smithi080:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2022-09-09T22:26:12.173 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2022-09-09T22:26:12.239 DEBUG:teuthology.orchestra.run.smithi080:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2022-09-09T22:26:12.253 DEBUG:teuthology.orchestra.run.smithi080:> rm -- /home/ubuntu/cephtest/c2.monmap 2022-09-09T22:26:12.260 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2022-09-09T22:26:12.263 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2022-09-09T22:26:12.263 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2022-09-09T22:26:12.304 INFO:tasks.ceph.c2.mon.a:Started 2022-09-09T22:26:12.304 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2022-09-09T22:26:12.305 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2022-09-09T22:26:12.305 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2022-09-09T22:26:12.308 INFO:tasks.ceph.c2.mgr.x:Started 2022-09-09T22:26:12.309 INFO:tasks.ceph:Setting crush tunables to default 2022-09-09T22:26:12.309 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph --cluster c2 osd crush tunables default 2022-09-09T22:26:12.545 INFO:teuthology.orchestra.run.smithi080.stderr:adjusted tunables profile to default 2022-09-09T22:26:12.556 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2022-09-09T22:26:12.556 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2022-09-09T22:26:12.557 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2022-09-09T22:26:12.571 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2022-09-09T22:26:12.571 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2022-09-09T22:26:12.589 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2022-09-09T22:26:12.590 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2022-09-09T22:26:12.603 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph --cluster c2 osd new 4e7d83bf-b636-4e88-8c67-f6a514236928 0 2022-09-09T22:26:12.870 INFO:teuthology.orchestra.run.smithi080.stdout:0 2022-09-09T22:26:12.878 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph --cluster c2 osd new a6eaacc9-f96c-42bc-8ea6-b678a37865d0 1 2022-09-09T22:26:12.943 INFO:tasks.ceph.c2.mgr.x.smithi080.stderr:2022-09-09T22:26:12.939+0000 7fd9dc3ea140 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-09-09T22:26:13.005 INFO:tasks.ceph.c2.mgr.x.smithi080.stderr:2022-09-09T22:26:12.999+0000 7fd9dc3ea140 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-09-09T22:26:13.082 INFO:teuthology.orchestra.run.smithi080.stdout:1 2022-09-09T22:26:13.095 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph --cluster c2 osd new cbad909f-77f0-4494-b443-5a662e8c1a56 2 2022-09-09T22:26:13.310 INFO:teuthology.orchestra.run.smithi080.stdout:2 2022-09-09T22:26:13.324 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2022-09-09T22:26:13.325 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2022-09-09T22:26:13.328 INFO:tasks.ceph.c2.osd.0:Started 2022-09-09T22:26:13.328 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2022-09-09T22:26:13.329 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2022-09-09T22:26:13.331 INFO:tasks.ceph.c2.osd.1:Started 2022-09-09T22:26:13.332 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2022-09-09T22:26:13.332 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2022-09-09T22:26:13.334 INFO:tasks.ceph.c2.osd.2:Started 2022-09-09T22:26:13.335 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2022-09-09T22:26:13.436 INFO:tasks.ceph.c2.mgr.x.smithi080.stderr:2022-09-09T22:26:13.431+0000 7fd9dc3ea140 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-09-09T22:26:13.495 INFO:tasks.ceph.c2.mgr.x.smithi080.stderr:2022-09-09T22:26:13.491+0000 7fd9dc3ea140 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-09-09T22:26:13.522 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:13.523 INFO:teuthology.orchestra.run.smithi080.stdout:{"epoch":5,"fsid":"f02fb5e7-d126-4010-a4f9-7dd384aa0e34","created":"2022-09-09T22:26:12.433144+0000","modified":"2022-09-09T22:26:13.307461+0000","last_up_change":"0.000000","last_in_change":"2022-09-09T22:26:13.307461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"4e7d83bf-b636-4e88-8c67-f6a514236928","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a6eaacc9-f96c-42bc-8ea6-b678a37865d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cbad909f-77f0-4494-b443-5a662e8c1a56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-09T22:26:13.539 INFO:tasks.ceph.ceph_manager.c2:[] 2022-09-09T22:26:13.540 INFO:tasks.ceph:Waiting for OSDs to come up 2022-09-09T22:26:13.593 INFO:tasks.ceph.c2.mgr.x.smithi080.stderr:2022-09-09T22:26:13.587+0000 7fd9dc3ea140 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-09-09T22:26:13.711 INFO:tasks.ceph.c2.osd.2.smithi080.stderr:2022-09-09T22:26:13.707+0000 7fbca06aa4c0 -1 Falling back to public interface 2022-09-09T22:26:13.726 INFO:tasks.ceph.c2.mgr.x.smithi080.stderr:2022-09-09T22:26:13.723+0000 7fd9dc3ea140 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-09-09T22:26:13.779 INFO:tasks.ceph.c2.mgr.x.smithi080.stderr:2022-09-09T22:26:13.775+0000 7fd9dc3ea140 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-09-09T22:26:13.842 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2022-09-09T22:26:13.927 INFO:tasks.ceph.c2.osd.1.smithi080.stderr:2022-09-09T22:26:13.923+0000 7f2e944424c0 -1 Falling back to public interface 2022-09-09T22:26:13.967 INFO:tasks.ceph.c2.osd.0.smithi080.stderr:2022-09-09T22:26:13.963+0000 7f13a7a074c0 -1 Falling back to public interface 2022-09-09T22:26:14.017 INFO:teuthology.misc.health.smithi080.stdout: 2022-09-09T22:26:14.017 INFO:teuthology.misc.health.smithi080.stdout:{"epoch":5,"fsid":"f02fb5e7-d126-4010-a4f9-7dd384aa0e34","created":"2022-09-09T22:26:12.433144+0000","modified":"2022-09-09T22:26:13.307461+0000","last_up_change":"0.000000","last_in_change":"2022-09-09T22:26:13.307461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"4e7d83bf-b636-4e88-8c67-f6a514236928","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a6eaacc9-f96c-42bc-8ea6-b678a37865d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cbad909f-77f0-4494-b443-5a662e8c1a56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-09T22:26:14.019 INFO:tasks.ceph.c2.mgr.x.smithi080.stderr:2022-09-09T22:26:14.015+0000 7fd9dc3ea140 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-09-09T22:26:14.027 DEBUG:teuthology.misc:0 of 3 OSDs are up 2022-09-09T22:26:14.182 INFO:tasks.ceph.c2.mgr.x.smithi080.stderr:2022-09-09T22:26:14.175+0000 7fd9dc3ea140 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-09-09T22:26:14.230 INFO:tasks.ceph.c2.mgr.x.smithi080.stderr:2022-09-09T22:26:14.227+0000 7fd9dc3ea140 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-09-09T22:26:14.852 INFO:tasks.ceph.c2.mgr.x.smithi080.stderr:2022-09-09T22:26:14.847+0000 7fd9dc3ea140 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-09-09T22:26:15.048 INFO:tasks.ceph.c2.mgr.x.smithi080.stderr:2022-09-09T22:26:15.039+0000 7fd9dc3ea140 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-09-09T22:26:15.655 INFO:tasks.ceph.c2.mgr.x.smithi080.stderr:2022-09-09T22:26:15.651+0000 7fd9dc3ea140 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-09-09T22:26:15.707 INFO:tasks.ceph.c2.mgr.x.smithi080.stderr:2022-09-09T22:26:15.703+0000 7fd9dc3ea140 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-09-09T22:26:15.770 INFO:tasks.ceph.c2.mgr.x.smithi080.stderr:2022-09-09T22:26:15.763+0000 7fd9dc3ea140 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-09-09T22:26:16.196 INFO:tasks.ceph.c2.osd.1.smithi080.stderr:2022-09-09T22:26:16.191+0000 7f2e944424c0 -1 osd.1 0 log_to_monitors true 2022-09-09T22:26:16.455 INFO:tasks.ceph.c2.mgr.x.smithi080.stderr:2022-09-09T22:26:16.451+0000 7fd9dc3ea140 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-09-09T22:26:16.512 INFO:tasks.ceph.c2.mgr.x.smithi080.stderr:2022-09-09T22:26:16.507+0000 7fd9dc3ea140 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-09-09T22:26:16.574 INFO:tasks.ceph.c2.mgr.x.smithi080.stderr:2022-09-09T22:26:16.571+0000 7fd9dc3ea140 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-09-09T22:26:16.790 INFO:tasks.ceph.c2.osd.2.smithi080.stderr:2022-09-09T22:26:16.787+0000 7fbca06aa4c0 -1 osd.2 0 log_to_monitors true 2022-09-09T22:26:16.870 INFO:tasks.ceph.c2.mgr.x.smithi080.stderr:2022-09-09T22:26:16.863+0000 7fd9dc3ea140 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-09-09T22:26:16.936 INFO:tasks.ceph.c2.mgr.x.smithi080.stderr:2022-09-09T22:26:16.931+0000 7fd9dc3ea140 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-09-09T22:26:16.989 INFO:tasks.ceph.c2.osd.0.smithi080.stderr:2022-09-09T22:26:16.983+0000 7f13a7a074c0 -1 osd.0 0 log_to_monitors true 2022-09-09T22:26:17.205 INFO:tasks.ceph.c2.mgr.x.smithi080.stderr:2022-09-09T22:26:17.199+0000 7fd9dc3ea140 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-09-09T22:26:17.321 INFO:tasks.ceph.c2.mgr.x.smithi080.stderr:2022-09-09T22:26:17.315+0000 7fd9dc3ea140 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-09-09T22:26:17.386 INFO:tasks.ceph.c2.mgr.x.smithi080.stderr:2022-09-09T22:26:17.379+0000 7fd9dc3ea140 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-09-09T22:26:17.977 INFO:tasks.ceph.c2.osd.1.smithi080.stderr:2022-09-09T22:26:17.967+0000 7f2e8fbe3700 -1 osd.1 0 waiting for initial osdmap 2022-09-09T22:26:17.978 INFO:tasks.ceph.c2.osd.1.smithi080.stderr:2022-09-09T22:26:17.975+0000 7f2e87eb8700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-09T22:26:18.465 INFO:tasks.ceph.c2.mgr.x.smithi080.stderr:2022-09-09T22:26:18.459+0000 7fd9c3b48700 -1 mgr.server handle_report got status from non-daemon mon.a 2022-09-09T22:26:19.385 INFO:tasks.ceph.c2.osd.0.smithi080.stderr:2022-09-09T22:26:19.379+0000 7f13a31a8700 -1 osd.0 0 waiting for initial osdmap 2022-09-09T22:26:19.390 INFO:tasks.ceph.c2.osd.0.smithi080.stderr:2022-09-09T22:26:19.387+0000 7f139b47d700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-09T22:26:19.508 INFO:tasks.ceph.c2.osd.2.smithi080.stderr:2022-09-09T22:26:19.503+0000 7fbc9be4b700 -1 osd.2 0 waiting for initial osdmap 2022-09-09T22:26:19.513 INFO:tasks.ceph.c2.osd.2.smithi080.stderr:2022-09-09T22:26:19.507+0000 7fbc94120700 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-09T22:26:20.331 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2022-09-09T22:26:20.623 INFO:teuthology.misc.health.smithi080.stdout: 2022-09-09T22:26:20.624 INFO:teuthology.misc.health.smithi080.stdout:{"epoch":10,"fsid":"f02fb5e7-d126-4010-a4f9-7dd384aa0e34","created":"2022-09-09T22:26:12.433144+0000","modified":"2022-09-09T22:26:20.514785+0000","last_up_change":"2022-09-09T22:26:20.514785+0000","last_in_change":"2022-09-09T22:26:13.307461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"4e7d83bf-b636-4e88-8c67-f6a514236928","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":15038},{"type":"v1","addr":"172.21.15.80:6817","nonce":15038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":15038},{"type":"v1","addr":"172.21.15.80:6819","nonce":15038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":15038},{"type":"v1","addr":"172.21.15.80:6823","nonce":15038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":15038},{"type":"v1","addr":"172.21.15.80:6821","nonce":15038}]},"public_addr":"172.21.15.80:6817/15038","cluster_addr":"172.21.15.80:6819/15038","heartbeat_back_addr":"172.21.15.80:6823/15038","heartbeat_front_addr":"172.21.15.80:6821/15038","state":["exists","up"]},{"osd":1,"uuid":"a6eaacc9-f96c-42bc-8ea6-b678a37865d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":15036},{"type":"v1","addr":"172.21.15.80:6809","nonce":15036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":15036},{"type":"v1","addr":"172.21.15.80:6811","nonce":15036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":15036},{"type":"v1","addr":"172.21.15.80:6815","nonce":15036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":15036},{"type":"v1","addr":"172.21.15.80:6813","nonce":15036}]},"public_addr":"172.21.15.80:6809/15036","cluster_addr":"172.21.15.80:6811/15036","heartbeat_back_addr":"172.21.15.80:6815/15036","heartbeat_front_addr":"172.21.15.80:6813/15036","state":["exists","up"]},{"osd":2,"uuid":"cbad909f-77f0-4494-b443-5a662e8c1a56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":15037},{"type":"v1","addr":"172.21.15.80:6801","nonce":15037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":15037},{"type":"v1","addr":"172.21.15.80:6803","nonce":15037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":15037},{"type":"v1","addr":"172.21.15.80:6807","nonce":15037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":15037},{"type":"v1","addr":"172.21.15.80:6805","nonce":15037}]},"public_addr":"172.21.15.80:6801/15037","cluster_addr":"172.21.15.80:6803/15037","heartbeat_back_addr":"172.21.15.80:6807/15037","heartbeat_front_addr":"172.21.15.80:6805/15037","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-09T22:26:17.958466+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-09T22:26:17.245815+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-09T22:26:20.638 DEBUG:teuthology.misc:3 of 3 OSDs are up 2022-09-09T22:26:20.638 INFO:tasks.ceph:Creating RBD pool 2022-09-09T22:26:20.639 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph --cluster c2 osd pool create rbd 8 2022-09-09T22:26:21.533 INFO:teuthology.orchestra.run.smithi080.stderr:pool 'rbd' created 2022-09-09T22:26:21.552 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2022-09-09T22:26:22.537 INFO:teuthology.orchestra.run.smithi080.stderr:enabled application 'rbd' on pool 'rbd' 2022-09-09T22:26:22.550 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2022-09-09T22:26:22.552 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2022-09-09T22:26:22.552 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2022-09-09T22:26:22.552 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2022-09-09T22:26:22.554 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2022-09-09T22:26:22.885 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:22.899 INFO:teuthology.orchestra.run.smithi080.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":14852},{"type":"v1","addr":"172.21.15.80:6825","nonce":14852}]},"active_addr":"172.21.15.80:6825/14852","active_change":"2022-09-09T22:26:17.405181+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.80:0","nonce":3533017395}]},{"addrvec":[{"type":"v2","addr":"172.21.15.80:0","nonce":1734740700}]},{"addrvec":[{"type":"v2","addr":"172.21.15.80:0","nonce":2913710660}]}]}} 2022-09-09T22:26:22.902 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2022-09-09T22:26:22.903 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2022-09-09T22:26:22.904 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2022-09-09T22:26:23.213 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:23.214 INFO:teuthology.orchestra.run.smithi080.stdout:{"epoch":12,"fsid":"f02fb5e7-d126-4010-a4f9-7dd384aa0e34","created":"2022-09-09T22:26:12.433144+0000","modified":"2022-09-09T22:26:22.520010+0000","last_up_change":"2022-09-09T22:26:20.514785+0000","last_in_change":"2022-09-09T22:26:13.307461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2022-09-09T22:26:20.964198+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2022-09-09T22:26:21.499783+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"4e7d83bf-b636-4e88-8c67-f6a514236928","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":15038},{"type":"v1","addr":"172.21.15.80:6817","nonce":15038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":15038},{"type":"v1","addr":"172.21.15.80:6819","nonce":15038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":15038},{"type":"v1","addr":"172.21.15.80:6823","nonce":15038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":15038},{"type":"v1","addr":"172.21.15.80:6821","nonce":15038}]},"public_addr":"172.21.15.80:6817/15038","cluster_addr":"172.21.15.80:6819/15038","heartbeat_back_addr":"172.21.15.80:6823/15038","heartbeat_front_addr":"172.21.15.80:6821/15038","state":["exists","up"]},{"osd":1,"uuid":"a6eaacc9-f96c-42bc-8ea6-b678a37865d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":15036},{"type":"v1","addr":"172.21.15.80:6809","nonce":15036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":15036},{"type":"v1","addr":"172.21.15.80:6811","nonce":15036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":15036},{"type":"v1","addr":"172.21.15.80:6815","nonce":15036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":15036},{"type":"v1","addr":"172.21.15.80:6813","nonce":15036}]},"public_addr":"172.21.15.80:6809/15036","cluster_addr":"172.21.15.80:6811/15036","heartbeat_back_addr":"172.21.15.80:6815/15036","heartbeat_front_addr":"172.21.15.80:6813/15036","state":["exists","up"]},{"osd":2,"uuid":"cbad909f-77f0-4494-b443-5a662e8c1a56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":15037},{"type":"v1","addr":"172.21.15.80:6801","nonce":15037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":15037},{"type":"v1","addr":"172.21.15.80:6803","nonce":15037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":15037},{"type":"v1","addr":"172.21.15.80:6807","nonce":15037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":15037},{"type":"v1","addr":"172.21.15.80:6805","nonce":15037}]},"public_addr":"172.21.15.80:6801/15037","cluster_addr":"172.21.15.80:6803/15037","heartbeat_back_addr":"172.21.15.80:6807/15037","heartbeat_front_addr":"172.21.15.80:6805/15037","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-09T22:26:17.958466+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-09T22:26:17.245815+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-09T22:26:17.779333+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-09T22:26:23.232 INFO:tasks.ceph.ceph_manager.c2:all up! 2022-09-09T22:26:23.233 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2022-09-09T22:26:23.516 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:23.517 INFO:teuthology.orchestra.run.smithi080.stdout:{"epoch":12,"fsid":"f02fb5e7-d126-4010-a4f9-7dd384aa0e34","created":"2022-09-09T22:26:12.433144+0000","modified":"2022-09-09T22:26:22.520010+0000","last_up_change":"2022-09-09T22:26:20.514785+0000","last_in_change":"2022-09-09T22:26:13.307461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2022-09-09T22:26:20.964198+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2022-09-09T22:26:21.499783+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"4e7d83bf-b636-4e88-8c67-f6a514236928","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":15038},{"type":"v1","addr":"172.21.15.80:6817","nonce":15038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":15038},{"type":"v1","addr":"172.21.15.80:6819","nonce":15038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":15038},{"type":"v1","addr":"172.21.15.80:6823","nonce":15038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":15038},{"type":"v1","addr":"172.21.15.80:6821","nonce":15038}]},"public_addr":"172.21.15.80:6817/15038","cluster_addr":"172.21.15.80:6819/15038","heartbeat_back_addr":"172.21.15.80:6823/15038","heartbeat_front_addr":"172.21.15.80:6821/15038","state":["exists","up"]},{"osd":1,"uuid":"a6eaacc9-f96c-42bc-8ea6-b678a37865d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":15036},{"type":"v1","addr":"172.21.15.80:6809","nonce":15036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":15036},{"type":"v1","addr":"172.21.15.80:6811","nonce":15036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":15036},{"type":"v1","addr":"172.21.15.80:6815","nonce":15036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":15036},{"type":"v1","addr":"172.21.15.80:6813","nonce":15036}]},"public_addr":"172.21.15.80:6809/15036","cluster_addr":"172.21.15.80:6811/15036","heartbeat_back_addr":"172.21.15.80:6815/15036","heartbeat_front_addr":"172.21.15.80:6813/15036","state":["exists","up"]},{"osd":2,"uuid":"cbad909f-77f0-4494-b443-5a662e8c1a56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":15037},{"type":"v1","addr":"172.21.15.80:6801","nonce":15037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":15037},{"type":"v1","addr":"172.21.15.80:6803","nonce":15037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":15037},{"type":"v1","addr":"172.21.15.80:6807","nonce":15037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":15037},{"type":"v1","addr":"172.21.15.80:6805","nonce":15037}]},"public_addr":"172.21.15.80:6801/15037","cluster_addr":"172.21.15.80:6803/15037","heartbeat_back_addr":"172.21.15.80:6807/15037","heartbeat_front_addr":"172.21.15.80:6805/15037","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-09T22:26:17.958466+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-09T22:26:17.245815+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-09T22:26:17.779333+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-09T22:26:23.532 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2022-09-09T22:26:23.533 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2022-09-09T22:26:23.534 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2022-09-09T22:26:23.684 INFO:teuthology.orchestra.run.smithi080.stdout:34359738371 2022-09-09T22:26:23.689 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2022-09-09T22:26:23.698 INFO:teuthology.orchestra.run.smithi080.stdout:38654705666 2022-09-09T22:26:23.699 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2022-09-09T22:26:23.700 INFO:teuthology.orchestra.run.smithi080.stdout:42949672965 2022-09-09T22:26:23.701 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2022-09-09T22:26:24.029 INFO:teuthology.orchestra.run.smithi080.stdout:38654705665 2022-09-09T22:26:24.031 INFO:teuthology.orchestra.run.smithi080.stdout:34359738369 2022-09-09T22:26:24.044 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705666 got 38654705665 for osd.0 2022-09-09T22:26:24.045 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738369 for osd.1 2022-09-09T22:26:24.046 INFO:teuthology.orchestra.run.smithi080.stdout:42949672964 2022-09-09T22:26:24.062 INFO:tasks.ceph.ceph_manager.c2:need seq 42949672965 got 42949672964 for osd.2 2022-09-09T22:26:25.046 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2022-09-09T22:26:25.047 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2022-09-09T22:26:25.064 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2022-09-09T22:26:25.342 INFO:teuthology.orchestra.run.smithi080.stdout:38654705665 2022-09-09T22:26:25.356 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705666 got 38654705665 for osd.0 2022-09-09T22:26:25.361 INFO:teuthology.orchestra.run.smithi080.stdout:34359738369 2022-09-09T22:26:25.375 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738369 for osd.1 2022-09-09T22:26:25.400 INFO:teuthology.orchestra.run.smithi080.stdout:42949672964 2022-09-09T22:26:25.413 INFO:tasks.ceph.ceph_manager.c2:need seq 42949672965 got 42949672964 for osd.2 2022-09-09T22:26:26.357 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2022-09-09T22:26:26.377 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2022-09-09T22:26:26.414 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2022-09-09T22:26:26.683 INFO:teuthology.orchestra.run.smithi080.stdout:38654705666 2022-09-09T22:26:26.698 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705666 got 38654705666 for osd.0 2022-09-09T22:26:26.699 DEBUG:teuthology.parallel:result is None 2022-09-09T22:26:26.707 INFO:teuthology.orchestra.run.smithi080.stdout:42949672965 2022-09-09T22:26:26.708 INFO:teuthology.orchestra.run.smithi080.stdout:34359738371 2022-09-09T22:26:26.721 INFO:tasks.ceph.ceph_manager.c2:need seq 42949672965 got 42949672965 for osd.2 2022-09-09T22:26:26.723 DEBUG:teuthology.parallel:result is None 2022-09-09T22:26:26.724 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.1 2022-09-09T22:26:26.724 DEBUG:teuthology.parallel:result is None 2022-09-09T22:26:26.725 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2022-09-09T22:26:26.727 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2022-09-09T22:26:27.054 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:27.056 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2022-09-09T22:26:27.067 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2022-09-09T22:26:25.430325+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":25420,"kb_used_data":1468,"kb_used_omap":0,"kb_used_meta":23296,"kb_avail":314547380,"statfs":{"total":322122547200,"available":322096517120,"internally_reserved":0,"allocated":1503232,"data_stored":1260388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23855104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-09T22:26:23.466830+0000","last_change":"2022-09-09T22:26:22.536958+0000","last_active":"2022-09-09T22:26:23.466830+0000","last_peered":"2022-09-09T22:26:23.466830+0000","last_clean":"2022-09-09T22:26:23.466830+0000","last_became_active":"2022-09-09T22:26:22.536701+0000","last_became_peered":"2022-09-09T22:26:22.536701+0000","last_unstale":"2022-09-09T22:26:23.466830+0000","last_undegraded":"2022-09-09T22:26:23.466830+0000","last_fullsized":"2022-09-09T22:26:23.466830+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_clean_scrub_stamp":"2022-09-09T22:26:21.516200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:29:52.428866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-09T22:26:23.466791+0000","last_change":"2022-09-09T22:26:22.535193+0000","last_active":"2022-09-09T22:26:23.466791+0000","last_peered":"2022-09-09T22:26:23.466791+0000","last_clean":"2022-09-09T22:26:23.466791+0000","last_became_active":"2022-09-09T22:26:22.534917+0000","last_became_peered":"2022-09-09T22:26:22.534917+0000","last_unstale":"2022-09-09T22:26:23.466791+0000","last_undegraded":"2022-09-09T22:26:23.466791+0000","last_fullsized":"2022-09-09T22:26:23.466791+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_clean_scrub_stamp":"2022-09-09T22:26:21.516200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-10T23:57:39.127006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-09T22:26:22.539885+0000","last_change":"2022-09-09T22:26:22.535844+0000","last_active":"2022-09-09T22:26:22.539885+0000","last_peered":"2022-09-09T22:26:22.539885+0000","last_clean":"2022-09-09T22:26:22.539885+0000","last_became_active":"2022-09-09T22:26:22.535382+0000","last_became_peered":"2022-09-09T22:26:22.535382+0000","last_unstale":"2022-09-09T22:26:22.539885+0000","last_undegraded":"2022-09-09T22:26:22.539885+0000","last_fullsized":"2022-09-09T22:26:22.539885+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_clean_scrub_stamp":"2022-09-09T22:26:21.516200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:18:30.862570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-09T22:26:23.466742+0000","last_change":"2022-09-09T22:26:22.535283+0000","last_active":"2022-09-09T22:26:23.466742+0000","last_peered":"2022-09-09T22:26:23.466742+0000","last_clean":"2022-09-09T22:26:23.466742+0000","last_became_active":"2022-09-09T22:26:22.535025+0000","last_became_peered":"2022-09-09T22:26:22.535025+0000","last_unstale":"2022-09-09T22:26:23.466742+0000","last_undegraded":"2022-09-09T22:26:23.466742+0000","last_fullsized":"2022-09-09T22:26:23.466742+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_clean_scrub_stamp":"2022-09-09T22:26:21.516200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T02:55:50.665218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":136,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-09T22:26:23.234223+0000","last_change":"2022-09-09T22:26:22.536361+0000","last_active":"2022-09-09T22:26:23.234223+0000","last_peered":"2022-09-09T22:26:23.234223+0000","last_clean":"2022-09-09T22:26:23.234223+0000","last_became_active":"2022-09-09T22:26:22.535296+0000","last_became_peered":"2022-09-09T22:26:22.535296+0000","last_unstale":"2022-09-09T22:26:23.234223+0000","last_undegraded":"2022-09-09T22:26:23.234223+0000","last_fullsized":"2022-09-09T22:26:23.234223+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_clean_scrub_stamp":"2022-09-09T22:26:21.516200+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T00:43:12.434491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-09T22:26:23.466672+0000","last_change":"2022-09-09T22:26:22.535456+0000","last_active":"2022-09-09T22:26:23.466672+0000","last_peered":"2022-09-09T22:26:23.466672+0000","last_clean":"2022-09-09T22:26:23.466672+0000","last_became_active":"2022-09-09T22:26:22.535196+0000","last_became_peered":"2022-09-09T22:26:22.535196+0000","last_unstale":"2022-09-09T22:26:23.466672+0000","last_undegraded":"2022-09-09T22:26:23.466672+0000","last_fullsized":"2022-09-09T22:26:23.466672+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_clean_scrub_stamp":"2022-09-09T22:26:21.516200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T03:49:38.119067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-09T22:26:22.537999+0000","last_change":"2022-09-09T22:26:22.536097+0000","last_active":"2022-09-09T22:26:22.537999+0000","last_peered":"2022-09-09T22:26:22.537999+0000","last_clean":"2022-09-09T22:26:22.537999+0000","last_became_active":"2022-09-09T22:26:22.535807+0000","last_became_peered":"2022-09-09T22:26:22.535807+0000","last_unstale":"2022-09-09T22:26:22.537999+0000","last_undegraded":"2022-09-09T22:26:22.537999+0000","last_fullsized":"2022-09-09T22:26:22.537999+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_clean_scrub_stamp":"2022-09-09T22:26:21.516200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:19:27.415415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-09T22:26:22.537660+0000","last_change":"2022-09-09T22:26:22.534124+0000","last_active":"2022-09-09T22:26:22.537660+0000","last_peered":"2022-09-09T22:26:22.537660+0000","last_clean":"2022-09-09T22:26:22.537660+0000","last_became_active":"2022-09-09T22:26:22.533825+0000","last_became_peered":"2022-09-09T22:26:22.533825+0000","last_unstale":"2022-09-09T22:26:22.537660+0000","last_undegraded":"2022-09-09T22:26:22.537660+0000","last_fullsized":"2022-09-09T22:26:22.537660+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_clean_scrub_stamp":"2022-09-09T22:26:21.516200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:17:59.934497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-09T22:26:23.466613+0000","last_change":"2022-09-09T22:26:22.534999+0000","last_active":"2022-09-09T22:26:23.466613+0000","last_peered":"2022-09-09T22:26:23.466613+0000","last_clean":"2022-09-09T22:26:23.466613+0000","last_became_active":"2022-09-09T22:26:22.534735+0000","last_became_peered":"2022-09-09T22:26:22.534735+0000","last_unstale":"2022-09-09T22:26:23.466613+0000","last_undegraded":"2022-09-09T22:26:23.466613+0000","last_fullsized":"2022-09-09T22:26:23.466613+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_clean_scrub_stamp":"2022-09-09T22:26:21.516200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T02:30:05.552093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672965,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8880,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":104848720,"statfs":{"total":107374182400,"available":107365089280,"internally_reserved":0,"allocated":700416,"data_stored":617253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28999999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705666,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7660,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849940,"statfs":{"total":107374182400,"available":107366338560,"internally_reserved":0,"allocated":102400,"data_stored":25882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8880,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":104848720,"statfs":{"total":107374182400,"available":107365089280,"internally_reserved":0,"allocated":700416,"data_stored":617253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-09-09T22:26:27.069 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2022-09-09T22:26:27.358 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:27.361 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2022-09-09T22:26:27.372 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2022-09-09T22:26:25.430325+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":25420,"kb_used_data":1468,"kb_used_omap":0,"kb_used_meta":23296,"kb_avail":314547380,"statfs":{"total":322122547200,"available":322096517120,"internally_reserved":0,"allocated":1503232,"data_stored":1260388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23855104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-09T22:26:23.466830+0000","last_change":"2022-09-09T22:26:22.536958+0000","last_active":"2022-09-09T22:26:23.466830+0000","last_peered":"2022-09-09T22:26:23.466830+0000","last_clean":"2022-09-09T22:26:23.466830+0000","last_became_active":"2022-09-09T22:26:22.536701+0000","last_became_peered":"2022-09-09T22:26:22.536701+0000","last_unstale":"2022-09-09T22:26:23.466830+0000","last_undegraded":"2022-09-09T22:26:23.466830+0000","last_fullsized":"2022-09-09T22:26:23.466830+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_clean_scrub_stamp":"2022-09-09T22:26:21.516200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:29:52.428866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-09T22:26:23.466791+0000","last_change":"2022-09-09T22:26:22.535193+0000","last_active":"2022-09-09T22:26:23.466791+0000","last_peered":"2022-09-09T22:26:23.466791+0000","last_clean":"2022-09-09T22:26:23.466791+0000","last_became_active":"2022-09-09T22:26:22.534917+0000","last_became_peered":"2022-09-09T22:26:22.534917+0000","last_unstale":"2022-09-09T22:26:23.466791+0000","last_undegraded":"2022-09-09T22:26:23.466791+0000","last_fullsized":"2022-09-09T22:26:23.466791+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_clean_scrub_stamp":"2022-09-09T22:26:21.516200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-10T23:57:39.127006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-09T22:26:22.539885+0000","last_change":"2022-09-09T22:26:22.535844+0000","last_active":"2022-09-09T22:26:22.539885+0000","last_peered":"2022-09-09T22:26:22.539885+0000","last_clean":"2022-09-09T22:26:22.539885+0000","last_became_active":"2022-09-09T22:26:22.535382+0000","last_became_peered":"2022-09-09T22:26:22.535382+0000","last_unstale":"2022-09-09T22:26:22.539885+0000","last_undegraded":"2022-09-09T22:26:22.539885+0000","last_fullsized":"2022-09-09T22:26:22.539885+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_clean_scrub_stamp":"2022-09-09T22:26:21.516200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:18:30.862570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-09T22:26:23.466742+0000","last_change":"2022-09-09T22:26:22.535283+0000","last_active":"2022-09-09T22:26:23.466742+0000","last_peered":"2022-09-09T22:26:23.466742+0000","last_clean":"2022-09-09T22:26:23.466742+0000","last_became_active":"2022-09-09T22:26:22.535025+0000","last_became_peered":"2022-09-09T22:26:22.535025+0000","last_unstale":"2022-09-09T22:26:23.466742+0000","last_undegraded":"2022-09-09T22:26:23.466742+0000","last_fullsized":"2022-09-09T22:26:23.466742+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_clean_scrub_stamp":"2022-09-09T22:26:21.516200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T02:55:50.665218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":136,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-09T22:26:23.234223+0000","last_change":"2022-09-09T22:26:22.536361+0000","last_active":"2022-09-09T22:26:23.234223+0000","last_peered":"2022-09-09T22:26:23.234223+0000","last_clean":"2022-09-09T22:26:23.234223+0000","last_became_active":"2022-09-09T22:26:22.535296+0000","last_became_peered":"2022-09-09T22:26:22.535296+0000","last_unstale":"2022-09-09T22:26:23.234223+0000","last_undegraded":"2022-09-09T22:26:23.234223+0000","last_fullsized":"2022-09-09T22:26:23.234223+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_clean_scrub_stamp":"2022-09-09T22:26:21.516200+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T00:43:12.434491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-09T22:26:23.466672+0000","last_change":"2022-09-09T22:26:22.535456+0000","last_active":"2022-09-09T22:26:23.466672+0000","last_peered":"2022-09-09T22:26:23.466672+0000","last_clean":"2022-09-09T22:26:23.466672+0000","last_became_active":"2022-09-09T22:26:22.535196+0000","last_became_peered":"2022-09-09T22:26:22.535196+0000","last_unstale":"2022-09-09T22:26:23.466672+0000","last_undegraded":"2022-09-09T22:26:23.466672+0000","last_fullsized":"2022-09-09T22:26:23.466672+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_clean_scrub_stamp":"2022-09-09T22:26:21.516200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T03:49:38.119067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-09T22:26:22.537999+0000","last_change":"2022-09-09T22:26:22.536097+0000","last_active":"2022-09-09T22:26:22.537999+0000","last_peered":"2022-09-09T22:26:22.537999+0000","last_clean":"2022-09-09T22:26:22.537999+0000","last_became_active":"2022-09-09T22:26:22.535807+0000","last_became_peered":"2022-09-09T22:26:22.535807+0000","last_unstale":"2022-09-09T22:26:22.537999+0000","last_undegraded":"2022-09-09T22:26:22.537999+0000","last_fullsized":"2022-09-09T22:26:22.537999+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_clean_scrub_stamp":"2022-09-09T22:26:21.516200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:19:27.415415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-09T22:26:22.537660+0000","last_change":"2022-09-09T22:26:22.534124+0000","last_active":"2022-09-09T22:26:22.537660+0000","last_peered":"2022-09-09T22:26:22.537660+0000","last_clean":"2022-09-09T22:26:22.537660+0000","last_became_active":"2022-09-09T22:26:22.533825+0000","last_became_peered":"2022-09-09T22:26:22.533825+0000","last_unstale":"2022-09-09T22:26:22.537660+0000","last_undegraded":"2022-09-09T22:26:22.537660+0000","last_fullsized":"2022-09-09T22:26:22.537660+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_clean_scrub_stamp":"2022-09-09T22:26:21.516200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:17:59.934497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-09T22:26:23.466613+0000","last_change":"2022-09-09T22:26:22.534999+0000","last_active":"2022-09-09T22:26:23.466613+0000","last_peered":"2022-09-09T22:26:23.466613+0000","last_clean":"2022-09-09T22:26:23.466613+0000","last_became_active":"2022-09-09T22:26:22.534735+0000","last_became_peered":"2022-09-09T22:26:22.534735+0000","last_unstale":"2022-09-09T22:26:23.466613+0000","last_undegraded":"2022-09-09T22:26:23.466613+0000","last_fullsized":"2022-09-09T22:26:23.466613+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-09T22:26:21.516200+0000","last_clean_scrub_stamp":"2022-09-09T22:26:21.516200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T02:30:05.552093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672965,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8880,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":104848720,"statfs":{"total":107374182400,"available":107365089280,"internally_reserved":0,"allocated":700416,"data_stored":617253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28999999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705666,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7660,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849940,"statfs":{"total":107374182400,"available":107366338560,"internally_reserved":0,"allocated":102400,"data_stored":25882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8880,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":104848720,"statfs":{"total":107374182400,"available":107365089280,"internally_reserved":0,"allocated":700416,"data_stored":617253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-09-09T22:26:27.374 INFO:tasks.ceph.ceph_manager.c2:clean! 2022-09-09T22:26:27.374 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2022-09-09T22:26:27.375 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2022-09-09T22:26:27.376 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2022-09-09T22:26:27.689 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T22:26:27.690 INFO:teuthology.orchestra.run.smithi080.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-09-09T22:26:27.703 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2022-09-09T22:26:27.704 INFO:teuthology.run_tasks:Running task rgw... 2022-09-09T22:26:27.726 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2022-09-09T22:26:27.727 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2022-09-09T22:26:27.728 INFO:tasks.rgw:Creating data pools 2022-09-09T22:26:27.728 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2022-09-09T22:26:27.729 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2022-09-09T22:26:28.591 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'default.rgw.buckets.data' created 2022-09-09T22:26:28.603 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2022-09-09T22:26:29.615 INFO:teuthology.orchestra.run.smithi005.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2022-09-09T22:26:29.627 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2022-09-09T22:26:30.605 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'default.rgw.buckets.index' created 2022-09-09T22:26:30.616 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2022-09-09T22:26:31.621 INFO:teuthology.orchestra.run.smithi005.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2022-09-09T22:26:31.639 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2022-09-09T22:26:31.639 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2022-09-09T22:26:31.930 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'default.rgw.buckets.data' already exists 2022-09-09T22:26:31.940 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2022-09-09T22:26:32.607 INFO:teuthology.orchestra.run.smithi005.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2022-09-09T22:26:32.619 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2022-09-09T22:26:32.922 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'default.rgw.buckets.index' already exists 2022-09-09T22:26:32.938 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2022-09-09T22:26:33.614 INFO:teuthology.orchestra.run.smithi005.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2022-09-09T22:26:33.629 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2022-09-09T22:26:33.630 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2022-09-09T22:26:34.581 INFO:teuthology.orchestra.run.smithi080.stderr:pool 'default.rgw.buckets.data' created 2022-09-09T22:26:34.591 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2022-09-09T22:26:35.601 INFO:teuthology.orchestra.run.smithi080.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2022-09-09T22:26:35.612 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2022-09-09T22:26:36.593 INFO:teuthology.orchestra.run.smithi080.stderr:pool 'default.rgw.buckets.index' created 2022-09-09T22:26:36.605 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2022-09-09T22:26:37.614 INFO:teuthology.orchestra.run.smithi080.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2022-09-09T22:26:37.628 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2022-09-09T22:26:37.628 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2022-09-09T22:26:37.918 INFO:teuthology.orchestra.run.smithi080.stderr:pool 'default.rgw.buckets.data' already exists 2022-09-09T22:26:37.929 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2022-09-09T22:26:38.625 INFO:teuthology.orchestra.run.smithi080.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2022-09-09T22:26:38.636 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2022-09-09T22:26:38.931 INFO:teuthology.orchestra.run.smithi080.stderr:pool 'default.rgw.buckets.index' already exists 2022-09-09T22:26:38.941 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2022-09-09T22:26:39.640 INFO:teuthology.orchestra.run.smithi080.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2022-09-09T22:26:39.651 DEBUG:tasks.rgw:Pools created 2022-09-09T22:26:39.652 INFO:tasks.rgw:Configuring compression type = random 2022-09-09T22:26:39.652 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2022-09-09T22:26:39.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2022-09-09T22:26:39.654 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2022-09-09T22:26:39.695 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:26:39.695 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:26:39.705 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:39.701+0000 7f96e7b1adc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:41.649 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.644+0000 7f96e7b1adc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:41.649 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.644+0000 7f96e7b1adc0 20 realm 2022-09-09T22:26:41.649 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.644+0000 7f96e7b1adc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:41.652 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.648+0000 7f96e7b1adc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:41.652 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.648+0000 7f96e7b1adc0 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-09-09T22:26:41.652 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.648+0000 7f96e7b1adc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:41.653 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.648+0000 7f96e7b1adc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:41.653 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.648+0000 7f96e7b1adc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:41.656 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.652+0000 7f96e7b1adc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:41.656 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.652+0000 7f96e7b1adc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:41.656 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.652+0000 7f96e7b1adc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:41.657 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.652+0000 7f96e7b1adc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:41.657 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.652+0000 7f96e7b1adc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:41.657 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.652+0000 7f96e7b1adc0 10 Creating default zonegroup 2022-09-09T22:26:41.657 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.652+0000 7f96e7b1adc0 10 couldn't find old data placement pools config, setting up new ones for the zone 2022-09-09T22:26:41.671 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.668+0000 7f96e7b1adc0 10 WARNING: store->list_zones() returned r=-2 2022-09-09T22:26:41.671 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.668+0000 7f96e7b1adc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:41.672 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.668+0000 7f96e7b1adc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:41.674 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.672+0000 7f96e7b1adc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:41.675 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.672+0000 7f96e7b1adc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:41.675 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.672+0000 7f96e7b1adc0 10 could not read realm id: (2) No such file or directory 2022-09-09T22:26:41.676 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.672+0000 7f96e7b1adc0 10 WARNING: failed to set zone as default, r=-22 2022-09-09T22:26:41.676 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.672+0000 7f96e7b1adc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:41.676 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.672+0000 7f96e7b1adc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:41.678 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.676+0000 7f96e7b1adc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:41.679 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.676+0000 7f96e7b1adc0 20 rados_obj.operate() r=0 bl.length=873 2022-09-09T22:26:41.679 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.676+0000 7f96e7b1adc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:41.680 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.676+0000 7f96e7b1adc0 20 rados_obj.operate() r=0 bl.length=398 2022-09-09T22:26:41.680 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.676+0000 7f96e7b1adc0 10 Using default name default 2022-09-09T22:26:41.680 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.676+0000 7f96e7b1adc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:41.681 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.676+0000 7f96e7b1adc0 20 rados_obj.operate() r=0 bl.length=46 2022-09-09T22:26:41.682 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.676+0000 7f96e7b1adc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:41.682 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.676+0000 7f96e7b1adc0 20 rados_obj.operate() r=0 bl.length=873 2022-09-09T22:26:41.682 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.676+0000 7f96e7b1adc0 20 zone default found 2022-09-09T22:26:41.682 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.676+0000 7f96e7b1adc0 4 Realm: () 2022-09-09T22:26:41.683 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.676+0000 7f96e7b1adc0 4 ZoneGroup: default (6e1bfd8f-09f6-48dc-a80a-d8edef1e99da) 2022-09-09T22:26:41.683 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.676+0000 7f96e7b1adc0 4 Zone: default (b1c42eb5-2bf1-4ea7-9122-853c40986081) 2022-09-09T22:26:41.683 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.676+0000 7f96e7b1adc0 10 cannot find current period zonegroup using local zonegroup configuration 2022-09-09T22:26:41.683 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.676+0000 7f96e7b1adc0 20 zonegroup default 2022-09-09T22:26:41.684 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.676+0000 7f96e7b1adc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:41.684 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.676+0000 7f96e7b1adc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:41.684 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:41.676+0000 7f96e7b1adc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:43.671 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:43.667+0000 7f96e7b1adc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:43.671 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:43.667+0000 7f96e7b1adc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:43.672 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:43.667+0000 7f96e7b1adc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:43.672 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:43.667+0000 7f96e7b1adc0 20 started sync module instance, tier type = 2022-09-09T22:26:43.673 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:43.667+0000 7f96e7b1adc0 20 started zone id=b1c42eb5-2bf1-4ea7-9122-853c40986081 (name=default) with tier type = 2022-09-09T22:26:45.708 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:45.703+0000 7f96e7b1adc0 20 add_watcher() i=0 2022-09-09T22:26:45.878 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:45.703+0000 7f96e7b1adc0 20 add_watcher() i=1 2022-09-09T22:26:45.879 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:45.703+0000 7f96e7b1adc0 20 add_watcher() i=2 2022-09-09T22:26:45.879 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:45.703+0000 7f96e7b1adc0 20 add_watcher() i=3 2022-09-09T22:26:45.880 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:45.703+0000 7f96e7b1adc0 20 add_watcher() i=4 2022-09-09T22:26:45.880 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:45.703+0000 7f96e7b1adc0 20 add_watcher() i=5 2022-09-09T22:26:45.881 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:45.703+0000 7f96e7b1adc0 20 add_watcher() i=6 2022-09-09T22:26:45.883 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:45.703+0000 7f96e7b1adc0 20 add_watcher() i=7 2022-09-09T22:26:45.884 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:45.703+0000 7f96e7b1adc0 2 all 8 watchers are set, enabling cache 2022-09-09T22:26:45.884 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:45.703+0000 7f96e7b1adc0 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2022-09-09T22:26:45.884 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:45.703+0000 7f96e7b1adc0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2022-09-09T22:26:45.885 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:45.703+0000 7f96e7b1adc0 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-09T22:26:45.885 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:45.703+0000 7f96e7b1adc0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2022-09-09T22:26:45.885 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:45.703+0000 7f96e7b1adc0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2022-09-09T22:26:45.885 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:45.703+0000 7f96e7b1adc0 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-09T22:26:45.886 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:45.703+0000 7f96e7b1adc0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2022-09-09T22:26:45.886 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:45.703+0000 7f96e7b1adc0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2022-09-09T22:26:45.886 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:45.703+0000 7f96e7b1adc0 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-09T22:26:45.887 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:45.707+0000 7f96e7b1adc0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2022-09-09T22:26:45.887 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:45.707+0000 7f96e7b1adc0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2022-09-09T22:26:45.887 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:45.707+0000 7f96e7b1adc0 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-09T22:26:45.887 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:45.707+0000 7f96e7b1adc0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2022-09-09T22:26:45.888 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:45.707+0000 7f96e7b1adc0 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1203 entering 2022-09-09T22:26:45.888 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:45.707+0000 7f96e7b1adc0 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-09T22:26:45.888 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:45.711+0000 7f96d0aa9700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-09T22:26:45.888 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:45.711+0000 7f96e7b1adc0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-09T22:26:45.889 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:45.711+0000 7f96e7b1adc0 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-09T22:26:48.691 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.687+0000 7f96e7b1adc0 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2022-09-09T22:26:48.692 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.687+0000 7f96e7b1adc0 5 note: GC not initialized 2022-09-09T22:26:48.767 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.763+0000 7f96cc2a0700 20 reqs_thread_entry: start 2022-09-09T22:26:48.768 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.763+0000 7f96e7b1adc0 20 init_complete bucket index max shards: 11 2022-09-09T22:26:48.768 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.763+0000 7f96e7b1adc0 10 rgw notify: Started notification manager with: 1 workers 2022-09-09T22:26:48.768 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.763+0000 7f96ca29c700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 9 22:27:19 2022 2022-09-09T22:26:48.768 INFO:teuthology.orchestra.run.smithi005.stderr: 2022-09-09T22:26:48.769 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.763+0000 7f96c9a7a700 20 reqs_thread_entry: start 2022-09-09T22:26:48.797 INFO:teuthology.orchestra.run.smithi005.stdout:[] 2022-09-09T22:26:48.798 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.791+0000 7f96e7b1adc0 20 remove_watcher() i=0 2022-09-09T22:26:48.798 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.791+0000 7f96e7b1adc0 2 removed watcher, disabling cache 2022-09-09T22:26:48.799 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.795+0000 7f96e7b1adc0 20 remove_watcher() i=1 2022-09-09T22:26:48.800 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.795+0000 7f96e7b1adc0 20 remove_watcher() i=2 2022-09-09T22:26:48.802 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.799+0000 7f96e7b1adc0 20 remove_watcher() i=3 2022-09-09T22:26:48.803 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.799+0000 7f96e7b1adc0 20 remove_watcher() i=4 2022-09-09T22:26:48.805 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.799+0000 7f96e7b1adc0 20 remove_watcher() i=5 2022-09-09T22:26:48.807 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.803+0000 7f96e7b1adc0 20 remove_watcher() i=6 2022-09-09T22:26:48.808 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.803+0000 7f96e7b1adc0 20 remove_watcher() i=7 2022-09-09T22:26:48.810 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.803+0000 7f96d0aa9700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-09T22:26:48.818 DEBUG:tasks.util.rgw: json result: [] 2022-09-09T22:26:48.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-09T22:26:48.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-09T22:26:48.820 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2022-09-09T22:26:48.910 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:26:48.911 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:26:48.922 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.915+0000 7f77f6e10dc0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-09T22:26:48.923 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.915+0000 7f77f6e10dc0 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-09T22:26:48.930 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.923+0000 7f77dc598700 20 reqs_thread_entry: start 2022-09-09T22:26:48.931 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.923+0000 7f77f6e10dc0 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2022-09-09T22:26:48.931 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.923+0000 7f77f6e10dc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:48.934 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.927+0000 7f77f6e10dc0 20 rados_obj.operate() r=0 bl.length=46 2022-09-09T22:26:48.935 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.927+0000 7f77f6e10dc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:48.935 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.927+0000 7f77f6e10dc0 20 rados_obj.operate() r=0 bl.length=873 2022-09-09T22:26:48.955 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.947+0000 7f77f6e10dc0 20 RGWRados::pool_iterate: got zone_info.b1c42eb5-2bf1-4ea7-9122-853c40986081 2022-09-09T22:26:48.957 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.947+0000 7f77f6e10dc0 20 RGWRados::pool_iterate: got zonegroup_info.6e1bfd8f-09f6-48dc-a80a-d8edef1e99da 2022-09-09T22:26:48.957 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.947+0000 7f77f6e10dc0 20 RGWRados::pool_iterate: got zone_names.default 2022-09-09T22:26:48.957 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.947+0000 7f77f6e10dc0 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-09-09T22:26:48.958 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.947+0000 7f77f6e10dc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:48.958 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.951+0000 7f77f6e10dc0 20 rados_obj.operate() r=0 bl.length=873 2022-09-09T22:26:48.958 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.951+0000 7f77f6e10dc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:48.959 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.951+0000 7f77f6e10dc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:48.959 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.951+0000 7f77f6e10dc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:48.959 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.951+0000 7f77f6e10dc0 20 rados_obj.operate() r=0 bl.length=46 2022-09-09T22:26:48.959 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.951+0000 7f77f6e10dc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:48.960 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:48.951+0000 7f77f6e10dc0 20 rados_obj.operate() r=0 bl.length=398 2022-09-09T22:26:48.966 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:26:48.966 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "b1c42eb5-2bf1-4ea7-9122-853c40986081", 2022-09-09T22:26:48.967 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "default", 2022-09-09T22:26:48.967 INFO:teuthology.orchestra.run.smithi005.stdout: "domain_root": "default.rgw.meta:root", 2022-09-09T22:26:48.967 INFO:teuthology.orchestra.run.smithi005.stdout: "control_pool": "default.rgw.control", 2022-09-09T22:26:48.967 INFO:teuthology.orchestra.run.smithi005.stdout: "gc_pool": "default.rgw.log:gc", 2022-09-09T22:26:48.968 INFO:teuthology.orchestra.run.smithi005.stdout: "lc_pool": "default.rgw.log:lc", 2022-09-09T22:26:48.968 INFO:teuthology.orchestra.run.smithi005.stdout: "log_pool": "default.rgw.log", 2022-09-09T22:26:48.968 INFO:teuthology.orchestra.run.smithi005.stdout: "intent_log_pool": "default.rgw.log:intent", 2022-09-09T22:26:48.968 INFO:teuthology.orchestra.run.smithi005.stdout: "usage_log_pool": "default.rgw.log:usage", 2022-09-09T22:26:48.969 INFO:teuthology.orchestra.run.smithi005.stdout: "roles_pool": "default.rgw.meta:roles", 2022-09-09T22:26:48.969 INFO:teuthology.orchestra.run.smithi005.stdout: "reshard_pool": "default.rgw.log:reshard", 2022-09-09T22:26:48.969 INFO:teuthology.orchestra.run.smithi005.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2022-09-09T22:26:48.969 INFO:teuthology.orchestra.run.smithi005.stdout: "user_email_pool": "default.rgw.meta:users.email", 2022-09-09T22:26:48.970 INFO:teuthology.orchestra.run.smithi005.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2022-09-09T22:26:48.970 INFO:teuthology.orchestra.run.smithi005.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2022-09-09T22:26:48.970 INFO:teuthology.orchestra.run.smithi005.stdout: "otp_pool": "default.rgw.otp", 2022-09-09T22:26:48.971 INFO:teuthology.orchestra.run.smithi005.stdout: "system_key": { 2022-09-09T22:26:48.971 INFO:teuthology.orchestra.run.smithi005.stdout: "access_key": "", 2022-09-09T22:26:48.971 INFO:teuthology.orchestra.run.smithi005.stdout: "secret_key": "" 2022-09-09T22:26:48.971 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:26:48.972 INFO:teuthology.orchestra.run.smithi005.stdout: "placement_pools": [ 2022-09-09T22:26:48.972 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:26:48.972 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "default-placement", 2022-09-09T22:26:48.973 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:26:48.974 INFO:teuthology.orchestra.run.smithi005.stdout: "index_pool": "default.rgw.buckets.index", 2022-09-09T22:26:48.974 INFO:teuthology.orchestra.run.smithi005.stdout: "storage_classes": { 2022-09-09T22:26:48.974 INFO:teuthology.orchestra.run.smithi005.stdout: "STANDARD": { 2022-09-09T22:26:48.974 INFO:teuthology.orchestra.run.smithi005.stdout: "data_pool": "default.rgw.buckets.data", 2022-09-09T22:26:48.975 INFO:teuthology.orchestra.run.smithi005.stdout: "compression_type": "random" 2022-09-09T22:26:48.975 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:26:48.975 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:26:48.975 INFO:teuthology.orchestra.run.smithi005.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2022-09-09T22:26:48.976 INFO:teuthology.orchestra.run.smithi005.stdout: "index_type": 0 2022-09-09T22:26:48.976 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:26:48.976 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:26:48.976 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:26:48.977 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_id": "", 2022-09-09T22:26:48.977 INFO:teuthology.orchestra.run.smithi005.stdout: "notif_pool": "default.rgw.log:notif" 2022-09-09T22:26:48.977 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:26:48.979 DEBUG:tasks.util.rgw: json result: {'id': 'b1c42eb5-2bf1-4ea7-9122-853c40986081', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2022-09-09T22:26:48.979 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2022-09-09T22:26:48.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2022-09-09T22:26:48.980 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2022-09-09T22:26:49.023 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:26:49.023 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:26:49.035 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.031+0000 7f331c54adc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:49.037 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.031+0000 7f331c54adc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:49.038 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.031+0000 7f331c54adc0 20 realm 2022-09-09T22:26:49.038 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.031+0000 7f331c54adc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:49.038 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.031+0000 7f331c54adc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:49.038 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.031+0000 7f331c54adc0 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-09-09T22:26:49.039 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.031+0000 7f331c54adc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:49.039 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.031+0000 7f331c54adc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:49.039 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.031+0000 7f331c54adc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:49.040 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.031+0000 7f331c54adc0 20 rados_obj.operate() r=0 bl.length=46 2022-09-09T22:26:49.040 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.031+0000 7f331c54adc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:49.040 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.035+0000 7f331c54adc0 20 rados_obj.operate() r=0 bl.length=889 2022-09-09T22:26:49.041 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.035+0000 7f331c54adc0 20 searching for the correct realm 2022-09-09T22:26:49.062 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.055+0000 7f331c54adc0 20 RGWRados::pool_iterate: got zone_info.b1c42eb5-2bf1-4ea7-9122-853c40986081 2022-09-09T22:26:49.063 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.055+0000 7f331c54adc0 20 RGWRados::pool_iterate: got zonegroup_info.6e1bfd8f-09f6-48dc-a80a-d8edef1e99da 2022-09-09T22:26:49.064 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.055+0000 7f331c54adc0 20 RGWRados::pool_iterate: got zone_names.default 2022-09-09T22:26:49.064 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.055+0000 7f331c54adc0 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-09-09T22:26:49.064 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.055+0000 7f331c54adc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:49.065 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.055+0000 7f331c54adc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:49.065 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.055+0000 7f331c54adc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:49.070 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.059+0000 7f331c54adc0 20 rados_obj.operate() r=0 bl.length=46 2022-09-09T22:26:49.070 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.059+0000 7f331c54adc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:49.072 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.059+0000 7f331c54adc0 20 rados_obj.operate() r=0 bl.length=398 2022-09-09T22:26:49.072 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.059+0000 7f331c54adc0 20 zone default found 2022-09-09T22:26:49.072 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.059+0000 7f331c54adc0 4 Realm: () 2022-09-09T22:26:49.073 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.059+0000 7f331c54adc0 4 ZoneGroup: default (6e1bfd8f-09f6-48dc-a80a-d8edef1e99da) 2022-09-09T22:26:49.073 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.059+0000 7f331c54adc0 4 Zone: default (b1c42eb5-2bf1-4ea7-9122-853c40986081) 2022-09-09T22:26:49.073 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.059+0000 7f331c54adc0 10 cannot find current period zonegroup using local zonegroup configuration 2022-09-09T22:26:49.073 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.059+0000 7f331c54adc0 20 zonegroup default 2022-09-09T22:26:49.074 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.059+0000 7f331c54adc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:49.074 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.059+0000 7f331c54adc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:49.074 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.059+0000 7f331c54adc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:49.074 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.059+0000 7f331c54adc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:49.075 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.059+0000 7f331c54adc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:49.075 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.059+0000 7f331c54adc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:49.075 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.059+0000 7f331c54adc0 20 started sync module instance, tier type = 2022-09-09T22:26:49.075 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.059+0000 7f331c54adc0 20 started zone id=b1c42eb5-2bf1-4ea7-9122-853c40986081 (name=default) with tier type = 2022-09-09T22:26:49.085 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.079+0000 7f331c54adc0 20 add_watcher() i=0 2022-09-09T22:26:49.085 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.079+0000 7f331c54adc0 20 add_watcher() i=1 2022-09-09T22:26:49.085 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.079+0000 7f331c54adc0 20 add_watcher() i=2 2022-09-09T22:26:49.085 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.079+0000 7f331c54adc0 20 add_watcher() i=3 2022-09-09T22:26:49.086 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.079+0000 7f331c54adc0 20 add_watcher() i=4 2022-09-09T22:26:49.086 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.079+0000 7f331c54adc0 20 add_watcher() i=5 2022-09-09T22:26:49.086 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.079+0000 7f331c54adc0 20 add_watcher() i=6 2022-09-09T22:26:49.086 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.079+0000 7f331c54adc0 20 add_watcher() i=7 2022-09-09T22:26:49.087 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.079+0000 7f331c54adc0 2 all 8 watchers are set, enabling cache 2022-09-09T22:26:49.088 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.083+0000 7f33054d9700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-09T22:26:49.089 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.083+0000 7f331c54adc0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-09T22:26:49.089 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.083+0000 7f331c54adc0 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-09T22:26:49.093 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.087+0000 7f331c54adc0 5 note: GC not initialized 2022-09-09T22:26:49.096 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.091+0000 7f3300cd0700 20 reqs_thread_entry: start 2022-09-09T22:26:49.097 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.091+0000 7f331c54adc0 20 init_complete bucket index max shards: 11 2022-09-09T22:26:49.097 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.091+0000 7f331c54adc0 10 rgw notify: Started notification manager with: 1 workers 2022-09-09T22:26:49.098 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.091+0000 7f32fe4cb700 20 reqs_thread_entry: start 2022-09-09T22:26:49.098 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.091+0000 7f32feccc700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 9 22:27:19 2022 2022-09-09T22:26:49.099 INFO:teuthology.orchestra.run.smithi005.stderr: 2022-09-09T22:26:49.118 INFO:teuthology.orchestra.run.smithi005.stdout:[] 2022-09-09T22:26:49.119 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.111+0000 7f331c54adc0 20 remove_watcher() i=0 2022-09-09T22:26:49.120 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.111+0000 7f331c54adc0 2 removed watcher, disabling cache 2022-09-09T22:26:49.121 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.115+0000 7f331c54adc0 20 remove_watcher() i=1 2022-09-09T22:26:49.122 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.119+0000 7f331c54adc0 20 remove_watcher() i=2 2022-09-09T22:26:49.124 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.119+0000 7f331c54adc0 20 remove_watcher() i=3 2022-09-09T22:26:49.127 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.123+0000 7f331c54adc0 20 remove_watcher() i=4 2022-09-09T22:26:49.129 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.123+0000 7f331c54adc0 20 remove_watcher() i=5 2022-09-09T22:26:49.131 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.127+0000 7f331c54adc0 20 remove_watcher() i=6 2022-09-09T22:26:49.133 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.127+0000 7f331c54adc0 20 remove_watcher() i=7 2022-09-09T22:26:49.135 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.131+0000 7f33054d9700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-09T22:26:49.145 DEBUG:tasks.util.rgw: json result: [] 2022-09-09T22:26:49.146 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-09T22:26:49.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-09T22:26:49.147 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2022-09-09T22:26:49.236 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:26:49.236 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:26:49.245 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.239+0000 7f6759c88dc0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-09T22:26:49.246 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.239+0000 7f6759c88dc0 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-09T22:26:49.251 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.247+0000 7f673f410700 20 reqs_thread_entry: start 2022-09-09T22:26:49.251 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.247+0000 7f6759c88dc0 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2022-09-09T22:26:49.252 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.247+0000 7f6759c88dc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:49.252 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.247+0000 7f6759c88dc0 20 rados_obj.operate() r=0 bl.length=46 2022-09-09T22:26:49.253 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.247+0000 7f6759c88dc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:49.253 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.247+0000 7f6759c88dc0 20 rados_obj.operate() r=0 bl.length=889 2022-09-09T22:26:49.275 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.271+0000 7f6759c88dc0 20 RGWRados::pool_iterate: got zone_info.b1c42eb5-2bf1-4ea7-9122-853c40986081 2022-09-09T22:26:49.275 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.271+0000 7f6759c88dc0 20 RGWRados::pool_iterate: got zonegroup_info.6e1bfd8f-09f6-48dc-a80a-d8edef1e99da 2022-09-09T22:26:49.276 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.271+0000 7f6759c88dc0 20 RGWRados::pool_iterate: got zone_names.default 2022-09-09T22:26:49.276 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.271+0000 7f6759c88dc0 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-09-09T22:26:49.276 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.271+0000 7f6759c88dc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:49.278 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.271+0000 7f6759c88dc0 20 rados_obj.operate() r=0 bl.length=889 2022-09-09T22:26:49.278 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.271+0000 7f6759c88dc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:49.278 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.271+0000 7f6759c88dc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:49.278 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.271+0000 7f6759c88dc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:49.279 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.271+0000 7f6759c88dc0 20 rados_obj.operate() r=0 bl.length=46 2022-09-09T22:26:49.279 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.271+0000 7f6759c88dc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:49.279 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:49.271+0000 7f6759c88dc0 20 rados_obj.operate() r=0 bl.length=398 2022-09-09T22:26:49.286 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:26:49.287 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "b1c42eb5-2bf1-4ea7-9122-853c40986081", 2022-09-09T22:26:49.287 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "default", 2022-09-09T22:26:49.287 INFO:teuthology.orchestra.run.smithi005.stdout: "domain_root": "default.rgw.meta:root", 2022-09-09T22:26:49.287 INFO:teuthology.orchestra.run.smithi005.stdout: "control_pool": "default.rgw.control", 2022-09-09T22:26:49.288 INFO:teuthology.orchestra.run.smithi005.stdout: "gc_pool": "default.rgw.log:gc", 2022-09-09T22:26:49.288 INFO:teuthology.orchestra.run.smithi005.stdout: "lc_pool": "default.rgw.log:lc", 2022-09-09T22:26:49.288 INFO:teuthology.orchestra.run.smithi005.stdout: "log_pool": "default.rgw.log", 2022-09-09T22:26:49.289 INFO:teuthology.orchestra.run.smithi005.stdout: "intent_log_pool": "default.rgw.log:intent", 2022-09-09T22:26:49.289 INFO:teuthology.orchestra.run.smithi005.stdout: "usage_log_pool": "default.rgw.log:usage", 2022-09-09T22:26:49.289 INFO:teuthology.orchestra.run.smithi005.stdout: "roles_pool": "default.rgw.meta:roles", 2022-09-09T22:26:49.289 INFO:teuthology.orchestra.run.smithi005.stdout: "reshard_pool": "default.rgw.log:reshard", 2022-09-09T22:26:49.289 INFO:teuthology.orchestra.run.smithi005.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2022-09-09T22:26:49.290 INFO:teuthology.orchestra.run.smithi005.stdout: "user_email_pool": "default.rgw.meta:users.email", 2022-09-09T22:26:49.290 INFO:teuthology.orchestra.run.smithi005.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2022-09-09T22:26:49.291 INFO:teuthology.orchestra.run.smithi005.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2022-09-09T22:26:49.291 INFO:teuthology.orchestra.run.smithi005.stdout: "otp_pool": "default.rgw.otp", 2022-09-09T22:26:49.292 INFO:teuthology.orchestra.run.smithi005.stdout: "system_key": { 2022-09-09T22:26:49.293 INFO:teuthology.orchestra.run.smithi005.stdout: "access_key": "", 2022-09-09T22:26:49.293 INFO:teuthology.orchestra.run.smithi005.stdout: "secret_key": "" 2022-09-09T22:26:49.294 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:26:49.294 INFO:teuthology.orchestra.run.smithi005.stdout: "placement_pools": [ 2022-09-09T22:26:49.295 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:26:49.295 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "default-placement", 2022-09-09T22:26:49.296 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:26:49.296 INFO:teuthology.orchestra.run.smithi005.stdout: "index_pool": "default.rgw.buckets.index", 2022-09-09T22:26:49.297 INFO:teuthology.orchestra.run.smithi005.stdout: "storage_classes": { 2022-09-09T22:26:49.297 INFO:teuthology.orchestra.run.smithi005.stdout: "STANDARD": { 2022-09-09T22:26:49.298 INFO:teuthology.orchestra.run.smithi005.stdout: "data_pool": "default.rgw.buckets.data", 2022-09-09T22:26:49.298 INFO:teuthology.orchestra.run.smithi005.stdout: "compression_type": "random" 2022-09-09T22:26:49.299 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:26:49.299 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:26:49.299 INFO:teuthology.orchestra.run.smithi005.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2022-09-09T22:26:49.299 INFO:teuthology.orchestra.run.smithi005.stdout: "index_type": 0 2022-09-09T22:26:49.300 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:26:49.300 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:26:49.300 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:26:49.300 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_id": "", 2022-09-09T22:26:49.301 INFO:teuthology.orchestra.run.smithi005.stdout: "notif_pool": "default.rgw.log:notif" 2022-09-09T22:26:49.301 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:26:49.303 DEBUG:tasks.util.rgw: json result: {'id': 'b1c42eb5-2bf1-4ea7-9122-853c40986081', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2022-09-09T22:26:49.303 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2022-09-09T22:26:49.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2022-09-09T22:26:49.304 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2022-09-09T22:26:49.345 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:26:49.346 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:26:49.359 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:49.354+0000 7f42cef5edc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:50.464 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.458+0000 7f42cef5edc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:50.465 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.458+0000 7f42cef5edc0 20 realm 2022-09-09T22:26:50.465 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.458+0000 7f42cef5edc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:50.466 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.458+0000 7f42cef5edc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:50.466 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.458+0000 7f42cef5edc0 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-09-09T22:26:50.466 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.458+0000 7f42cef5edc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:50.467 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.462+0000 7f42cef5edc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:50.467 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.462+0000 7f42cef5edc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:50.468 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.462+0000 7f42cef5edc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:50.468 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.462+0000 7f42cef5edc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:50.468 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.462+0000 7f42cef5edc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:50.469 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.462+0000 7f42cef5edc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:50.469 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.462+0000 7f42cef5edc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:50.469 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.462+0000 7f42cef5edc0 10 Creating default zonegroup 2022-09-09T22:26:50.469 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.462+0000 7f42cef5edc0 10 couldn't find old data placement pools config, setting up new ones for the zone 2022-09-09T22:26:50.480 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.474+0000 7f42cef5edc0 10 WARNING: store->list_zones() returned r=-2 2022-09-09T22:26:50.480 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.474+0000 7f42cef5edc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:50.481 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.474+0000 7f42cef5edc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:50.482 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.478+0000 7f42cef5edc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:50.483 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.478+0000 7f42cef5edc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:50.483 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.478+0000 7f42cef5edc0 10 could not read realm id: (2) No such file or directory 2022-09-09T22:26:50.483 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.478+0000 7f42cef5edc0 10 WARNING: failed to set zone as default, r=-22 2022-09-09T22:26:50.484 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.478+0000 7f42cef5edc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:50.484 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.478+0000 7f42cef5edc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:50.485 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.482+0000 7f42cef5edc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:50.486 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.482+0000 7f42cef5edc0 20 rados_obj.operate() r=0 bl.length=873 2022-09-09T22:26:50.486 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.482+0000 7f42cef5edc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:50.488 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.482+0000 7f42cef5edc0 20 rados_obj.operate() r=0 bl.length=398 2022-09-09T22:26:50.488 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.482+0000 7f42cef5edc0 10 Using default name default 2022-09-09T22:26:50.488 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.482+0000 7f42cef5edc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:50.490 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.482+0000 7f42cef5edc0 20 rados_obj.operate() r=0 bl.length=46 2022-09-09T22:26:50.490 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.482+0000 7f42cef5edc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:50.490 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.482+0000 7f42cef5edc0 20 rados_obj.operate() r=0 bl.length=873 2022-09-09T22:26:50.490 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.482+0000 7f42cef5edc0 20 zone default found 2022-09-09T22:26:50.491 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.482+0000 7f42cef5edc0 4 Realm: () 2022-09-09T22:26:50.491 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.482+0000 7f42cef5edc0 4 ZoneGroup: default (1ecb35fe-e13d-49d1-8197-482fec54126c) 2022-09-09T22:26:50.491 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.482+0000 7f42cef5edc0 4 Zone: default (7ca550f9-8225-4860-9ded-c693cff8398b) 2022-09-09T22:26:50.491 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.482+0000 7f42cef5edc0 10 cannot find current period zonegroup using local zonegroup configuration 2022-09-09T22:26:50.492 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.482+0000 7f42cef5edc0 20 zonegroup default 2022-09-09T22:26:50.492 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.482+0000 7f42cef5edc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:50.492 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.482+0000 7f42cef5edc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:50.492 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:50.482+0000 7f42cef5edc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:52.491 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:52.486+0000 7f42cef5edc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:52.491 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:52.486+0000 7f42cef5edc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:52.497 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:52.486+0000 7f42cef5edc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:52.497 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:52.486+0000 7f42cef5edc0 20 started sync module instance, tier type = 2022-09-09T22:26:52.498 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:52.486+0000 7f42cef5edc0 20 started zone id=7ca550f9-8225-4860-9ded-c693cff8398b (name=default) with tier type = 2022-09-09T22:26:54.526 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:54.518+0000 7f42cef5edc0 20 add_watcher() i=0 2022-09-09T22:26:54.526 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:54.518+0000 7f42cef5edc0 20 add_watcher() i=1 2022-09-09T22:26:54.527 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:54.518+0000 7f42cef5edc0 20 add_watcher() i=2 2022-09-09T22:26:54.527 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:54.518+0000 7f42cef5edc0 20 add_watcher() i=3 2022-09-09T22:26:54.528 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:54.518+0000 7f42cef5edc0 20 add_watcher() i=4 2022-09-09T22:26:54.529 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:54.518+0000 7f42cef5edc0 20 add_watcher() i=5 2022-09-09T22:26:54.529 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:54.518+0000 7f42cef5edc0 20 add_watcher() i=6 2022-09-09T22:26:54.534 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:54.522+0000 7f42cef5edc0 20 add_watcher() i=7 2022-09-09T22:26:54.534 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:54.522+0000 7f42cef5edc0 2 all 8 watchers are set, enabling cache 2022-09-09T22:26:54.536 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:54.522+0000 7f42cef5edc0 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2022-09-09T22:26:54.536 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:54.522+0000 7f42cef5edc0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2022-09-09T22:26:54.537 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:54.522+0000 7f42cef5edc0 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-09T22:26:54.537 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:54.522+0000 7f42cef5edc0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2022-09-09T22:26:54.538 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:54.522+0000 7f42cef5edc0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2022-09-09T22:26:54.538 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:54.522+0000 7f42cef5edc0 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-09T22:26:54.539 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:54.522+0000 7f42cef5edc0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2022-09-09T22:26:54.540 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:54.522+0000 7f42cef5edc0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2022-09-09T22:26:54.541 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:54.522+0000 7f42cef5edc0 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-09T22:26:54.541 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:54.522+0000 7f42cef5edc0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2022-09-09T22:26:54.542 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:54.522+0000 7f42cef5edc0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2022-09-09T22:26:54.542 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:54.522+0000 7f42cef5edc0 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-09T22:26:54.542 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:54.526+0000 7f42cef5edc0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2022-09-09T22:26:54.543 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:54.526+0000 7f42cef5edc0 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1203 entering 2022-09-09T22:26:54.543 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:54.526+0000 7f42cef5edc0 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-09T22:26:54.545 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:54.534+0000 7f42b7eed700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-09T22:26:54.545 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:54.534+0000 7f42cef5edc0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-09T22:26:54.545 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:54.534+0000 7f42cef5edc0 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-09T22:26:57.518 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.510+0000 7f42cef5edc0 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2022-09-09T22:26:57.518 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.510+0000 7f42cef5edc0 5 note: GC not initialized 2022-09-09T22:26:57.543 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.534+0000 7f42b36e4700 20 reqs_thread_entry: start 2022-09-09T22:26:57.544 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.534+0000 7f42cef5edc0 20 init_complete bucket index max shards: 11 2022-09-09T22:26:57.544 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.534+0000 7f42cef5edc0 10 rgw notify: Started notification manager with: 1 workers 2022-09-09T22:26:57.545 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.534+0000 7f42b16e0700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 9 22:27:27 2022 2022-09-09T22:26:57.545 INFO:teuthology.orchestra.run.smithi080.stderr: 2022-09-09T22:26:57.546 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.534+0000 7f42b0ebe700 20 reqs_thread_entry: start 2022-09-09T22:26:57.575 INFO:teuthology.orchestra.run.smithi080.stdout:[] 2022-09-09T22:26:57.576 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.566+0000 7f42cef5edc0 20 remove_watcher() i=0 2022-09-09T22:26:57.576 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.566+0000 7f42cef5edc0 2 removed watcher, disabling cache 2022-09-09T22:26:57.576 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.570+0000 7f42cef5edc0 20 remove_watcher() i=1 2022-09-09T22:26:57.579 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.570+0000 7f42cef5edc0 20 remove_watcher() i=2 2022-09-09T22:26:57.579 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.574+0000 7f42cef5edc0 20 remove_watcher() i=3 2022-09-09T22:26:57.580 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.574+0000 7f42cef5edc0 20 remove_watcher() i=4 2022-09-09T22:26:57.582 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.578+0000 7f42cef5edc0 20 remove_watcher() i=5 2022-09-09T22:26:57.583 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.578+0000 7f42cef5edc0 20 remove_watcher() i=6 2022-09-09T22:26:57.584 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.578+0000 7f42cef5edc0 20 remove_watcher() i=7 2022-09-09T22:26:57.586 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.582+0000 7f42b7eed700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-09T22:26:57.596 DEBUG:tasks.util.rgw: json result: [] 2022-09-09T22:26:57.596 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-09T22:26:57.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-09T22:26:57.597 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2022-09-09T22:26:57.686 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:26:57.688 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:26:57.697 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.690+0000 7f69a000cdc0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-09T22:26:57.698 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.690+0000 7f69a000cdc0 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-09T22:26:57.708 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.698+0000 7f6985794700 20 reqs_thread_entry: start 2022-09-09T22:26:57.709 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.698+0000 7f69a000cdc0 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2022-09-09T22:26:57.709 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.698+0000 7f69a000cdc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:57.709 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.698+0000 7f69a000cdc0 20 rados_obj.operate() r=0 bl.length=46 2022-09-09T22:26:57.710 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.698+0000 7f69a000cdc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:57.710 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.702+0000 7f69a000cdc0 20 rados_obj.operate() r=0 bl.length=873 2022-09-09T22:26:57.728 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.722+0000 7f69a000cdc0 20 RGWRados::pool_iterate: got zonegroup_info.1ecb35fe-e13d-49d1-8197-482fec54126c 2022-09-09T22:26:57.729 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.722+0000 7f69a000cdc0 20 RGWRados::pool_iterate: got zone_info.7ca550f9-8225-4860-9ded-c693cff8398b 2022-09-09T22:26:57.729 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.722+0000 7f69a000cdc0 20 RGWRados::pool_iterate: got zone_names.default 2022-09-09T22:26:57.730 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.722+0000 7f69a000cdc0 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-09-09T22:26:57.730 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.722+0000 7f69a000cdc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:57.730 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.722+0000 7f69a000cdc0 20 rados_obj.operate() r=0 bl.length=873 2022-09-09T22:26:57.731 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.722+0000 7f69a000cdc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:57.732 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.722+0000 7f69a000cdc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:57.732 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.722+0000 7f69a000cdc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:57.733 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.722+0000 7f69a000cdc0 20 rados_obj.operate() r=0 bl.length=46 2022-09-09T22:26:57.733 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.722+0000 7f69a000cdc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:57.733 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.726+0000 7f69a000cdc0 20 rados_obj.operate() r=0 bl.length=398 2022-09-09T22:26:57.738 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:26:57.738 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "7ca550f9-8225-4860-9ded-c693cff8398b", 2022-09-09T22:26:57.739 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "default", 2022-09-09T22:26:57.739 INFO:teuthology.orchestra.run.smithi080.stdout: "domain_root": "default.rgw.meta:root", 2022-09-09T22:26:57.739 INFO:teuthology.orchestra.run.smithi080.stdout: "control_pool": "default.rgw.control", 2022-09-09T22:26:57.740 INFO:teuthology.orchestra.run.smithi080.stdout: "gc_pool": "default.rgw.log:gc", 2022-09-09T22:26:57.740 INFO:teuthology.orchestra.run.smithi080.stdout: "lc_pool": "default.rgw.log:lc", 2022-09-09T22:26:57.740 INFO:teuthology.orchestra.run.smithi080.stdout: "log_pool": "default.rgw.log", 2022-09-09T22:26:57.740 INFO:teuthology.orchestra.run.smithi080.stdout: "intent_log_pool": "default.rgw.log:intent", 2022-09-09T22:26:57.741 INFO:teuthology.orchestra.run.smithi080.stdout: "usage_log_pool": "default.rgw.log:usage", 2022-09-09T22:26:57.741 INFO:teuthology.orchestra.run.smithi080.stdout: "roles_pool": "default.rgw.meta:roles", 2022-09-09T22:26:57.741 INFO:teuthology.orchestra.run.smithi080.stdout: "reshard_pool": "default.rgw.log:reshard", 2022-09-09T22:26:57.742 INFO:teuthology.orchestra.run.smithi080.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2022-09-09T22:26:57.742 INFO:teuthology.orchestra.run.smithi080.stdout: "user_email_pool": "default.rgw.meta:users.email", 2022-09-09T22:26:57.742 INFO:teuthology.orchestra.run.smithi080.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2022-09-09T22:26:57.742 INFO:teuthology.orchestra.run.smithi080.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2022-09-09T22:26:57.743 INFO:teuthology.orchestra.run.smithi080.stdout: "otp_pool": "default.rgw.otp", 2022-09-09T22:26:57.743 INFO:teuthology.orchestra.run.smithi080.stdout: "system_key": { 2022-09-09T22:26:57.743 INFO:teuthology.orchestra.run.smithi080.stdout: "access_key": "", 2022-09-09T22:26:57.743 INFO:teuthology.orchestra.run.smithi080.stdout: "secret_key": "" 2022-09-09T22:26:57.743 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:26:57.744 INFO:teuthology.orchestra.run.smithi080.stdout: "placement_pools": [ 2022-09-09T22:26:57.744 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:26:57.744 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "default-placement", 2022-09-09T22:26:57.744 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:26:57.745 INFO:teuthology.orchestra.run.smithi080.stdout: "index_pool": "default.rgw.buckets.index", 2022-09-09T22:26:57.745 INFO:teuthology.orchestra.run.smithi080.stdout: "storage_classes": { 2022-09-09T22:26:57.745 INFO:teuthology.orchestra.run.smithi080.stdout: "STANDARD": { 2022-09-09T22:26:57.745 INFO:teuthology.orchestra.run.smithi080.stdout: "data_pool": "default.rgw.buckets.data", 2022-09-09T22:26:57.746 INFO:teuthology.orchestra.run.smithi080.stdout: "compression_type": "random" 2022-09-09T22:26:57.746 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:26:57.748 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:26:57.748 INFO:teuthology.orchestra.run.smithi080.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2022-09-09T22:26:57.749 INFO:teuthology.orchestra.run.smithi080.stdout: "index_type": 0 2022-09-09T22:26:57.749 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:26:57.749 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:26:57.750 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:26:57.750 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "", 2022-09-09T22:26:57.751 INFO:teuthology.orchestra.run.smithi080.stdout: "notif_pool": "default.rgw.log:notif" 2022-09-09T22:26:57.751 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:26:57.753 DEBUG:tasks.util.rgw: json result: {'id': '7ca550f9-8225-4860-9ded-c693cff8398b', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2022-09-09T22:26:57.754 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2022-09-09T22:26:57.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2022-09-09T22:26:57.756 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2022-09-09T22:26:57.798 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:26:57.798 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:26:57.808 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.802+0000 7f1896fc5dc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:57.809 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.806+0000 7f1896fc5dc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:57.810 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.806+0000 7f1896fc5dc0 20 realm 2022-09-09T22:26:57.810 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.806+0000 7f1896fc5dc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:57.812 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.806+0000 7f1896fc5dc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:57.812 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.806+0000 7f1896fc5dc0 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-09-09T22:26:57.812 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.806+0000 7f1896fc5dc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:57.812 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.806+0000 7f1896fc5dc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:57.813 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.806+0000 7f1896fc5dc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:57.813 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.806+0000 7f1896fc5dc0 20 rados_obj.operate() r=0 bl.length=46 2022-09-09T22:26:57.813 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.806+0000 7f1896fc5dc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:57.814 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.806+0000 7f1896fc5dc0 20 rados_obj.operate() r=0 bl.length=889 2022-09-09T22:26:57.814 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.810+0000 7f1896fc5dc0 20 searching for the correct realm 2022-09-09T22:26:57.834 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.826+0000 7f1896fc5dc0 20 RGWRados::pool_iterate: got zonegroup_info.1ecb35fe-e13d-49d1-8197-482fec54126c 2022-09-09T22:26:57.835 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.826+0000 7f1896fc5dc0 20 RGWRados::pool_iterate: got zone_info.7ca550f9-8225-4860-9ded-c693cff8398b 2022-09-09T22:26:57.835 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.826+0000 7f1896fc5dc0 20 RGWRados::pool_iterate: got zone_names.default 2022-09-09T22:26:57.836 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.826+0000 7f1896fc5dc0 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-09-09T22:26:57.836 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.826+0000 7f1896fc5dc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:57.836 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.830+0000 7f1896fc5dc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:57.837 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.830+0000 7f1896fc5dc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:57.837 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.830+0000 7f1896fc5dc0 20 rados_obj.operate() r=0 bl.length=46 2022-09-09T22:26:57.838 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.830+0000 7f1896fc5dc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:57.839 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.830+0000 7f1896fc5dc0 20 rados_obj.operate() r=0 bl.length=398 2022-09-09T22:26:57.840 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.830+0000 7f1896fc5dc0 20 zone default found 2022-09-09T22:26:57.840 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.830+0000 7f1896fc5dc0 4 Realm: () 2022-09-09T22:26:57.841 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.830+0000 7f1896fc5dc0 4 ZoneGroup: default (1ecb35fe-e13d-49d1-8197-482fec54126c) 2022-09-09T22:26:57.841 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.830+0000 7f1896fc5dc0 4 Zone: default (7ca550f9-8225-4860-9ded-c693cff8398b) 2022-09-09T22:26:57.842 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.830+0000 7f1896fc5dc0 10 cannot find current period zonegroup using local zonegroup configuration 2022-09-09T22:26:57.842 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.830+0000 7f1896fc5dc0 20 zonegroup default 2022-09-09T22:26:57.843 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.830+0000 7f1896fc5dc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:57.847 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.830+0000 7f1896fc5dc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:57.847 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.830+0000 7f1896fc5dc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:57.848 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.830+0000 7f1896fc5dc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:57.848 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.830+0000 7f1896fc5dc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:57.848 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.830+0000 7f1896fc5dc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:57.849 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.830+0000 7f1896fc5dc0 20 started sync module instance, tier type = 2022-09-09T22:26:57.849 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.830+0000 7f1896fc5dc0 20 started zone id=7ca550f9-8225-4860-9ded-c693cff8398b (name=default) with tier type = 2022-09-09T22:26:57.860 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.854+0000 7f1896fc5dc0 20 add_watcher() i=0 2022-09-09T22:26:57.860 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.854+0000 7f1896fc5dc0 20 add_watcher() i=1 2022-09-09T22:26:57.861 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.854+0000 7f1896fc5dc0 20 add_watcher() i=2 2022-09-09T22:26:57.861 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.854+0000 7f1896fc5dc0 20 add_watcher() i=3 2022-09-09T22:26:57.861 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.854+0000 7f1896fc5dc0 20 add_watcher() i=4 2022-09-09T22:26:57.861 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.854+0000 7f1896fc5dc0 20 add_watcher() i=5 2022-09-09T22:26:57.862 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.854+0000 7f1896fc5dc0 20 add_watcher() i=6 2022-09-09T22:26:57.862 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.858+0000 7f1896fc5dc0 20 add_watcher() i=7 2022-09-09T22:26:57.862 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.858+0000 7f1896fc5dc0 2 all 8 watchers are set, enabling cache 2022-09-09T22:26:57.865 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.858+0000 7f187ff54700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-09T22:26:57.865 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.858+0000 7f1896fc5dc0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-09T22:26:57.866 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.858+0000 7f1896fc5dc0 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-09T22:26:57.873 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.866+0000 7f1896fc5dc0 5 note: GC not initialized 2022-09-09T22:26:57.876 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.870+0000 7f187b74b700 20 reqs_thread_entry: start 2022-09-09T22:26:57.876 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.870+0000 7f1896fc5dc0 20 init_complete bucket index max shards: 11 2022-09-09T22:26:57.877 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.870+0000 7f1896fc5dc0 10 rgw notify: Started notification manager with: 1 workers 2022-09-09T22:26:57.877 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.870+0000 7f1879747700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 9 22:27:28 2022 2022-09-09T22:26:57.878 INFO:teuthology.orchestra.run.smithi080.stderr: 2022-09-09T22:26:57.878 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.870+0000 7f1878f25700 20 reqs_thread_entry: start 2022-09-09T22:26:57.898 INFO:teuthology.orchestra.run.smithi080.stdout:[] 2022-09-09T22:26:57.899 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.894+0000 7f1896fc5dc0 20 remove_watcher() i=0 2022-09-09T22:26:57.900 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.894+0000 7f1896fc5dc0 2 removed watcher, disabling cache 2022-09-09T22:26:57.901 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.894+0000 7f1896fc5dc0 20 remove_watcher() i=1 2022-09-09T22:26:57.902 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.898+0000 7f1896fc5dc0 20 remove_watcher() i=2 2022-09-09T22:26:57.904 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.898+0000 7f1896fc5dc0 20 remove_watcher() i=3 2022-09-09T22:26:57.908 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.902+0000 7f1896fc5dc0 20 remove_watcher() i=4 2022-09-09T22:26:57.909 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.902+0000 7f1896fc5dc0 20 remove_watcher() i=5 2022-09-09T22:26:57.911 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.906+0000 7f1896fc5dc0 20 remove_watcher() i=6 2022-09-09T22:26:57.913 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.906+0000 7f1896fc5dc0 20 remove_watcher() i=7 2022-09-09T22:26:57.915 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.910+0000 7f187ff54700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-09T22:26:57.935 DEBUG:tasks.util.rgw: json result: [] 2022-09-09T22:26:57.936 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-09T22:26:57.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-09T22:26:57.938 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2022-09-09T22:26:57.993 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:26:57.994 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:26:57.998 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.994+0000 7fe443a8ddc0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-09T22:26:57.998 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:57.994+0000 7fe443a8ddc0 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-09T22:26:58.015 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:58.002+0000 7fe429215700 20 reqs_thread_entry: start 2022-09-09T22:26:58.016 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:58.002+0000 7fe443a8ddc0 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2022-09-09T22:26:58.016 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:58.002+0000 7fe443a8ddc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:58.016 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:58.002+0000 7fe443a8ddc0 20 rados_obj.operate() r=0 bl.length=46 2022-09-09T22:26:58.017 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:58.002+0000 7fe443a8ddc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:58.017 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:58.002+0000 7fe443a8ddc0 20 rados_obj.operate() r=0 bl.length=889 2022-09-09T22:26:58.032 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:58.022+0000 7fe443a8ddc0 20 RGWRados::pool_iterate: got zonegroup_info.1ecb35fe-e13d-49d1-8197-482fec54126c 2022-09-09T22:26:58.033 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:58.022+0000 7fe443a8ddc0 20 RGWRados::pool_iterate: got zone_info.7ca550f9-8225-4860-9ded-c693cff8398b 2022-09-09T22:26:58.033 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:58.022+0000 7fe443a8ddc0 20 RGWRados::pool_iterate: got zone_names.default 2022-09-09T22:26:58.034 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:58.022+0000 7fe443a8ddc0 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-09-09T22:26:58.035 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:58.022+0000 7fe443a8ddc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:58.035 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:58.026+0000 7fe443a8ddc0 20 rados_obj.operate() r=0 bl.length=889 2022-09-09T22:26:58.037 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:58.026+0000 7fe443a8ddc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:58.037 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:58.026+0000 7fe443a8ddc0 20 rados_obj.operate() r=-2 bl.length=0 2022-09-09T22:26:58.038 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:58.026+0000 7fe443a8ddc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:58.038 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:58.026+0000 7fe443a8ddc0 20 rados_obj.operate() r=0 bl.length=46 2022-09-09T22:26:58.039 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:58.026+0000 7fe443a8ddc0 20 rados->read ofs=0 len=0 2022-09-09T22:26:58.039 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:26:58.026+0000 7fe443a8ddc0 20 rados_obj.operate() r=0 bl.length=398 2022-09-09T22:26:58.043 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:26:58.043 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "7ca550f9-8225-4860-9ded-c693cff8398b", 2022-09-09T22:26:58.043 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "default", 2022-09-09T22:26:58.044 INFO:teuthology.orchestra.run.smithi080.stdout: "domain_root": "default.rgw.meta:root", 2022-09-09T22:26:58.044 INFO:teuthology.orchestra.run.smithi080.stdout: "control_pool": "default.rgw.control", 2022-09-09T22:26:58.045 INFO:teuthology.orchestra.run.smithi080.stdout: "gc_pool": "default.rgw.log:gc", 2022-09-09T22:26:58.046 INFO:teuthology.orchestra.run.smithi080.stdout: "lc_pool": "default.rgw.log:lc", 2022-09-09T22:26:58.046 INFO:teuthology.orchestra.run.smithi080.stdout: "log_pool": "default.rgw.log", 2022-09-09T22:26:58.047 INFO:teuthology.orchestra.run.smithi080.stdout: "intent_log_pool": "default.rgw.log:intent", 2022-09-09T22:26:58.047 INFO:teuthology.orchestra.run.smithi080.stdout: "usage_log_pool": "default.rgw.log:usage", 2022-09-09T22:26:58.047 INFO:teuthology.orchestra.run.smithi080.stdout: "roles_pool": "default.rgw.meta:roles", 2022-09-09T22:26:58.048 INFO:teuthology.orchestra.run.smithi080.stdout: "reshard_pool": "default.rgw.log:reshard", 2022-09-09T22:26:58.048 INFO:teuthology.orchestra.run.smithi080.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2022-09-09T22:26:58.049 INFO:teuthology.orchestra.run.smithi080.stdout: "user_email_pool": "default.rgw.meta:users.email", 2022-09-09T22:26:58.049 INFO:teuthology.orchestra.run.smithi080.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2022-09-09T22:26:58.050 INFO:teuthology.orchestra.run.smithi080.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2022-09-09T22:26:58.051 INFO:teuthology.orchestra.run.smithi080.stdout: "otp_pool": "default.rgw.otp", 2022-09-09T22:26:58.052 INFO:teuthology.orchestra.run.smithi080.stdout: "system_key": { 2022-09-09T22:26:58.052 INFO:teuthology.orchestra.run.smithi080.stdout: "access_key": "", 2022-09-09T22:26:58.053 INFO:teuthology.orchestra.run.smithi080.stdout: "secret_key": "" 2022-09-09T22:26:58.053 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:26:58.054 INFO:teuthology.orchestra.run.smithi080.stdout: "placement_pools": [ 2022-09-09T22:26:58.054 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:26:58.055 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "default-placement", 2022-09-09T22:26:58.055 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:26:58.056 INFO:teuthology.orchestra.run.smithi080.stdout: "index_pool": "default.rgw.buckets.index", 2022-09-09T22:26:58.057 INFO:teuthology.orchestra.run.smithi080.stdout: "storage_classes": { 2022-09-09T22:26:58.057 INFO:teuthology.orchestra.run.smithi080.stdout: "STANDARD": { 2022-09-09T22:26:58.057 INFO:teuthology.orchestra.run.smithi080.stdout: "data_pool": "default.rgw.buckets.data", 2022-09-09T22:26:58.058 INFO:teuthology.orchestra.run.smithi080.stdout: "compression_type": "random" 2022-09-09T22:26:58.058 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:26:58.059 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:26:58.061 INFO:teuthology.orchestra.run.smithi080.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2022-09-09T22:26:58.062 INFO:teuthology.orchestra.run.smithi080.stdout: "index_type": 0 2022-09-09T22:26:58.062 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:26:58.063 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:26:58.063 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:26:58.064 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "", 2022-09-09T22:26:58.064 INFO:teuthology.orchestra.run.smithi080.stdout: "notif_pool": "default.rgw.log:notif" 2022-09-09T22:26:58.066 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:26:58.067 DEBUG:tasks.util.rgw: json result: {'id': '7ca550f9-8225-4860-9ded-c693cff8398b', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2022-09-09T22:26:58.067 INFO:tasks.rgw:Starting rgw... 2022-09-09T22:26:58.068 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2022-09-09T22:26:58.069 INFO:tasks.rgw:Using beast as radosgw frontend 2022-09-09T22:26:58.070 INFO:tasks.rgw.c1.client.0:Restarting daemon 2022-09-09T22:26:58.071 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2022-09-09T22:26:58.075 INFO:tasks.rgw.c1.client.0:Started 2022-09-09T22:26:58.076 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2022-09-09T22:26:58.076 INFO:tasks.rgw:Using beast as radosgw frontend 2022-09-09T22:26:58.077 INFO:tasks.rgw.c1.client.1:Restarting daemon 2022-09-09T22:26:58.077 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2022-09-09T22:26:58.080 INFO:tasks.rgw.c1.client.1:Started 2022-09-09T22:26:58.081 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2022-09-09T22:26:58.081 INFO:tasks.rgw:Using beast as radosgw frontend 2022-09-09T22:26:58.082 INFO:tasks.rgw.c2.client.0:Restarting daemon 2022-09-09T22:26:58.083 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2022-09-09T22:26:58.088 INFO:tasks.rgw.c2.client.0:Started 2022-09-09T22:26:58.089 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2022-09-09T22:26:58.089 INFO:tasks.rgw:Using beast as radosgw frontend 2022-09-09T22:26:58.090 INFO:tasks.rgw.c2.client.1:Restarting daemon 2022-09-09T22:26:58.090 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2022-09-09T22:26:58.099 INFO:tasks.rgw.c2.client.1:Started 2022-09-09T22:26:58.099 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi005.front.sepia.ceph.com:8000/ 2022-09-09T22:26:58.100 DEBUG:teuthology.orchestra.run.smithi005:> curl http://smithi005.front.sepia.ceph.com:8000/ 2022-09-09T22:26:58.141 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-09T22:26:58.143 INFO:teuthology.orchestra.run.smithi005.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-09T22:26:58.144 INFO:teuthology.orchestra.run.smithi005.stderr: Dload Upload Total Spent Left Speed 2022-09-09T22:26:58.144 INFO:teuthology.orchestra.run.smithi005.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2022-09-09T22:26:58.146 INFO:teuthology.orchestra.run.smithi005.stderr:curl: (7) Failed to connect to smithi005.front.sepia.ceph.com port 8000: Connection refused 2022-09-09T22:26:59.147 DEBUG:teuthology.orchestra.run.smithi005:> curl http://smithi005.front.sepia.ceph.com:8000/ 2022-09-09T22:26:59.168 INFO:teuthology.orchestra.run.smithi005.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-09T22:26:59.170 INFO:teuthology.orchestra.run.smithi005.stderr: Dload Upload Total Spent Left Speed 2022-09-09T22:26:59.174 INFO:teuthology.orchestra.run.smithi005.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 53500 0 --:--:-- --:--:-- --:--:-- 53500 2022-09-09T22:26:59.176 INFO:teuthology.orchestra.run.smithi005.stdout:anonymous 2022-09-09T22:26:59.177 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi005.front.sepia.ceph.com:8001/ 2022-09-09T22:26:59.178 DEBUG:teuthology.orchestra.run.smithi005:> curl http://smithi005.front.sepia.ceph.com:8001/ 2022-09-09T22:26:59.197 INFO:teuthology.orchestra.run.smithi005.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-09T22:26:59.197 INFO:teuthology.orchestra.run.smithi005.stderr: Dload Upload Total Spent Left Speed 2022-09-09T22:26:59.202 INFO:teuthology.orchestra.run.smithi005.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 53500 0 --:--:-- --:--:-- --:--:-- 53500 2022-09-09T22:26:59.203 INFO:teuthology.orchestra.run.smithi005.stdout:anonymous 2022-09-09T22:26:59.204 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi080.front.sepia.ceph.com:8000/ 2022-09-09T22:26:59.204 DEBUG:teuthology.orchestra.run.smithi080:> curl http://smithi080.front.sepia.ceph.com:8000/ 2022-09-09T22:26:59.240 INFO:teuthology.orchestra.run.smithi080.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-09T22:26:59.241 INFO:teuthology.orchestra.run.smithi080.stderr: Dload Upload Total Spent Left Speed 2022-09-09T22:26:59.243 INFO:teuthology.orchestra.run.smithi080.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 71333 0 --:--:-- --:--:-- --:--:-- 71333 2022-09-09T22:26:59.244 INFO:teuthology.orchestra.run.smithi080.stdout:anonymous 2022-09-09T22:26:59.245 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi080.front.sepia.ceph.com:8001/ 2022-09-09T22:26:59.246 DEBUG:teuthology.orchestra.run.smithi080:> curl http://smithi080.front.sepia.ceph.com:8001/ 2022-09-09T22:26:59.307 INFO:teuthology.orchestra.run.smithi080.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-09T22:26:59.308 INFO:teuthology.orchestra.run.smithi080.stderr: Dload Upload Total Spent Left Speed 2022-09-09T22:26:59.308 INFO:teuthology.orchestra.run.smithi080.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2022-09-09T22:26:59.310 INFO:teuthology.orchestra.run.smithi080.stdout:anonymous 2022-09-09T22:26:59.311 INFO:teuthology.run_tasks:Running task rgw-multisite... 2022-09-09T22:26:59.742 INFO:tasks.rgw_multisite:creating realm.. 2022-09-09T22:26:59.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:26:59.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:26:59.743 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-09T22:26:59.787 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:26:59.788 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:26:59.861 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:26:59.862 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T22:26:59.862 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-realm", 2022-09-09T22:26:59.863 INFO:teuthology.orchestra.run.smithi005.stdout: "current_period": "588f0e6a-ed72-4f68-b3ff-9c4cf92a2fa8", 2022-09-09T22:26:59.863 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1 2022-09-09T22:26:59.863 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:26:59.865 INFO:tasks.rgw_multisite:creating master zonegroup.. 2022-09-09T22:26:59.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:26:59.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:26:59.868 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-09T22:26:59.913 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:26:59.913 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:26:59.937 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:26:59.931+0000 7fd9e5342dc0 -1 old format 2022-09-09T22:26:59.949 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:26:59.949 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "test-zonegroup", 2022-09-09T22:26:59.949 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zonegroup", 2022-09-09T22:26:59.950 INFO:teuthology.orchestra.run.smithi005.stdout: "api_name": "", 2022-09-09T22:26:59.950 INFO:teuthology.orchestra.run.smithi005.stdout: "is_master": true, 2022-09-09T22:26:59.950 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T22:26:59.951 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi005.front.sepia.ceph.com:8000" 2022-09-09T22:26:59.951 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:26:59.951 INFO:teuthology.orchestra.run.smithi005.stdout: "hostnames": [], 2022-09-09T22:26:59.951 INFO:teuthology.orchestra.run.smithi005.stdout: "hostnames_s3website": [], 2022-09-09T22:26:59.952 INFO:teuthology.orchestra.run.smithi005.stdout: "master_zone": "", 2022-09-09T22:26:59.952 INFO:teuthology.orchestra.run.smithi005.stdout: "zones": [], 2022-09-09T22:26:59.952 INFO:teuthology.orchestra.run.smithi005.stdout: "placement_targets": [], 2022-09-09T22:26:59.953 INFO:teuthology.orchestra.run.smithi005.stdout: "default_placement": "", 2022-09-09T22:26:59.953 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T22:26:59.953 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_policy": { 2022-09-09T22:26:59.953 INFO:teuthology.orchestra.run.smithi005.stdout: "groups": [] 2022-09-09T22:26:59.954 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:26:59.954 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled_features": [ 2022-09-09T22:26:59.954 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T22:26:59.954 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:26:59.955 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:26:59.956 INFO:tasks.rgw_multisite:creating master zone.. 2022-09-09T22:26:59.956 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '3U0V7WQYC6MLEXED', '--secret', 'jA4iNzYKmfzg78oKBQTy4mNpjERzasVo', '--endpoints', 'http://smithi005.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:26:59.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '3U0V7WQYC6MLEXED', '--secret', 'jA4iNzYKmfzg78oKBQTy4mNpjERzasVo', '--endpoints', 'http://smithi005.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:26:59.957 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key 3U0V7WQYC6MLEXED --secret jA4iNzYKmfzg78oKBQTy4mNpjERzasVo --endpoints http://smithi005.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-09T22:27:00.052 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:27:00.053 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:27:00.121 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:27:00.122 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T22:27:00.122 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zone1", 2022-09-09T22:27:00.123 INFO:teuthology.orchestra.run.smithi005.stdout: "domain_root": "test-zone1.rgw.meta:root", 2022-09-09T22:27:00.123 INFO:teuthology.orchestra.run.smithi005.stdout: "control_pool": "test-zone1.rgw.control", 2022-09-09T22:27:00.124 INFO:teuthology.orchestra.run.smithi005.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2022-09-09T22:27:00.124 INFO:teuthology.orchestra.run.smithi005.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2022-09-09T22:27:00.125 INFO:teuthology.orchestra.run.smithi005.stdout: "log_pool": "test-zone1.rgw.log", 2022-09-09T22:27:00.125 INFO:teuthology.orchestra.run.smithi005.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2022-09-09T22:27:00.125 INFO:teuthology.orchestra.run.smithi005.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2022-09-09T22:27:00.126 INFO:teuthology.orchestra.run.smithi005.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2022-09-09T22:27:00.126 INFO:teuthology.orchestra.run.smithi005.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2022-09-09T22:27:00.127 INFO:teuthology.orchestra.run.smithi005.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2022-09-09T22:27:00.127 INFO:teuthology.orchestra.run.smithi005.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2022-09-09T22:27:00.128 INFO:teuthology.orchestra.run.smithi005.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2022-09-09T22:27:00.128 INFO:teuthology.orchestra.run.smithi005.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2022-09-09T22:27:00.129 INFO:teuthology.orchestra.run.smithi005.stdout: "otp_pool": "test-zone1.rgw.otp", 2022-09-09T22:27:00.129 INFO:teuthology.orchestra.run.smithi005.stdout: "system_key": { 2022-09-09T22:27:00.129 INFO:teuthology.orchestra.run.smithi005.stdout: "access_key": "3U0V7WQYC6MLEXED", 2022-09-09T22:27:00.130 INFO:teuthology.orchestra.run.smithi005.stdout: "secret_key": "jA4iNzYKmfzg78oKBQTy4mNpjERzasVo" 2022-09-09T22:27:00.130 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:27:00.131 INFO:teuthology.orchestra.run.smithi005.stdout: "placement_pools": [ 2022-09-09T22:27:00.131 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:27:00.132 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "default-placement", 2022-09-09T22:27:00.132 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:27:00.132 INFO:teuthology.orchestra.run.smithi005.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2022-09-09T22:27:00.133 INFO:teuthology.orchestra.run.smithi005.stdout: "storage_classes": { 2022-09-09T22:27:00.133 INFO:teuthology.orchestra.run.smithi005.stdout: "STANDARD": { 2022-09-09T22:27:00.134 INFO:teuthology.orchestra.run.smithi005.stdout: "data_pool": "test-zone1.rgw.buckets.data" 2022-09-09T22:27:00.134 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:27:00.135 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:27:00.135 INFO:teuthology.orchestra.run.smithi005.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2022-09-09T22:27:00.135 INFO:teuthology.orchestra.run.smithi005.stdout: "index_type": 0 2022-09-09T22:27:00.136 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:27:00.136 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:27:00.137 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:27:00.137 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T22:27:00.138 INFO:teuthology.orchestra.run.smithi005.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2022-09-09T22:27:00.138 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:27:00.140 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2022-09-09T22:27:00.547 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'test-zone1.rgw.buckets.data' created 2022-09-09T22:27:00.557 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2022-09-09T22:27:01.555 INFO:teuthology.orchestra.run.smithi005.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2022-09-09T22:27:01.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:27:01.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:27:01.583 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-09T22:27:01.625 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:27:01.626 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:27:01.677 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:27:01.677 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T22:27:01.678 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zone1", 2022-09-09T22:27:01.678 INFO:teuthology.orchestra.run.smithi005.stdout: "domain_root": "test-zone1.rgw.meta:root", 2022-09-09T22:27:01.678 INFO:teuthology.orchestra.run.smithi005.stdout: "control_pool": "test-zone1.rgw.control", 2022-09-09T22:27:01.678 INFO:teuthology.orchestra.run.smithi005.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2022-09-09T22:27:01.679 INFO:teuthology.orchestra.run.smithi005.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2022-09-09T22:27:01.679 INFO:teuthology.orchestra.run.smithi005.stdout: "log_pool": "test-zone1.rgw.log", 2022-09-09T22:27:01.679 INFO:teuthology.orchestra.run.smithi005.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2022-09-09T22:27:01.679 INFO:teuthology.orchestra.run.smithi005.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2022-09-09T22:27:01.680 INFO:teuthology.orchestra.run.smithi005.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2022-09-09T22:27:01.680 INFO:teuthology.orchestra.run.smithi005.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2022-09-09T22:27:01.680 INFO:teuthology.orchestra.run.smithi005.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2022-09-09T22:27:01.681 INFO:teuthology.orchestra.run.smithi005.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2022-09-09T22:27:01.681 INFO:teuthology.orchestra.run.smithi005.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2022-09-09T22:27:01.681 INFO:teuthology.orchestra.run.smithi005.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2022-09-09T22:27:01.681 INFO:teuthology.orchestra.run.smithi005.stdout: "otp_pool": "test-zone1.rgw.otp", 2022-09-09T22:27:01.682 INFO:teuthology.orchestra.run.smithi005.stdout: "system_key": { 2022-09-09T22:27:01.682 INFO:teuthology.orchestra.run.smithi005.stdout: "access_key": "3U0V7WQYC6MLEXED", 2022-09-09T22:27:01.682 INFO:teuthology.orchestra.run.smithi005.stdout: "secret_key": "jA4iNzYKmfzg78oKBQTy4mNpjERzasVo" 2022-09-09T22:27:01.682 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:27:01.683 INFO:teuthology.orchestra.run.smithi005.stdout: "placement_pools": [ 2022-09-09T22:27:01.683 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:27:01.683 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "default-placement", 2022-09-09T22:27:01.684 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:27:01.684 INFO:teuthology.orchestra.run.smithi005.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2022-09-09T22:27:01.685 INFO:teuthology.orchestra.run.smithi005.stdout: "storage_classes": { 2022-09-09T22:27:01.685 INFO:teuthology.orchestra.run.smithi005.stdout: "STANDARD": { 2022-09-09T22:27:01.685 INFO:teuthology.orchestra.run.smithi005.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2022-09-09T22:27:01.686 INFO:teuthology.orchestra.run.smithi005.stdout: "compression_type": "random" 2022-09-09T22:27:01.686 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:27:01.687 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:27:01.687 INFO:teuthology.orchestra.run.smithi005.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2022-09-09T22:27:01.688 INFO:teuthology.orchestra.run.smithi005.stdout: "index_type": 0 2022-09-09T22:27:01.688 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:27:01.689 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:27:01.689 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:27:01.689 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_id": "", 2022-09-09T22:27:01.690 INFO:teuthology.orchestra.run.smithi005.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2022-09-09T22:27:01.690 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:27:01.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:27:01.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:27:01.694 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-09T22:27:01.749 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:27:01.749 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:27:01.767 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:27:01.763+0000 7f2b18eaddc0 0 period (588f0e6a-ed72-4f68-b3ff-9c4cf92a2fa8 does not have zone 4b05608c-7197-4e9b-bd29-7989607fa277 configured 2022-09-09T22:27:08.692 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:27:08.693 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:27:08.693 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2022-09-09T22:27:08.693 INFO:teuthology.orchestra.run.smithi005.stdout: "predecessor_uuid": "588f0e6a-ed72-4f68-b3ff-9c4cf92a2fa8", 2022-09-09T22:27:08.694 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": [], 2022-09-09T22:27:08.694 INFO:teuthology.orchestra.run.smithi005.stdout: "period_map": { 2022-09-09T22:27:08.694 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:27:08.694 INFO:teuthology.orchestra.run.smithi005.stdout: "zonegroups": [ 2022-09-09T22:27:08.695 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:27:08.695 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "test-zonegroup", 2022-09-09T22:27:08.695 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zonegroup", 2022-09-09T22:27:08.695 INFO:teuthology.orchestra.run.smithi005.stdout: "api_name": "", 2022-09-09T22:27:08.696 INFO:teuthology.orchestra.run.smithi005.stdout: "is_master": true, 2022-09-09T22:27:08.696 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T22:27:08.696 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi005.front.sepia.ceph.com:8000" 2022-09-09T22:27:08.697 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:27:08.697 INFO:teuthology.orchestra.run.smithi005.stdout: "hostnames": [], 2022-09-09T22:27:08.698 INFO:teuthology.orchestra.run.smithi005.stdout: "hostnames_s3website": [], 2022-09-09T22:27:08.698 INFO:teuthology.orchestra.run.smithi005.stdout: "master_zone": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T22:27:08.699 INFO:teuthology.orchestra.run.smithi005.stdout: "zones": [ 2022-09-09T22:27:08.699 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:27:08.699 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T22:27:08.700 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zone1", 2022-09-09T22:27:08.700 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T22:27:08.701 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi005.front.sepia.ceph.com:8000" 2022-09-09T22:27:08.701 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:27:08.702 INFO:teuthology.orchestra.run.smithi005.stdout: "log_meta": false, 2022-09-09T22:27:08.702 INFO:teuthology.orchestra.run.smithi005.stdout: "log_data": false, 2022-09-09T22:27:08.702 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_index_max_shards": 11, 2022-09-09T22:27:08.703 INFO:teuthology.orchestra.run.smithi005.stdout: "read_only": false, 2022-09-09T22:27:08.703 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_type": "", 2022-09-09T22:27:08.704 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from_all": true, 2022-09-09T22:27:08.704 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from": [], 2022-09-09T22:27:08.704 INFO:teuthology.orchestra.run.smithi005.stdout: "redirect_zone": "", 2022-09-09T22:27:08.705 INFO:teuthology.orchestra.run.smithi005.stdout: "supported_features": [ 2022-09-09T22:27:08.705 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T22:27:08.706 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:27:08.706 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:27:08.707 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:27:08.707 INFO:teuthology.orchestra.run.smithi005.stdout: "placement_targets": [ 2022-09-09T22:27:08.708 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:27:08.708 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "default-placement", 2022-09-09T22:27:08.708 INFO:teuthology.orchestra.run.smithi005.stdout: "tags": [], 2022-09-09T22:27:08.709 INFO:teuthology.orchestra.run.smithi005.stdout: "storage_classes": [ 2022-09-09T22:27:08.709 INFO:teuthology.orchestra.run.smithi005.stdout: "STANDARD" 2022-09-09T22:27:08.710 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:27:08.710 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:27:08.711 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:27:08.711 INFO:teuthology.orchestra.run.smithi005.stdout: "default_placement": "default-placement", 2022-09-09T22:27:08.711 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T22:27:08.712 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_policy": { 2022-09-09T22:27:08.712 INFO:teuthology.orchestra.run.smithi005.stdout: "groups": [] 2022-09-09T22:27:08.713 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:27:08.713 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled_features": [ 2022-09-09T22:27:08.713 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T22:27:08.713 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:27:08.714 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:27:08.714 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:27:08.714 INFO:teuthology.orchestra.run.smithi005.stdout: "short_zone_ids": [ 2022-09-09T22:27:08.715 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:27:08.715 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T22:27:08.715 INFO:teuthology.orchestra.run.smithi005.stdout: "val": 1301724575 2022-09-09T22:27:08.715 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:27:08.716 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:27:08.716 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:27:08.716 INFO:teuthology.orchestra.run.smithi005.stdout: "master_zonegroup": "test-zonegroup", 2022-09-09T22:27:08.716 INFO:teuthology.orchestra.run.smithi005.stdout: "master_zone": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T22:27:08.717 INFO:teuthology.orchestra.run.smithi005.stdout: "period_config": { 2022-09-09T22:27:08.717 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_quota": { 2022-09-09T22:27:08.717 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false, 2022-09-09T22:27:08.717 INFO:teuthology.orchestra.run.smithi005.stdout: "check_on_raw": false, 2022-09-09T22:27:08.717 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size": -1, 2022-09-09T22:27:08.718 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size_kb": 0, 2022-09-09T22:27:08.718 INFO:teuthology.orchestra.run.smithi005.stdout: "max_objects": -1 2022-09-09T22:27:08.719 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:27:08.719 INFO:teuthology.orchestra.run.smithi005.stdout: "user_quota": { 2022-09-09T22:27:08.719 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false, 2022-09-09T22:27:08.719 INFO:teuthology.orchestra.run.smithi005.stdout: "check_on_raw": false, 2022-09-09T22:27:08.720 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size": -1, 2022-09-09T22:27:08.720 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size_kb": 0, 2022-09-09T22:27:08.720 INFO:teuthology.orchestra.run.smithi005.stdout: "max_objects": -1 2022-09-09T22:27:08.720 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:27:08.720 INFO:teuthology.orchestra.run.smithi005.stdout: "user_ratelimit": { 2022-09-09T22:27:08.721 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_ops": 0, 2022-09-09T22:27:08.721 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_ops": 0, 2022-09-09T22:27:08.721 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_bytes": 0, 2022-09-09T22:27:08.721 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_bytes": 0, 2022-09-09T22:27:08.722 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false 2022-09-09T22:27:08.722 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:27:08.722 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_ratelimit": { 2022-09-09T22:27:08.722 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_ops": 0, 2022-09-09T22:27:08.723 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_ops": 0, 2022-09-09T22:27:08.723 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_bytes": 0, 2022-09-09T22:27:08.723 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_bytes": 0, 2022-09-09T22:27:08.723 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false 2022-09-09T22:27:08.724 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:27:08.724 INFO:teuthology.orchestra.run.smithi005.stdout: "anonymous_ratelimit": { 2022-09-09T22:27:08.724 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_ops": 0, 2022-09-09T22:27:08.724 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_ops": 0, 2022-09-09T22:27:08.725 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_bytes": 0, 2022-09-09T22:27:08.725 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_bytes": 0, 2022-09-09T22:27:08.725 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false 2022-09-09T22:27:08.725 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:27:08.726 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:27:08.726 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T22:27:08.726 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_name": "test-realm", 2022-09-09T22:27:08.726 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:27:08.727 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:27:08.729 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2022-09-09T22:27:08.730 INFO:teuthology.orchestra.run:waiting for 300 2022-09-09T22:27:08.771 INFO:tasks.rgw.c1.client.0.smithi005.stdout:2022-09-09T22:27:08.767+0000 7fd7c21ca700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 18352) UID: 0 2022-09-09T22:27:08.771 INFO:tasks.rgw.c1.client.0.smithi005.stdout:2022-09-09T22:27:08.767+0000 7fd7c5a28dc0 -1 shutting down 2022-09-09T22:27:14.646 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2022-09-09T22:27:14.832 INFO:tasks.rgw.c1.client.0:Stopped 2022-09-09T22:27:14.832 INFO:tasks.rgw.c1.client.0:Restarting daemon 2022-09-09T22:27:14.833 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2022-09-09T22:27:14.836 INFO:tasks.rgw.c1.client.0:Started 2022-09-09T22:27:14.836 DEBUG:teuthology.orchestra.run.smithi005:> curl http://smithi005.front.sepia.ceph.com:8000 2022-09-09T22:27:14.860 INFO:teuthology.orchestra.run.smithi005.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-09T22:27:14.861 INFO:teuthology.orchestra.run.smithi005.stderr: Dload Upload Total Spent Left Speed 2022-09-09T22:27:14.861 INFO:teuthology.orchestra.run.smithi005.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2022-09-09T22:27:14.862 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-09T22:27:14.863 INFO:teuthology.orchestra.run.smithi005.stderr:curl: (7) Failed to connect to smithi005.front.sepia.ceph.com port 8000: Connection refused 2022-09-09T22:27:15.864 DEBUG:teuthology.orchestra.run.smithi005:> curl http://smithi005.front.sepia.ceph.com:8000 2022-09-09T22:27:15.881 INFO:teuthology.orchestra.run.smithi005.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-09T22:27:15.882 INFO:teuthology.orchestra.run.smithi005.stderr: Dload Upload Total Spent Left Speed 2022-09-09T22:27:15.909 INFO:teuthology.orchestra.run.smithi005.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 7925 0 --:--:-- --:--:-- --:--:-- 7925 2022-09-09T22:27:15.911 INFO:teuthology.orchestra.run.smithi005.stdout:anonymous 2022-09-09T22:27:15.911 INFO:tasks.rgw_multisite:creating admin user.. 2022-09-09T22:27:15.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', '3U0V7WQYC6MLEXED', '--secret', 'jA4iNzYKmfzg78oKBQTy4mNpjERzasVo', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:27:15.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', '3U0V7WQYC6MLEXED', '--secret', 'jA4iNzYKmfzg78oKBQTy4mNpjERzasVo', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:27:15.913 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key 3U0V7WQYC6MLEXED --secret jA4iNzYKmfzg78oKBQTy4mNpjERzasVo --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-09T22:27:15.967 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:27:15.967 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:27:16.028 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:27:16.029 INFO:teuthology.orchestra.run.smithi005.stdout: "user_id": "realm-admin", 2022-09-09T22:27:16.029 INFO:teuthology.orchestra.run.smithi005.stdout: "display_name": "Realm Admin", 2022-09-09T22:27:16.029 INFO:teuthology.orchestra.run.smithi005.stdout: "email": "", 2022-09-09T22:27:16.029 INFO:teuthology.orchestra.run.smithi005.stdout: "suspended": 0, 2022-09-09T22:27:16.030 INFO:teuthology.orchestra.run.smithi005.stdout: "max_buckets": 1000, 2022-09-09T22:27:16.030 INFO:teuthology.orchestra.run.smithi005.stdout: "subusers": [], 2022-09-09T22:27:16.030 INFO:teuthology.orchestra.run.smithi005.stdout: "keys": [ 2022-09-09T22:27:16.030 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:27:16.031 INFO:teuthology.orchestra.run.smithi005.stdout: "user": "realm-admin", 2022-09-09T22:27:16.031 INFO:teuthology.orchestra.run.smithi005.stdout: "access_key": "3U0V7WQYC6MLEXED", 2022-09-09T22:27:16.031 INFO:teuthology.orchestra.run.smithi005.stdout: "secret_key": "jA4iNzYKmfzg78oKBQTy4mNpjERzasVo" 2022-09-09T22:27:16.031 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:27:16.032 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:27:16.032 INFO:teuthology.orchestra.run.smithi005.stdout: "swift_keys": [], 2022-09-09T22:27:16.032 INFO:teuthology.orchestra.run.smithi005.stdout: "caps": [], 2022-09-09T22:27:16.032 INFO:teuthology.orchestra.run.smithi005.stdout: "op_mask": "read, write, delete", 2022-09-09T22:27:16.032 INFO:teuthology.orchestra.run.smithi005.stdout: "system": true, 2022-09-09T22:27:16.033 INFO:teuthology.orchestra.run.smithi005.stdout: "default_placement": "", 2022-09-09T22:27:16.033 INFO:teuthology.orchestra.run.smithi005.stdout: "default_storage_class": "", 2022-09-09T22:27:16.033 INFO:teuthology.orchestra.run.smithi005.stdout: "placement_tags": [], 2022-09-09T22:27:16.033 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_quota": { 2022-09-09T22:27:16.034 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false, 2022-09-09T22:27:16.034 INFO:teuthology.orchestra.run.smithi005.stdout: "check_on_raw": false, 2022-09-09T22:27:16.034 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size": -1, 2022-09-09T22:27:16.034 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size_kb": 0, 2022-09-09T22:27:16.035 INFO:teuthology.orchestra.run.smithi005.stdout: "max_objects": -1 2022-09-09T22:27:16.035 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:27:16.035 INFO:teuthology.orchestra.run.smithi005.stdout: "user_quota": { 2022-09-09T22:27:16.035 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false, 2022-09-09T22:27:16.036 INFO:teuthology.orchestra.run.smithi005.stdout: "check_on_raw": false, 2022-09-09T22:27:16.036 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size": -1, 2022-09-09T22:27:16.036 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size_kb": 0, 2022-09-09T22:27:16.036 INFO:teuthology.orchestra.run.smithi005.stdout: "max_objects": -1 2022-09-09T22:27:16.036 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:27:16.037 INFO:teuthology.orchestra.run.smithi005.stdout: "temp_url_keys": [], 2022-09-09T22:27:16.037 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "rgw", 2022-09-09T22:27:16.037 INFO:teuthology.orchestra.run.smithi005.stdout: "mfa_ids": [] 2022-09-09T22:27:16.037 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:27:16.038 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:27:16.054 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2022-09-09T22:27:16.055 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi005.front.sepia.ceph.com:8000', '--access-key', '3U0V7WQYC6MLEXED', '--secret', 'jA4iNzYKmfzg78oKBQTy4mNpjERzasVo', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:27:16.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi005.front.sepia.ceph.com:8000', '--access-key', '3U0V7WQYC6MLEXED', '--secret', 'jA4iNzYKmfzg78oKBQTy4mNpjERzasVo', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:27:16.056 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi005.front.sepia.ceph.com:8000 --access-key 3U0V7WQYC6MLEXED --secret jA4iNzYKmfzg78oKBQTy4mNpjERzasVo --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-09T22:27:16.105 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:27:16.105 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:27:16.187 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:27:16.187 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T22:27:16.188 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-realm", 2022-09-09T22:27:16.188 INFO:teuthology.orchestra.run.smithi080.stdout: "current_period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:27:16.189 INFO:teuthology.orchestra.run.smithi080.stdout: "epoch": 2 2022-09-09T22:27:16.189 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:27:16.191 INFO:tasks.rgw_multisite:creating zone.. 2022-09-09T22:27:16.192 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', '3U0V7WQYC6MLEXED', '--secret', 'jA4iNzYKmfzg78oKBQTy4mNpjERzasVo', '--endpoints', 'http://smithi080.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:27:16.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', '3U0V7WQYC6MLEXED', '--secret', 'jA4iNzYKmfzg78oKBQTy4mNpjERzasVo', '--endpoints', 'http://smithi080.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:27:16.194 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key 3U0V7WQYC6MLEXED --secret jA4iNzYKmfzg78oKBQTy4mNpjERzasVo --endpoints http://smithi080.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-09T22:27:16.245 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:27:16.246 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:27:16.298 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:27:16.294+0000 7f6acc72cdc0 0 failed reading obj info from .rgw.root:zone_info.4b05608c-7197-4e9b-bd29-7989607fa277: (2) No such file or directory 2022-09-09T22:27:16.298 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:27:16.294+0000 7f6acc72cdc0 0 WARNING: could not read zone params for zone id=4b05608c-7197-4e9b-bd29-7989607fa277 name=test-zone1 2022-09-09T22:27:16.307 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:27:16.307 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T22:27:16.308 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone2", 2022-09-09T22:27:16.308 INFO:teuthology.orchestra.run.smithi080.stdout: "domain_root": "test-zone2.rgw.meta:root", 2022-09-09T22:27:16.309 INFO:teuthology.orchestra.run.smithi080.stdout: "control_pool": "test-zone2.rgw.control", 2022-09-09T22:27:16.309 INFO:teuthology.orchestra.run.smithi080.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2022-09-09T22:27:16.309 INFO:teuthology.orchestra.run.smithi080.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2022-09-09T22:27:16.309 INFO:teuthology.orchestra.run.smithi080.stdout: "log_pool": "test-zone2.rgw.log", 2022-09-09T22:27:16.310 INFO:teuthology.orchestra.run.smithi080.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2022-09-09T22:27:16.310 INFO:teuthology.orchestra.run.smithi080.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2022-09-09T22:27:16.310 INFO:teuthology.orchestra.run.smithi080.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2022-09-09T22:27:16.310 INFO:teuthology.orchestra.run.smithi080.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2022-09-09T22:27:16.311 INFO:teuthology.orchestra.run.smithi080.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2022-09-09T22:27:16.311 INFO:teuthology.orchestra.run.smithi080.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2022-09-09T22:27:16.311 INFO:teuthology.orchestra.run.smithi080.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2022-09-09T22:27:16.311 INFO:teuthology.orchestra.run.smithi080.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2022-09-09T22:27:16.311 INFO:teuthology.orchestra.run.smithi080.stdout: "otp_pool": "test-zone2.rgw.otp", 2022-09-09T22:27:16.312 INFO:teuthology.orchestra.run.smithi080.stdout: "system_key": { 2022-09-09T22:27:16.312 INFO:teuthology.orchestra.run.smithi080.stdout: "access_key": "3U0V7WQYC6MLEXED", 2022-09-09T22:27:16.312 INFO:teuthology.orchestra.run.smithi080.stdout: "secret_key": "jA4iNzYKmfzg78oKBQTy4mNpjERzasVo" 2022-09-09T22:27:16.312 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:16.313 INFO:teuthology.orchestra.run.smithi080.stdout: "placement_pools": [ 2022-09-09T22:27:16.313 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:27:16.313 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "default-placement", 2022-09-09T22:27:16.313 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:27:16.314 INFO:teuthology.orchestra.run.smithi080.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2022-09-09T22:27:16.314 INFO:teuthology.orchestra.run.smithi080.stdout: "storage_classes": { 2022-09-09T22:27:16.314 INFO:teuthology.orchestra.run.smithi080.stdout: "STANDARD": { 2022-09-09T22:27:16.314 INFO:teuthology.orchestra.run.smithi080.stdout: "data_pool": "test-zone2.rgw.buckets.data" 2022-09-09T22:27:16.315 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:27:16.315 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:16.315 INFO:teuthology.orchestra.run.smithi080.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2022-09-09T22:27:16.315 INFO:teuthology.orchestra.run.smithi080.stdout: "index_type": 0 2022-09-09T22:27:16.316 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:27:16.316 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:27:16.316 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:27:16.316 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T22:27:16.316 INFO:teuthology.orchestra.run.smithi080.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2022-09-09T22:27:16.317 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:27:16.318 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2022-09-09T22:27:17.548 INFO:teuthology.orchestra.run.smithi080.stderr:pool 'test-zone2.rgw.buckets.data' created 2022-09-09T22:27:17.562 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2022-09-09T22:27:18.552 INFO:teuthology.orchestra.run.smithi080.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2022-09-09T22:27:18.569 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:27:18.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:27:18.570 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-09T22:27:18.615 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:27:18.616 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:27:18.664 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:27:18.665 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T22:27:18.665 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone2", 2022-09-09T22:27:18.665 INFO:teuthology.orchestra.run.smithi080.stdout: "domain_root": "test-zone2.rgw.meta:root", 2022-09-09T22:27:18.666 INFO:teuthology.orchestra.run.smithi080.stdout: "control_pool": "test-zone2.rgw.control", 2022-09-09T22:27:18.666 INFO:teuthology.orchestra.run.smithi080.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2022-09-09T22:27:18.666 INFO:teuthology.orchestra.run.smithi080.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2022-09-09T22:27:18.666 INFO:teuthology.orchestra.run.smithi080.stdout: "log_pool": "test-zone2.rgw.log", 2022-09-09T22:27:18.667 INFO:teuthology.orchestra.run.smithi080.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2022-09-09T22:27:18.667 INFO:teuthology.orchestra.run.smithi080.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2022-09-09T22:27:18.667 INFO:teuthology.orchestra.run.smithi080.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2022-09-09T22:27:18.668 INFO:teuthology.orchestra.run.smithi080.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2022-09-09T22:27:18.669 INFO:teuthology.orchestra.run.smithi080.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2022-09-09T22:27:18.669 INFO:teuthology.orchestra.run.smithi080.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2022-09-09T22:27:18.669 INFO:teuthology.orchestra.run.smithi080.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2022-09-09T22:27:18.670 INFO:teuthology.orchestra.run.smithi080.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2022-09-09T22:27:18.670 INFO:teuthology.orchestra.run.smithi080.stdout: "otp_pool": "test-zone2.rgw.otp", 2022-09-09T22:27:18.670 INFO:teuthology.orchestra.run.smithi080.stdout: "system_key": { 2022-09-09T22:27:18.670 INFO:teuthology.orchestra.run.smithi080.stdout: "access_key": "3U0V7WQYC6MLEXED", 2022-09-09T22:27:18.671 INFO:teuthology.orchestra.run.smithi080.stdout: "secret_key": "jA4iNzYKmfzg78oKBQTy4mNpjERzasVo" 2022-09-09T22:27:18.671 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:18.671 INFO:teuthology.orchestra.run.smithi080.stdout: "placement_pools": [ 2022-09-09T22:27:18.671 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:27:18.672 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "default-placement", 2022-09-09T22:27:18.672 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:27:18.672 INFO:teuthology.orchestra.run.smithi080.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2022-09-09T22:27:18.672 INFO:teuthology.orchestra.run.smithi080.stdout: "storage_classes": { 2022-09-09T22:27:18.673 INFO:teuthology.orchestra.run.smithi080.stdout: "STANDARD": { 2022-09-09T22:27:18.673 INFO:teuthology.orchestra.run.smithi080.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2022-09-09T22:27:18.673 INFO:teuthology.orchestra.run.smithi080.stdout: "compression_type": "random" 2022-09-09T22:27:18.674 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:27:18.674 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:18.675 INFO:teuthology.orchestra.run.smithi080.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2022-09-09T22:27:18.675 INFO:teuthology.orchestra.run.smithi080.stdout: "index_type": 0 2022-09-09T22:27:18.676 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:27:18.676 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:27:18.676 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:27:18.677 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "", 2022-09-09T22:27:18.677 INFO:teuthology.orchestra.run.smithi080.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2022-09-09T22:27:18.678 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:27:18.680 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:27:18.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:27:18.681 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-09T22:27:18.766 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:27:18.767 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:27:18.787 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:27:18.782+0000 7f7def452dc0 0 period (503b1d61-4513-48c8-a308-17b76219610c does not have zone cc1fc6ed-f06f-45bc-961e-adec328c8651 configured 2022-09-09T22:27:20.353 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2022-09-09T22:27:25.618 INFO:teuthology.orchestra.run.smithi080.stderr:Sending period to new master zone 4b05608c-7197-4e9b-bd29-7989607fa277 2022-09-09T22:27:25.664 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:27:25.664 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:27:25.665 INFO:teuthology.orchestra.run.smithi080.stdout: "epoch": 2, 2022-09-09T22:27:25.665 INFO:teuthology.orchestra.run.smithi080.stdout: "predecessor_uuid": "588f0e6a-ed72-4f68-b3ff-9c4cf92a2fa8", 2022-09-09T22:27:25.665 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": [], 2022-09-09T22:27:25.666 INFO:teuthology.orchestra.run.smithi080.stdout: "period_map": { 2022-09-09T22:27:25.666 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:27:25.667 INFO:teuthology.orchestra.run.smithi080.stdout: "zonegroups": [ 2022-09-09T22:27:25.667 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:27:25.667 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "test-zonegroup", 2022-09-09T22:27:25.668 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zonegroup", 2022-09-09T22:27:25.668 INFO:teuthology.orchestra.run.smithi080.stdout: "api_name": "", 2022-09-09T22:27:25.668 INFO:teuthology.orchestra.run.smithi080.stdout: "is_master": true, 2022-09-09T22:27:25.668 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T22:27:25.669 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi005.front.sepia.ceph.com:8000" 2022-09-09T22:27:25.669 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:27:25.669 INFO:teuthology.orchestra.run.smithi080.stdout: "hostnames": [], 2022-09-09T22:27:25.669 INFO:teuthology.orchestra.run.smithi080.stdout: "hostnames_s3website": [], 2022-09-09T22:27:25.670 INFO:teuthology.orchestra.run.smithi080.stdout: "master_zone": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T22:27:25.670 INFO:teuthology.orchestra.run.smithi080.stdout: "zones": [ 2022-09-09T22:27:25.671 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:27:25.671 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T22:27:25.672 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone1", 2022-09-09T22:27:25.672 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T22:27:25.672 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi005.front.sepia.ceph.com:8000" 2022-09-09T22:27:25.672 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:27:25.673 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": false, 2022-09-09T22:27:25.673 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": true, 2022-09-09T22:27:25.673 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2022-09-09T22:27:25.673 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": false, 2022-09-09T22:27:25.674 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "", 2022-09-09T22:27:25.674 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": true, 2022-09-09T22:27:25.674 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [], 2022-09-09T22:27:25.674 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "", 2022-09-09T22:27:25.675 INFO:teuthology.orchestra.run.smithi080.stdout: "supported_features": [ 2022-09-09T22:27:25.675 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T22:27:25.675 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:27:25.675 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:25.675 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:27:25.676 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T22:27:25.676 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone2", 2022-09-09T22:27:25.676 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T22:27:25.676 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi080.front.sepia.ceph.com:8000" 2022-09-09T22:27:25.677 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:27:25.677 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": false, 2022-09-09T22:27:25.677 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": true, 2022-09-09T22:27:25.677 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2022-09-09T22:27:25.677 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": false, 2022-09-09T22:27:25.678 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "", 2022-09-09T22:27:25.678 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": true, 2022-09-09T22:27:25.678 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [], 2022-09-09T22:27:25.678 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "", 2022-09-09T22:27:25.679 INFO:teuthology.orchestra.run.smithi080.stdout: "supported_features": [ 2022-09-09T22:27:25.679 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T22:27:25.679 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:27:25.679 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:27:25.679 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:27:25.680 INFO:teuthology.orchestra.run.smithi080.stdout: "placement_targets": [ 2022-09-09T22:27:25.680 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:27:25.680 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "default-placement", 2022-09-09T22:27:25.680 INFO:teuthology.orchestra.run.smithi080.stdout: "tags": [], 2022-09-09T22:27:25.681 INFO:teuthology.orchestra.run.smithi080.stdout: "storage_classes": [ 2022-09-09T22:27:25.681 INFO:teuthology.orchestra.run.smithi080.stdout: "STANDARD" 2022-09-09T22:27:25.681 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:27:25.682 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:27:25.682 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:27:25.682 INFO:teuthology.orchestra.run.smithi080.stdout: "default_placement": "default-placement", 2022-09-09T22:27:25.683 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T22:27:25.683 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_policy": { 2022-09-09T22:27:25.683 INFO:teuthology.orchestra.run.smithi080.stdout: "groups": [] 2022-09-09T22:27:25.683 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:25.683 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled_features": [ 2022-09-09T22:27:25.684 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T22:27:25.684 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:27:25.684 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:27:25.684 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:27:25.685 INFO:teuthology.orchestra.run.smithi080.stdout: "short_zone_ids": [ 2022-09-09T22:27:25.685 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:27:25.685 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T22:27:25.685 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 1301724575 2022-09-09T22:27:25.686 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:25.686 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:27:25.686 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T22:27:25.686 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 1899625128 2022-09-09T22:27:25.687 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:27:25.687 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:27:25.687 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:25.687 INFO:teuthology.orchestra.run.smithi080.stdout: "master_zonegroup": "test-zonegroup", 2022-09-09T22:27:25.688 INFO:teuthology.orchestra.run.smithi080.stdout: "master_zone": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T22:27:25.688 INFO:teuthology.orchestra.run.smithi080.stdout: "period_config": { 2022-09-09T22:27:25.688 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_quota": { 2022-09-09T22:27:25.688 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false, 2022-09-09T22:27:25.688 INFO:teuthology.orchestra.run.smithi080.stdout: "check_on_raw": false, 2022-09-09T22:27:25.689 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size": -1, 2022-09-09T22:27:25.689 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size_kb": 0, 2022-09-09T22:27:25.689 INFO:teuthology.orchestra.run.smithi080.stdout: "max_objects": -1 2022-09-09T22:27:25.689 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:25.690 INFO:teuthology.orchestra.run.smithi080.stdout: "user_quota": { 2022-09-09T22:27:25.690 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false, 2022-09-09T22:27:25.690 INFO:teuthology.orchestra.run.smithi080.stdout: "check_on_raw": false, 2022-09-09T22:27:25.690 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size": -1, 2022-09-09T22:27:25.691 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size_kb": 0, 2022-09-09T22:27:25.691 INFO:teuthology.orchestra.run.smithi080.stdout: "max_objects": -1 2022-09-09T22:27:25.691 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:25.691 INFO:teuthology.orchestra.run.smithi080.stdout: "user_ratelimit": { 2022-09-09T22:27:25.692 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_ops": 0, 2022-09-09T22:27:25.692 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_ops": 0, 2022-09-09T22:27:25.692 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_bytes": 0, 2022-09-09T22:27:25.692 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_bytes": 0, 2022-09-09T22:27:25.692 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false 2022-09-09T22:27:25.693 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:25.693 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_ratelimit": { 2022-09-09T22:27:25.693 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_ops": 0, 2022-09-09T22:27:25.693 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_ops": 0, 2022-09-09T22:27:25.694 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_bytes": 0, 2022-09-09T22:27:25.694 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_bytes": 0, 2022-09-09T22:27:25.694 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false 2022-09-09T22:27:25.694 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:25.696 INFO:teuthology.orchestra.run.smithi080.stdout: "anonymous_ratelimit": { 2022-09-09T22:27:25.696 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_ops": 0, 2022-09-09T22:27:25.696 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_ops": 0, 2022-09-09T22:27:25.696 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_bytes": 0, 2022-09-09T22:27:25.697 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_bytes": 0, 2022-09-09T22:27:25.697 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false 2022-09-09T22:27:25.698 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:27:25.698 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:25.698 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T22:27:25.699 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_name": "test-realm", 2022-09-09T22:27:25.699 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:27:25.699 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:27:25.700 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2022-09-09T22:27:25.701 INFO:teuthology.orchestra.run:waiting for 300 2022-09-09T22:27:25.702 INFO:tasks.rgw.c2.client.0.smithi080.stdout:2022-09-09T22:27:25.698+0000 7fb843205700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 19753) UID: 0 2022-09-09T22:27:25.703 INFO:tasks.rgw.c2.client.0.smithi080.stdout:2022-09-09T22:27:25.698+0000 7fb846a63dc0 -1 shutting down 2022-09-09T22:27:30.227 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2022-09-09T22:27:31.803 INFO:tasks.rgw.c2.client.0:Stopped 2022-09-09T22:27:31.804 INFO:tasks.rgw.c2.client.0:Restarting daemon 2022-09-09T22:27:31.804 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2022-09-09T22:27:31.808 INFO:tasks.rgw.c2.client.0:Started 2022-09-09T22:27:31.809 DEBUG:teuthology.orchestra.run.smithi080:> curl http://smithi080.front.sepia.ceph.com:8000 2022-09-09T22:27:31.829 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-09T22:27:31.831 INFO:teuthology.orchestra.run.smithi080.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-09T22:27:31.831 INFO:teuthology.orchestra.run.smithi080.stderr: Dload Upload Total Spent Left Speed 2022-09-09T22:27:31.831 INFO:teuthology.orchestra.run.smithi080.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2022-09-09T22:27:31.832 INFO:teuthology.orchestra.run.smithi080.stderr:curl: (7) Failed to connect to smithi080.front.sepia.ceph.com port 8000: Connection refused 2022-09-09T22:27:32.833 DEBUG:teuthology.orchestra.run.smithi080:> curl http://smithi080.front.sepia.ceph.com:8000 2022-09-09T22:27:32.854 INFO:teuthology.orchestra.run.smithi080.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-09T22:27:32.855 INFO:teuthology.orchestra.run.smithi080.stderr: Dload Upload Total Spent Left Speed 2022-09-09T22:27:32.856 INFO:teuthology.orchestra.run.smithi080.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 53500 0 --:--:-- --:--:-- --:--:-- 53500 2022-09-09T22:27:32.858 INFO:teuthology.orchestra.run.smithi080.stdout:anonymous 2022-09-09T22:27:32.859 INFO:tasks.rgw_multisite:creating zone.. 2022-09-09T22:27:32.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '3U0V7WQYC6MLEXED', '--secret', 'jA4iNzYKmfzg78oKBQTy4mNpjERzasVo', '--endpoints', 'http://smithi005.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:27:32.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '3U0V7WQYC6MLEXED', '--secret', 'jA4iNzYKmfzg78oKBQTy4mNpjERzasVo', '--endpoints', 'http://smithi005.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:27:32.861 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key 3U0V7WQYC6MLEXED --secret jA4iNzYKmfzg78oKBQTy4mNpjERzasVo --endpoints http://smithi005.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-09T22:27:32.915 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:27:32.916 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:27:32.977 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:27:32.971+0000 7f4a36b53dc0 0 failed reading obj info from .rgw.root:zone_info.cc1fc6ed-f06f-45bc-961e-adec328c8651: (2) No such file or directory 2022-09-09T22:27:32.978 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:27:32.971+0000 7f4a36b53dc0 0 WARNING: could not read zone params for zone id=cc1fc6ed-f06f-45bc-961e-adec328c8651 name=test-zone2 2022-09-09T22:27:32.985 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:27:32.986 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5", 2022-09-09T22:27:32.987 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zone3", 2022-09-09T22:27:32.987 INFO:teuthology.orchestra.run.smithi005.stdout: "domain_root": "test-zone3.rgw.meta:root", 2022-09-09T22:27:32.987 INFO:teuthology.orchestra.run.smithi005.stdout: "control_pool": "test-zone3.rgw.control", 2022-09-09T22:27:32.988 INFO:teuthology.orchestra.run.smithi005.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2022-09-09T22:27:32.988 INFO:teuthology.orchestra.run.smithi005.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2022-09-09T22:27:32.989 INFO:teuthology.orchestra.run.smithi005.stdout: "log_pool": "test-zone3.rgw.log", 2022-09-09T22:27:32.989 INFO:teuthology.orchestra.run.smithi005.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2022-09-09T22:27:32.990 INFO:teuthology.orchestra.run.smithi005.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2022-09-09T22:27:32.990 INFO:teuthology.orchestra.run.smithi005.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2022-09-09T22:27:32.991 INFO:teuthology.orchestra.run.smithi005.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2022-09-09T22:27:32.991 INFO:teuthology.orchestra.run.smithi005.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2022-09-09T22:27:32.992 INFO:teuthology.orchestra.run.smithi005.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2022-09-09T22:27:32.992 INFO:teuthology.orchestra.run.smithi005.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2022-09-09T22:27:32.993 INFO:teuthology.orchestra.run.smithi005.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2022-09-09T22:27:32.993 INFO:teuthology.orchestra.run.smithi005.stdout: "otp_pool": "test-zone3.rgw.otp", 2022-09-09T22:27:32.994 INFO:teuthology.orchestra.run.smithi005.stdout: "system_key": { 2022-09-09T22:27:32.994 INFO:teuthology.orchestra.run.smithi005.stdout: "access_key": "3U0V7WQYC6MLEXED", 2022-09-09T22:27:32.995 INFO:teuthology.orchestra.run.smithi005.stdout: "secret_key": "jA4iNzYKmfzg78oKBQTy4mNpjERzasVo" 2022-09-09T22:27:32.995 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:27:32.996 INFO:teuthology.orchestra.run.smithi005.stdout: "placement_pools": [ 2022-09-09T22:27:32.996 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:27:32.997 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "default-placement", 2022-09-09T22:27:32.997 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:27:32.998 INFO:teuthology.orchestra.run.smithi005.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2022-09-09T22:27:32.998 INFO:teuthology.orchestra.run.smithi005.stdout: "storage_classes": { 2022-09-09T22:27:32.999 INFO:teuthology.orchestra.run.smithi005.stdout: "STANDARD": { 2022-09-09T22:27:32.999 INFO:teuthology.orchestra.run.smithi005.stdout: "data_pool": "test-zone3.rgw.buckets.data" 2022-09-09T22:27:33.000 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:27:33.000 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:27:33.001 INFO:teuthology.orchestra.run.smithi005.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2022-09-09T22:27:33.001 INFO:teuthology.orchestra.run.smithi005.stdout: "index_type": 0 2022-09-09T22:27:33.001 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:27:33.002 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:27:33.002 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:27:33.003 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T22:27:33.003 INFO:teuthology.orchestra.run.smithi005.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2022-09-09T22:27:33.004 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:27:33.006 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2022-09-09T22:27:33.688 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'test-zone3.rgw.buckets.data' created 2022-09-09T22:27:33.707 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2022-09-09T22:27:34.710 INFO:teuthology.orchestra.run.smithi005.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2022-09-09T22:27:34.728 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:27:34.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:27:34.729 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-09T22:27:34.769 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:27:34.770 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:27:34.812 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:27:34.812 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5", 2022-09-09T22:27:34.813 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zone3", 2022-09-09T22:27:34.813 INFO:teuthology.orchestra.run.smithi005.stdout: "domain_root": "test-zone3.rgw.meta:root", 2022-09-09T22:27:34.813 INFO:teuthology.orchestra.run.smithi005.stdout: "control_pool": "test-zone3.rgw.control", 2022-09-09T22:27:34.813 INFO:teuthology.orchestra.run.smithi005.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2022-09-09T22:27:34.814 INFO:teuthology.orchestra.run.smithi005.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2022-09-09T22:27:34.814 INFO:teuthology.orchestra.run.smithi005.stdout: "log_pool": "test-zone3.rgw.log", 2022-09-09T22:27:34.814 INFO:teuthology.orchestra.run.smithi005.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2022-09-09T22:27:34.815 INFO:teuthology.orchestra.run.smithi005.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2022-09-09T22:27:34.815 INFO:teuthology.orchestra.run.smithi005.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2022-09-09T22:27:34.815 INFO:teuthology.orchestra.run.smithi005.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2022-09-09T22:27:34.815 INFO:teuthology.orchestra.run.smithi005.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2022-09-09T22:27:34.816 INFO:teuthology.orchestra.run.smithi005.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2022-09-09T22:27:34.816 INFO:teuthology.orchestra.run.smithi005.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2022-09-09T22:27:34.816 INFO:teuthology.orchestra.run.smithi005.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2022-09-09T22:27:34.816 INFO:teuthology.orchestra.run.smithi005.stdout: "otp_pool": "test-zone3.rgw.otp", 2022-09-09T22:27:34.817 INFO:teuthology.orchestra.run.smithi005.stdout: "system_key": { 2022-09-09T22:27:34.817 INFO:teuthology.orchestra.run.smithi005.stdout: "access_key": "3U0V7WQYC6MLEXED", 2022-09-09T22:27:34.817 INFO:teuthology.orchestra.run.smithi005.stdout: "secret_key": "jA4iNzYKmfzg78oKBQTy4mNpjERzasVo" 2022-09-09T22:27:34.817 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:27:34.818 INFO:teuthology.orchestra.run.smithi005.stdout: "placement_pools": [ 2022-09-09T22:27:34.818 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:27:34.819 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "default-placement", 2022-09-09T22:27:34.819 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:27:34.819 INFO:teuthology.orchestra.run.smithi005.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2022-09-09T22:27:34.819 INFO:teuthology.orchestra.run.smithi005.stdout: "storage_classes": { 2022-09-09T22:27:34.820 INFO:teuthology.orchestra.run.smithi005.stdout: "STANDARD": { 2022-09-09T22:27:34.820 INFO:teuthology.orchestra.run.smithi005.stdout: "data_pool": "test-zone3.rgw.buckets.data", 2022-09-09T22:27:34.821 INFO:teuthology.orchestra.run.smithi005.stdout: "compression_type": "random" 2022-09-09T22:27:34.821 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:27:34.822 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:27:34.822 INFO:teuthology.orchestra.run.smithi005.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2022-09-09T22:27:34.822 INFO:teuthology.orchestra.run.smithi005.stdout: "index_type": 0 2022-09-09T22:27:34.823 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:27:34.823 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:27:34.824 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:27:34.824 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_id": "", 2022-09-09T22:27:34.825 INFO:teuthology.orchestra.run.smithi005.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2022-09-09T22:27:34.825 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:27:34.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:27:34.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:27:34.828 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-09T22:27:34.913 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:27:34.915 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:27:34.932 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:27:34.927+0000 7fd9630a0dc0 0 period (503b1d61-4513-48c8-a308-17b76219610c does not have zone 79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5 configured 2022-09-09T22:27:35.934 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2022-09-09T22:27:41.799 INFO:teuthology.orchestra.run.smithi005.stderr:Sending period to new master zone 4b05608c-7197-4e9b-bd29-7989607fa277 2022-09-09T22:27:41.842 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:27:41.843 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:27:41.843 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 3, 2022-09-09T22:27:41.843 INFO:teuthology.orchestra.run.smithi005.stdout: "predecessor_uuid": "588f0e6a-ed72-4f68-b3ff-9c4cf92a2fa8", 2022-09-09T22:27:41.843 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": [], 2022-09-09T22:27:41.844 INFO:teuthology.orchestra.run.smithi005.stdout: "period_map": { 2022-09-09T22:27:41.844 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:27:41.844 INFO:teuthology.orchestra.run.smithi005.stdout: "zonegroups": [ 2022-09-09T22:27:41.844 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:27:41.845 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "test-zonegroup", 2022-09-09T22:27:41.845 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zonegroup", 2022-09-09T22:27:41.845 INFO:teuthology.orchestra.run.smithi005.stdout: "api_name": "", 2022-09-09T22:27:41.846 INFO:teuthology.orchestra.run.smithi005.stdout: "is_master": true, 2022-09-09T22:27:41.846 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T22:27:41.847 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi005.front.sepia.ceph.com:8000" 2022-09-09T22:27:41.847 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:27:41.847 INFO:teuthology.orchestra.run.smithi005.stdout: "hostnames": [], 2022-09-09T22:27:41.847 INFO:teuthology.orchestra.run.smithi005.stdout: "hostnames_s3website": [], 2022-09-09T22:27:41.848 INFO:teuthology.orchestra.run.smithi005.stdout: "master_zone": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T22:27:41.848 INFO:teuthology.orchestra.run.smithi005.stdout: "zones": [ 2022-09-09T22:27:41.848 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:27:41.849 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T22:27:41.849 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zone1", 2022-09-09T22:27:41.849 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T22:27:41.850 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi005.front.sepia.ceph.com:8000" 2022-09-09T22:27:41.850 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:27:41.850 INFO:teuthology.orchestra.run.smithi005.stdout: "log_meta": false, 2022-09-09T22:27:41.850 INFO:teuthology.orchestra.run.smithi005.stdout: "log_data": true, 2022-09-09T22:27:41.851 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_index_max_shards": 11, 2022-09-09T22:27:41.851 INFO:teuthology.orchestra.run.smithi005.stdout: "read_only": false, 2022-09-09T22:27:41.851 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_type": "", 2022-09-09T22:27:41.851 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from_all": true, 2022-09-09T22:27:41.852 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from": [], 2022-09-09T22:27:41.852 INFO:teuthology.orchestra.run.smithi005.stdout: "redirect_zone": "", 2022-09-09T22:27:41.852 INFO:teuthology.orchestra.run.smithi005.stdout: "supported_features": [ 2022-09-09T22:27:41.852 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T22:27:41.853 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:27:41.853 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:27:41.853 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:27:41.853 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5", 2022-09-09T22:27:41.854 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zone3", 2022-09-09T22:27:41.854 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T22:27:41.854 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi005.front.sepia.ceph.com:8001" 2022-09-09T22:27:41.854 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:27:41.855 INFO:teuthology.orchestra.run.smithi005.stdout: "log_meta": false, 2022-09-09T22:27:41.855 INFO:teuthology.orchestra.run.smithi005.stdout: "log_data": true, 2022-09-09T22:27:41.855 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_index_max_shards": 11, 2022-09-09T22:27:41.855 INFO:teuthology.orchestra.run.smithi005.stdout: "read_only": false, 2022-09-09T22:27:41.855 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_type": "", 2022-09-09T22:27:41.856 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from_all": true, 2022-09-09T22:27:41.856 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from": [], 2022-09-09T22:27:41.856 INFO:teuthology.orchestra.run.smithi005.stdout: "redirect_zone": "", 2022-09-09T22:27:41.856 INFO:teuthology.orchestra.run.smithi005.stdout: "supported_features": [ 2022-09-09T22:27:41.857 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T22:27:41.857 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:27:41.857 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:27:41.857 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:27:41.858 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T22:27:41.858 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zone2", 2022-09-09T22:27:41.858 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T22:27:41.858 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi080.front.sepia.ceph.com:8000" 2022-09-09T22:27:41.859 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:27:41.859 INFO:teuthology.orchestra.run.smithi005.stdout: "log_meta": false, 2022-09-09T22:27:41.859 INFO:teuthology.orchestra.run.smithi005.stdout: "log_data": true, 2022-09-09T22:27:41.859 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_index_max_shards": 11, 2022-09-09T22:27:41.860 INFO:teuthology.orchestra.run.smithi005.stdout: "read_only": false, 2022-09-09T22:27:41.860 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_type": "", 2022-09-09T22:27:41.860 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from_all": true, 2022-09-09T22:27:41.860 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from": [], 2022-09-09T22:27:41.861 INFO:teuthology.orchestra.run.smithi005.stdout: "redirect_zone": "", 2022-09-09T22:27:41.861 INFO:teuthology.orchestra.run.smithi005.stdout: "supported_features": [ 2022-09-09T22:27:41.861 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T22:27:41.861 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:27:41.862 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:27:41.862 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:27:41.862 INFO:teuthology.orchestra.run.smithi005.stdout: "placement_targets": [ 2022-09-09T22:27:41.862 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:27:41.863 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "default-placement", 2022-09-09T22:27:41.863 INFO:teuthology.orchestra.run.smithi005.stdout: "tags": [], 2022-09-09T22:27:41.863 INFO:teuthology.orchestra.run.smithi005.stdout: "storage_classes": [ 2022-09-09T22:27:41.863 INFO:teuthology.orchestra.run.smithi005.stdout: "STANDARD" 2022-09-09T22:27:41.864 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:27:41.864 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:27:41.864 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:27:41.864 INFO:teuthology.orchestra.run.smithi005.stdout: "default_placement": "default-placement", 2022-09-09T22:27:41.865 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T22:27:41.865 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_policy": { 2022-09-09T22:27:41.865 INFO:teuthology.orchestra.run.smithi005.stdout: "groups": [] 2022-09-09T22:27:41.865 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:27:41.865 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled_features": [ 2022-09-09T22:27:41.866 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T22:27:41.866 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:27:41.866 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:27:41.866 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:27:41.867 INFO:teuthology.orchestra.run.smithi005.stdout: "short_zone_ids": [ 2022-09-09T22:27:41.867 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:27:41.867 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T22:27:41.867 INFO:teuthology.orchestra.run.smithi005.stdout: "val": 1301724575 2022-09-09T22:27:41.868 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:27:41.868 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:27:41.868 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5", 2022-09-09T22:27:41.868 INFO:teuthology.orchestra.run.smithi005.stdout: "val": 2787153217 2022-09-09T22:27:41.868 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:27:41.869 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:27:41.869 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T22:27:41.869 INFO:teuthology.orchestra.run.smithi005.stdout: "val": 1899625128 2022-09-09T22:27:41.869 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:27:41.870 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:27:41.870 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:27:41.871 INFO:teuthology.orchestra.run.smithi005.stdout: "master_zonegroup": "test-zonegroup", 2022-09-09T22:27:41.871 INFO:teuthology.orchestra.run.smithi005.stdout: "master_zone": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T22:27:41.872 INFO:teuthology.orchestra.run.smithi005.stdout: "period_config": { 2022-09-09T22:27:41.872 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_quota": { 2022-09-09T22:27:41.872 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false, 2022-09-09T22:27:41.872 INFO:teuthology.orchestra.run.smithi005.stdout: "check_on_raw": false, 2022-09-09T22:27:41.873 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size": -1, 2022-09-09T22:27:41.873 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size_kb": 0, 2022-09-09T22:27:41.873 INFO:teuthology.orchestra.run.smithi005.stdout: "max_objects": -1 2022-09-09T22:27:41.873 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:27:41.874 INFO:teuthology.orchestra.run.smithi005.stdout: "user_quota": { 2022-09-09T22:27:41.874 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false, 2022-09-09T22:27:41.874 INFO:teuthology.orchestra.run.smithi005.stdout: "check_on_raw": false, 2022-09-09T22:27:41.874 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size": -1, 2022-09-09T22:27:41.874 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size_kb": 0, 2022-09-09T22:27:41.875 INFO:teuthology.orchestra.run.smithi005.stdout: "max_objects": -1 2022-09-09T22:27:41.875 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:27:41.875 INFO:teuthology.orchestra.run.smithi005.stdout: "user_ratelimit": { 2022-09-09T22:27:41.875 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_ops": 0, 2022-09-09T22:27:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_ops": 0, 2022-09-09T22:27:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_bytes": 0, 2022-09-09T22:27:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_bytes": 0, 2022-09-09T22:27:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false 2022-09-09T22:27:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:27:41.877 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_ratelimit": { 2022-09-09T22:27:41.877 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_ops": 0, 2022-09-09T22:27:41.877 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_ops": 0, 2022-09-09T22:27:41.877 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_bytes": 0, 2022-09-09T22:27:41.878 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_bytes": 0, 2022-09-09T22:27:41.878 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false 2022-09-09T22:27:41.878 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:27:41.878 INFO:teuthology.orchestra.run.smithi005.stdout: "anonymous_ratelimit": { 2022-09-09T22:27:41.879 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_ops": 0, 2022-09-09T22:27:41.879 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_ops": 0, 2022-09-09T22:27:41.879 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_bytes": 0, 2022-09-09T22:27:41.879 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_bytes": 0, 2022-09-09T22:27:41.879 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false 2022-09-09T22:27:41.880 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:27:41.880 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:27:41.880 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T22:27:41.880 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_name": "test-realm", 2022-09-09T22:27:41.881 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:27:41.881 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:27:41.881 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2022-09-09T22:27:41.882 INFO:teuthology.orchestra.run:waiting for 300 2022-09-09T22:27:41.883 INFO:tasks.rgw.c1.client.1.smithi005.stdout:2022-09-09T22:27:41.879+0000 7f539567f700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 18354) UID: 0 2022-09-09T22:27:41.883 INFO:tasks.rgw.c1.client.1.smithi005.stdout:2022-09-09T22:27:41.879+0000 7f5398edddc0 -1 shutting down 2022-09-09T22:27:43.079 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2022-09-09T22:27:47.984 INFO:tasks.rgw.c1.client.1:Stopped 2022-09-09T22:27:47.985 INFO:tasks.rgw.c1.client.1:Restarting daemon 2022-09-09T22:27:47.985 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2022-09-09T22:27:47.988 INFO:tasks.rgw.c1.client.1:Started 2022-09-09T22:27:47.989 DEBUG:teuthology.orchestra.run.smithi005:> curl http://smithi005.front.sepia.ceph.com:8001 2022-09-09T22:27:48.009 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-09T22:27:48.011 INFO:teuthology.orchestra.run.smithi005.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-09T22:27:48.011 INFO:teuthology.orchestra.run.smithi005.stderr: Dload Upload Total Spent Left Speed 2022-09-09T22:27:48.011 INFO:teuthology.orchestra.run.smithi005.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2022-09-09T22:27:48.011 INFO:teuthology.orchestra.run.smithi005.stderr:curl: (7) Failed to connect to smithi005.front.sepia.ceph.com port 8001: Connection refused 2022-09-09T22:27:48.786 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2022-09-09T22:27:49.013 DEBUG:teuthology.orchestra.run.smithi005:> curl http://smithi005.front.sepia.ceph.com:8001 2022-09-09T22:27:49.032 INFO:teuthology.orchestra.run.smithi005.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-09T22:27:49.032 INFO:teuthology.orchestra.run.smithi005.stderr: Dload Upload Total Spent Left Speed 2022-09-09T22:27:49.036 INFO:teuthology.orchestra.run.smithi005.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 53500 0 --:--:-- --:--:-- --:--:-- 53500 2022-09-09T22:27:49.037 INFO:teuthology.orchestra.run.smithi005.stdout:anonymous 2022-09-09T22:27:49.038 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2022-09-09T22:27:49.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi005.front.sepia.ceph.com:8000', '--access-key', '3U0V7WQYC6MLEXED', '--secret', 'jA4iNzYKmfzg78oKBQTy4mNpjERzasVo', '--url', 'http://smithi005.front.sepia.ceph.com:8000', '--access-key', '3U0V7WQYC6MLEXED', '--secret', 'jA4iNzYKmfzg78oKBQTy4mNpjERzasVo', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:27:49.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi005.front.sepia.ceph.com:8000', '--access-key', '3U0V7WQYC6MLEXED', '--secret', 'jA4iNzYKmfzg78oKBQTy4mNpjERzasVo', '--url', 'http://smithi005.front.sepia.ceph.com:8000', '--access-key', '3U0V7WQYC6MLEXED', '--secret', 'jA4iNzYKmfzg78oKBQTy4mNpjERzasVo', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:27:49.039 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi005.front.sepia.ceph.com:8000 --access-key 3U0V7WQYC6MLEXED --secret jA4iNzYKmfzg78oKBQTy4mNpjERzasVo --url http://smithi005.front.sepia.ceph.com:8000 --access-key 3U0V7WQYC6MLEXED --secret jA4iNzYKmfzg78oKBQTy4mNpjERzasVo --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-09T22:27:49.095 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:27:49.096 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:27:49.162 INFO:tasks.rgw.c2.client.0.smithi080.stdout:2022-09-09T22:27:49.158+0000 7fbf1e481700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.49e55708-9d21-40b2-99ec-1140f8e8ae95.control err=-107 2022-09-09T22:27:49.182 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:27:49.182 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T22:27:49.182 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-realm", 2022-09-09T22:27:49.183 INFO:teuthology.orchestra.run.smithi080.stdout: "current_period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:27:49.183 INFO:teuthology.orchestra.run.smithi080.stdout: "epoch": 2 2022-09-09T22:27:49.183 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:27:49.185 INFO:tasks.rgw_multisite:creating zone.. 2022-09-09T22:27:49.185 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '3U0V7WQYC6MLEXED', '--secret', 'jA4iNzYKmfzg78oKBQTy4mNpjERzasVo', '--endpoints', 'http://smithi080.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:27:49.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '3U0V7WQYC6MLEXED', '--secret', 'jA4iNzYKmfzg78oKBQTy4mNpjERzasVo', '--endpoints', 'http://smithi080.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:27:49.187 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key 3U0V7WQYC6MLEXED --secret jA4iNzYKmfzg78oKBQTy4mNpjERzasVo --endpoints http://smithi080.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-09T22:27:49.243 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:27:49.244 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:27:49.297 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:27:49.294+0000 7f4ff36cadc0 0 failed reading obj info from .rgw.root:zone_info.4b05608c-7197-4e9b-bd29-7989607fa277: (2) No such file or directory 2022-09-09T22:27:49.298 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:27:49.294+0000 7f4ff36cadc0 0 WARNING: could not read zone params for zone id=4b05608c-7197-4e9b-bd29-7989607fa277 name=test-zone1 2022-09-09T22:27:49.299 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:27:49.294+0000 7f4ff36cadc0 0 failed reading obj info from .rgw.root:zone_info.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5: (2) No such file or directory 2022-09-09T22:27:49.299 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:27:49.294+0000 7f4ff36cadc0 0 WARNING: could not read zone params for zone id=79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5 name=test-zone3 2022-09-09T22:27:49.305 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:27:49.306 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "3ff609e7-37d9-4b47-8369-34e6c99c4c03", 2022-09-09T22:27:49.307 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone4", 2022-09-09T22:27:49.307 INFO:teuthology.orchestra.run.smithi080.stdout: "domain_root": "test-zone4.rgw.meta:root", 2022-09-09T22:27:49.307 INFO:teuthology.orchestra.run.smithi080.stdout: "control_pool": "test-zone4.rgw.control", 2022-09-09T22:27:49.308 INFO:teuthology.orchestra.run.smithi080.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2022-09-09T22:27:49.308 INFO:teuthology.orchestra.run.smithi080.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2022-09-09T22:27:49.308 INFO:teuthology.orchestra.run.smithi080.stdout: "log_pool": "test-zone4.rgw.log", 2022-09-09T22:27:49.308 INFO:teuthology.orchestra.run.smithi080.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2022-09-09T22:27:49.308 INFO:teuthology.orchestra.run.smithi080.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2022-09-09T22:27:49.309 INFO:teuthology.orchestra.run.smithi080.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2022-09-09T22:27:49.309 INFO:teuthology.orchestra.run.smithi080.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2022-09-09T22:27:49.309 INFO:teuthology.orchestra.run.smithi080.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2022-09-09T22:27:49.309 INFO:teuthology.orchestra.run.smithi080.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2022-09-09T22:27:49.310 INFO:teuthology.orchestra.run.smithi080.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2022-09-09T22:27:49.310 INFO:teuthology.orchestra.run.smithi080.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2022-09-09T22:27:49.310 INFO:teuthology.orchestra.run.smithi080.stdout: "otp_pool": "test-zone4.rgw.otp", 2022-09-09T22:27:49.310 INFO:teuthology.orchestra.run.smithi080.stdout: "system_key": { 2022-09-09T22:27:49.311 INFO:teuthology.orchestra.run.smithi080.stdout: "access_key": "3U0V7WQYC6MLEXED", 2022-09-09T22:27:49.311 INFO:teuthology.orchestra.run.smithi080.stdout: "secret_key": "jA4iNzYKmfzg78oKBQTy4mNpjERzasVo" 2022-09-09T22:27:49.311 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:49.311 INFO:teuthology.orchestra.run.smithi080.stdout: "placement_pools": [ 2022-09-09T22:27:49.312 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:27:49.312 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "default-placement", 2022-09-09T22:27:49.312 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:27:49.312 INFO:teuthology.orchestra.run.smithi080.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2022-09-09T22:27:49.313 INFO:teuthology.orchestra.run.smithi080.stdout: "storage_classes": { 2022-09-09T22:27:49.313 INFO:teuthology.orchestra.run.smithi080.stdout: "STANDARD": { 2022-09-09T22:27:49.313 INFO:teuthology.orchestra.run.smithi080.stdout: "data_pool": "test-zone4.rgw.buckets.data" 2022-09-09T22:27:49.313 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:27:49.313 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:49.314 INFO:teuthology.orchestra.run.smithi080.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2022-09-09T22:27:49.314 INFO:teuthology.orchestra.run.smithi080.stdout: "index_type": 0 2022-09-09T22:27:49.314 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:27:49.314 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:27:49.315 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:27:49.315 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_config": { 2022-09-09T22:27:49.315 INFO:teuthology.orchestra.run.smithi080.stdout: "event_retention_days": 7, 2022-09-09T22:27:49.315 INFO:teuthology.orchestra.run.smithi080.stdout: "start_with_full_sync": false 2022-09-09T22:27:49.316 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:49.316 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T22:27:49.316 INFO:teuthology.orchestra.run.smithi080.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2022-09-09T22:27:49.316 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:27:49.318 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2022-09-09T22:27:50.560 INFO:teuthology.orchestra.run.smithi080.stderr:pool 'test-zone4.rgw.buckets.data' created 2022-09-09T22:27:50.572 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2022-09-09T22:27:51.564 INFO:teuthology.orchestra.run.smithi080.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2022-09-09T22:27:51.595 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:27:51.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:27:51.597 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-09T22:27:51.638 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:27:51.639 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:27:51.690 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:27:51.690 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "3ff609e7-37d9-4b47-8369-34e6c99c4c03", 2022-09-09T22:27:51.690 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone4", 2022-09-09T22:27:51.691 INFO:teuthology.orchestra.run.smithi080.stdout: "domain_root": "test-zone4.rgw.meta:root", 2022-09-09T22:27:51.691 INFO:teuthology.orchestra.run.smithi080.stdout: "control_pool": "test-zone4.rgw.control", 2022-09-09T22:27:51.691 INFO:teuthology.orchestra.run.smithi080.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2022-09-09T22:27:51.691 INFO:teuthology.orchestra.run.smithi080.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2022-09-09T22:27:51.692 INFO:teuthology.orchestra.run.smithi080.stdout: "log_pool": "test-zone4.rgw.log", 2022-09-09T22:27:51.692 INFO:teuthology.orchestra.run.smithi080.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2022-09-09T22:27:51.692 INFO:teuthology.orchestra.run.smithi080.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2022-09-09T22:27:51.693 INFO:teuthology.orchestra.run.smithi080.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2022-09-09T22:27:51.693 INFO:teuthology.orchestra.run.smithi080.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2022-09-09T22:27:51.694 INFO:teuthology.orchestra.run.smithi080.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2022-09-09T22:27:51.694 INFO:teuthology.orchestra.run.smithi080.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2022-09-09T22:27:51.695 INFO:teuthology.orchestra.run.smithi080.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2022-09-09T22:27:51.695 INFO:teuthology.orchestra.run.smithi080.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2022-09-09T22:27:51.695 INFO:teuthology.orchestra.run.smithi080.stdout: "otp_pool": "test-zone4.rgw.otp", 2022-09-09T22:27:51.696 INFO:teuthology.orchestra.run.smithi080.stdout: "system_key": { 2022-09-09T22:27:51.696 INFO:teuthology.orchestra.run.smithi080.stdout: "access_key": "3U0V7WQYC6MLEXED", 2022-09-09T22:27:51.697 INFO:teuthology.orchestra.run.smithi080.stdout: "secret_key": "jA4iNzYKmfzg78oKBQTy4mNpjERzasVo" 2022-09-09T22:27:51.697 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:51.697 INFO:teuthology.orchestra.run.smithi080.stdout: "placement_pools": [ 2022-09-09T22:27:51.701 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:27:51.702 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "default-placement", 2022-09-09T22:27:51.702 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:27:51.703 INFO:teuthology.orchestra.run.smithi080.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2022-09-09T22:27:51.703 INFO:teuthology.orchestra.run.smithi080.stdout: "storage_classes": { 2022-09-09T22:27:51.704 INFO:teuthology.orchestra.run.smithi080.stdout: "STANDARD": { 2022-09-09T22:27:51.704 INFO:teuthology.orchestra.run.smithi080.stdout: "data_pool": "test-zone4.rgw.buckets.data", 2022-09-09T22:27:51.704 INFO:teuthology.orchestra.run.smithi080.stdout: "compression_type": "random" 2022-09-09T22:27:51.705 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:27:51.705 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:51.705 INFO:teuthology.orchestra.run.smithi080.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2022-09-09T22:27:51.706 INFO:teuthology.orchestra.run.smithi080.stdout: "index_type": 0 2022-09-09T22:27:51.706 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:27:51.707 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:27:51.707 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:27:51.708 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_config": { 2022-09-09T22:27:51.708 INFO:teuthology.orchestra.run.smithi080.stdout: "event_retention_days": 7, 2022-09-09T22:27:51.709 INFO:teuthology.orchestra.run.smithi080.stdout: "start_with_full_sync": false 2022-09-09T22:27:51.709 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:51.709 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "", 2022-09-09T22:27:51.710 INFO:teuthology.orchestra.run.smithi080.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2022-09-09T22:27:51.711 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:27:51.712 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:27:51.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:27:51.713 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-09T22:27:51.787 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:27:51.787 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:27:51.802 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:27:51.798+0000 7f34fd6c3dc0 0 period (503b1d61-4513-48c8-a308-17b76219610c does not have zone 3ff609e7-37d9-4b47-8369-34e6c99c4c03 configured 2022-09-09T22:27:58.669 INFO:teuthology.orchestra.run.smithi080.stderr:Sending period to new master zone 4b05608c-7197-4e9b-bd29-7989607fa277 2022-09-09T22:27:58.726 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:27:58.727 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:27:58.727 INFO:teuthology.orchestra.run.smithi080.stdout: "epoch": 4, 2022-09-09T22:27:58.728 INFO:teuthology.orchestra.run.smithi080.stdout: "predecessor_uuid": "588f0e6a-ed72-4f68-b3ff-9c4cf92a2fa8", 2022-09-09T22:27:58.729 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": [], 2022-09-09T22:27:58.729 INFO:teuthology.orchestra.run.smithi080.stdout: "period_map": { 2022-09-09T22:27:58.729 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:27:58.730 INFO:teuthology.orchestra.run.smithi080.stdout: "zonegroups": [ 2022-09-09T22:27:58.730 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:27:58.731 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "test-zonegroup", 2022-09-09T22:27:58.731 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zonegroup", 2022-09-09T22:27:58.731 INFO:teuthology.orchestra.run.smithi080.stdout: "api_name": "", 2022-09-09T22:27:58.732 INFO:teuthology.orchestra.run.smithi080.stdout: "is_master": true, 2022-09-09T22:27:58.732 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T22:27:58.733 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi005.front.sepia.ceph.com:8000" 2022-09-09T22:27:58.733 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:27:58.734 INFO:teuthology.orchestra.run.smithi080.stdout: "hostnames": [], 2022-09-09T22:27:58.734 INFO:teuthology.orchestra.run.smithi080.stdout: "hostnames_s3website": [], 2022-09-09T22:27:58.735 INFO:teuthology.orchestra.run.smithi080.stdout: "master_zone": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T22:27:58.735 INFO:teuthology.orchestra.run.smithi080.stdout: "zones": [ 2022-09-09T22:27:58.736 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:27:58.736 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "3ff609e7-37d9-4b47-8369-34e6c99c4c03", 2022-09-09T22:27:58.737 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone4", 2022-09-09T22:27:58.737 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T22:27:58.737 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi080.front.sepia.ceph.com:8001" 2022-09-09T22:27:58.738 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:27:58.738 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": false, 2022-09-09T22:27:58.739 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": true, 2022-09-09T22:27:58.739 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2022-09-09T22:27:58.740 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": false, 2022-09-09T22:27:58.740 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "pubsub", 2022-09-09T22:27:58.740 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": false, 2022-09-09T22:27:58.741 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [ 2022-09-09T22:27:58.741 INFO:teuthology.orchestra.run.smithi080.stdout: "test-zone1" 2022-09-09T22:27:58.742 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:27:58.742 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "", 2022-09-09T22:27:58.743 INFO:teuthology.orchestra.run.smithi080.stdout: "supported_features": [ 2022-09-09T22:27:58.743 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T22:27:58.743 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:27:58.744 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:58.744 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:27:58.745 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T22:27:58.745 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone1", 2022-09-09T22:27:58.746 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T22:27:58.746 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi005.front.sepia.ceph.com:8000" 2022-09-09T22:27:58.747 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:27:58.747 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": false, 2022-09-09T22:27:58.748 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": true, 2022-09-09T22:27:58.748 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2022-09-09T22:27:58.749 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": false, 2022-09-09T22:27:58.749 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "", 2022-09-09T22:27:58.749 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": true, 2022-09-09T22:27:58.750 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [], 2022-09-09T22:27:58.750 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "", 2022-09-09T22:27:58.751 INFO:teuthology.orchestra.run.smithi080.stdout: "supported_features": [ 2022-09-09T22:27:58.752 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T22:27:58.753 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:27:58.753 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:58.755 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:27:58.755 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5", 2022-09-09T22:27:58.755 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone3", 2022-09-09T22:27:58.756 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T22:27:58.756 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi005.front.sepia.ceph.com:8001" 2022-09-09T22:27:58.756 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:27:58.757 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": false, 2022-09-09T22:27:58.757 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": true, 2022-09-09T22:27:58.757 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2022-09-09T22:27:58.757 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": false, 2022-09-09T22:27:58.758 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "", 2022-09-09T22:27:58.758 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": true, 2022-09-09T22:27:58.758 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [], 2022-09-09T22:27:58.758 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "", 2022-09-09T22:27:58.759 INFO:teuthology.orchestra.run.smithi080.stdout: "supported_features": [ 2022-09-09T22:27:58.759 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T22:27:58.759 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:27:58.759 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:58.760 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:27:58.760 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T22:27:58.760 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone2", 2022-09-09T22:27:58.760 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T22:27:58.761 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi080.front.sepia.ceph.com:8000" 2022-09-09T22:27:58.761 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:27:58.761 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": false, 2022-09-09T22:27:58.761 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": true, 2022-09-09T22:27:58.762 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2022-09-09T22:27:58.762 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": false, 2022-09-09T22:27:58.763 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "", 2022-09-09T22:27:58.763 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": true, 2022-09-09T22:27:58.764 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [], 2022-09-09T22:27:58.764 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "", 2022-09-09T22:27:58.764 INFO:teuthology.orchestra.run.smithi080.stdout: "supported_features": [ 2022-09-09T22:27:58.765 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T22:27:58.765 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:27:58.765 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:27:58.766 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:27:58.766 INFO:teuthology.orchestra.run.smithi080.stdout: "placement_targets": [ 2022-09-09T22:27:58.767 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:27:58.767 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "default-placement", 2022-09-09T22:27:58.768 INFO:teuthology.orchestra.run.smithi080.stdout: "tags": [], 2022-09-09T22:27:58.768 INFO:teuthology.orchestra.run.smithi080.stdout: "storage_classes": [ 2022-09-09T22:27:58.771 INFO:teuthology.orchestra.run.smithi080.stdout: "STANDARD" 2022-09-09T22:27:58.771 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:27:58.772 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:27:58.772 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:27:58.773 INFO:teuthology.orchestra.run.smithi080.stdout: "default_placement": "default-placement", 2022-09-09T22:27:58.773 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T22:27:58.773 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_policy": { 2022-09-09T22:27:58.774 INFO:teuthology.orchestra.run.smithi080.stdout: "groups": [] 2022-09-09T22:27:58.774 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:58.775 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled_features": [ 2022-09-09T22:27:58.775 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T22:27:58.775 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:27:58.775 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:27:58.776 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:27:58.776 INFO:teuthology.orchestra.run.smithi080.stdout: "short_zone_ids": [ 2022-09-09T22:27:58.776 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:27:58.776 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "3ff609e7-37d9-4b47-8369-34e6c99c4c03", 2022-09-09T22:27:58.777 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 2593176765 2022-09-09T22:27:58.777 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:58.778 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:27:58.778 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T22:27:58.778 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 1301724575 2022-09-09T22:27:58.779 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:58.779 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:27:58.779 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5", 2022-09-09T22:27:58.779 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 2787153217 2022-09-09T22:27:58.780 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:58.780 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:27:58.780 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T22:27:58.781 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 1899625128 2022-09-09T22:27:58.781 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:27:58.781 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:27:58.781 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:58.781 INFO:teuthology.orchestra.run.smithi080.stdout: "master_zonegroup": "test-zonegroup", 2022-09-09T22:27:58.782 INFO:teuthology.orchestra.run.smithi080.stdout: "master_zone": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T22:27:58.782 INFO:teuthology.orchestra.run.smithi080.stdout: "period_config": { 2022-09-09T22:27:58.782 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_quota": { 2022-09-09T22:27:58.783 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false, 2022-09-09T22:27:58.783 INFO:teuthology.orchestra.run.smithi080.stdout: "check_on_raw": false, 2022-09-09T22:27:58.784 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size": -1, 2022-09-09T22:27:58.784 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size_kb": 0, 2022-09-09T22:27:58.784 INFO:teuthology.orchestra.run.smithi080.stdout: "max_objects": -1 2022-09-09T22:27:58.785 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:58.785 INFO:teuthology.orchestra.run.smithi080.stdout: "user_quota": { 2022-09-09T22:27:58.786 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false, 2022-09-09T22:27:58.786 INFO:teuthology.orchestra.run.smithi080.stdout: "check_on_raw": false, 2022-09-09T22:27:58.787 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size": -1, 2022-09-09T22:27:58.787 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size_kb": 0, 2022-09-09T22:27:58.788 INFO:teuthology.orchestra.run.smithi080.stdout: "max_objects": -1 2022-09-09T22:27:58.788 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:58.789 INFO:teuthology.orchestra.run.smithi080.stdout: "user_ratelimit": { 2022-09-09T22:27:58.789 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_ops": 0, 2022-09-09T22:27:58.790 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_ops": 0, 2022-09-09T22:27:58.790 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_bytes": 0, 2022-09-09T22:27:58.791 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_bytes": 0, 2022-09-09T22:27:58.791 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false 2022-09-09T22:27:58.792 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:58.792 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_ratelimit": { 2022-09-09T22:27:58.793 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_ops": 0, 2022-09-09T22:27:58.793 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_ops": 0, 2022-09-09T22:27:58.794 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_bytes": 0, 2022-09-09T22:27:58.794 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_bytes": 0, 2022-09-09T22:27:58.795 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false 2022-09-09T22:27:58.795 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:58.796 INFO:teuthology.orchestra.run.smithi080.stdout: "anonymous_ratelimit": { 2022-09-09T22:27:58.796 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_ops": 0, 2022-09-09T22:27:58.797 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_ops": 0, 2022-09-09T22:27:58.797 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_bytes": 0, 2022-09-09T22:27:58.798 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_bytes": 0, 2022-09-09T22:27:58.798 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false 2022-09-09T22:27:58.798 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:27:58.799 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:27:58.799 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T22:27:58.800 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_name": "test-realm", 2022-09-09T22:27:58.800 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:27:58.801 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:27:58.802 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2022-09-09T22:27:58.803 INFO:teuthology.orchestra.run:waiting for 300 2022-09-09T22:27:58.804 INFO:tasks.rgw.c2.client.1.smithi080.stdout:2022-09-09T22:27:58.802+0000 7fa511eba700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 19757) UID: 0 2022-09-09T22:27:58.805 INFO:tasks.rgw.c2.client.1.smithi080.stdout:2022-09-09T22:27:58.802+0000 7fa515718dc0 -1 shutting down 2022-09-09T22:28:04.379 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2022-09-09T22:28:04.905 INFO:tasks.rgw.c2.client.1:Stopped 2022-09-09T22:28:04.906 INFO:tasks.rgw.c2.client.1:Restarting daemon 2022-09-09T22:28:04.907 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2022-09-09T22:28:04.911 INFO:tasks.rgw.c2.client.1:Started 2022-09-09T22:28:04.912 DEBUG:teuthology.orchestra.run.smithi080:> curl http://smithi080.front.sepia.ceph.com:8001 2022-09-09T22:28:04.935 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-09T22:28:04.936 INFO:teuthology.orchestra.run.smithi080.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-09T22:28:04.938 INFO:teuthology.orchestra.run.smithi080.stderr: Dload Upload Total Spent Left Speed 2022-09-09T22:28:04.938 INFO:teuthology.orchestra.run.smithi080.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2022-09-09T22:28:04.939 INFO:teuthology.orchestra.run.smithi080.stderr:curl: (7) Failed to connect to smithi080.front.sepia.ceph.com port 8001: Connection refused 2022-09-09T22:28:05.940 DEBUG:teuthology.orchestra.run.smithi080:> curl http://smithi080.front.sepia.ceph.com:8001 2022-09-09T22:28:05.975 INFO:teuthology.orchestra.run.smithi080.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-09T22:28:05.976 INFO:teuthology.orchestra.run.smithi080.stderr: Dload Upload Total Spent Left Speed 2022-09-09T22:28:05.978 INFO:teuthology.orchestra.run.smithi080.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 67000 0 --:--:-- --:--:-- --:--:-- 67000 2022-09-09T22:28:05.979 INFO:teuthology.orchestra.run.smithi080.stdout:{"Code":"MethodNotAllowed","RequestId":"tx00000ec6e4728464dce36-00631bbdf5-10ee-test-zone4","HostId":"10ee-test-zone4-test-zonegroup"} 2022-09-09T22:28:05.980 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2022-09-09T22:28:05.981 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2022-09-09T22:28:06.000 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/git.ceph.com_ceph-c_28758235ba2ab1c1d8afa0f88d00f07b5687a51e/qa/../src/test/rgw/rgw_multi 2022-09-09T22:28:06.030 INFO:tasks.rgw_multisite_tests:creating test user.. 2022-09-09T22:28:06.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:28:06.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:28:06.032 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --caps 'roles=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-09T22:28:06.090 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:28:06.091 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:28:06.152 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:28:06.153 INFO:teuthology.orchestra.run.smithi005.stdout: "user_id": "rgw-multisite-test-user", 2022-09-09T22:28:06.153 INFO:teuthology.orchestra.run.smithi005.stdout: "display_name": "Multisite Test User", 2022-09-09T22:28:06.154 INFO:teuthology.orchestra.run.smithi005.stdout: "email": "", 2022-09-09T22:28:06.155 INFO:teuthology.orchestra.run.smithi005.stdout: "suspended": 0, 2022-09-09T22:28:06.155 INFO:teuthology.orchestra.run.smithi005.stdout: "max_buckets": 1000, 2022-09-09T22:28:06.155 INFO:teuthology.orchestra.run.smithi005.stdout: "subusers": [], 2022-09-09T22:28:06.156 INFO:teuthology.orchestra.run.smithi005.stdout: "keys": [ 2022-09-09T22:28:06.156 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:28:06.157 INFO:teuthology.orchestra.run.smithi005.stdout: "user": "rgw-multisite-test-user", 2022-09-09T22:28:06.157 INFO:teuthology.orchestra.run.smithi005.stdout: "access_key": "VQ6YV8EJ6TF7N01I9BCH", 2022-09-09T22:28:06.158 INFO:teuthology.orchestra.run.smithi005.stdout: "secret_key": "AAmWmF89NxvNpOc3ZV3HlsBlzqO1mM2RbjSBqCZd" 2022-09-09T22:28:06.158 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:28:06.159 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:28:06.159 INFO:teuthology.orchestra.run.smithi005.stdout: "swift_keys": [], 2022-09-09T22:28:06.160 INFO:teuthology.orchestra.run.smithi005.stdout: "caps": [ 2022-09-09T22:28:06.160 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:28:06.161 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "roles", 2022-09-09T22:28:06.161 INFO:teuthology.orchestra.run.smithi005.stdout: "perm": "*" 2022-09-09T22:28:06.166 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:28:06.166 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:28:06.167 INFO:teuthology.orchestra.run.smithi005.stdout: "op_mask": "read, write, delete", 2022-09-09T22:28:06.167 INFO:teuthology.orchestra.run.smithi005.stdout: "default_placement": "", 2022-09-09T22:28:06.168 INFO:teuthology.orchestra.run.smithi005.stdout: "default_storage_class": "", 2022-09-09T22:28:06.168 INFO:teuthology.orchestra.run.smithi005.stdout: "placement_tags": [], 2022-09-09T22:28:06.169 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_quota": { 2022-09-09T22:28:06.170 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false, 2022-09-09T22:28:06.170 INFO:teuthology.orchestra.run.smithi005.stdout: "check_on_raw": false, 2022-09-09T22:28:06.171 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size": -1, 2022-09-09T22:28:06.171 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size_kb": 0, 2022-09-09T22:28:06.172 INFO:teuthology.orchestra.run.smithi005.stdout: "max_objects": -1 2022-09-09T22:28:06.172 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:28:06.172 INFO:teuthology.orchestra.run.smithi005.stdout: "user_quota": { 2022-09-09T22:28:06.173 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false, 2022-09-09T22:28:06.173 INFO:teuthology.orchestra.run.smithi005.stdout: "check_on_raw": false, 2022-09-09T22:28:06.174 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size": -1, 2022-09-09T22:28:06.174 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size_kb": 0, 2022-09-09T22:28:06.174 INFO:teuthology.orchestra.run.smithi005.stdout: "max_objects": -1 2022-09-09T22:28:06.175 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:28:06.175 INFO:teuthology.orchestra.run.smithi005.stdout: "temp_url_keys": [], 2022-09-09T22:28:06.176 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "rgw", 2022-09-09T22:28:06.176 INFO:teuthology.orchestra.run.smithi005.stdout: "mfa_ids": [] 2022-09-09T22:28:06.177 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:28:06.177 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T22:28:06.179 INFO:rgw_multi.tests:meta checkpoint 2022-09-09T22:28:06.180 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:28:06.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:28:06.181 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:28:06.224 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:28:06.225 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:28:06.279 INFO:teuthology.orchestra.run.smithi005.stderr:No --period given, using current period=503b1d61-4513-48c8-a308-17b76219610c 2022-09-09T22:28:06.309 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T22:28:06.309 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:28:06.310 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:28:06.311 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:28:06.146147Z" 2022-09-09T22:28:06.311 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:28:06.311 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:28:06.312 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:28:06.312 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:28:06.313 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:28:06.313 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:28:06.314 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:28:06.314 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:28:06.315 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:28:06.315 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:28:06.316 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:28:06.316 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:28:06.316 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:28:06.317 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T22:28:06.319 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-09T22:28:06.320 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:28:06.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:28:06.331 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:28:06.380 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:28:06.380 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:28:06.442 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:28:06.438+0000 7f563c1a7dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:28:06.443 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:28:06.438+0000 7f563c1a7dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:28:06.477 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:28:06.478 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:28:06.478 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:28:06.479 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:28:06.479 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:28:06.480 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:28:06.480 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:06.481 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:06.481 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:28:06.482 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:06.482 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:28:06.482 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:06.484 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:06.485 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:06.485 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:06.486 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:28:06.486 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:06.487 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:06.488 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:06.488 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:06.488 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:06.489 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:06.489 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:28:06.490 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:06.491 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:06.491 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:06.491 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:06.492 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:06.493 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:06.493 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:06.493 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:06.494 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:06.494 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:06.495 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:06.495 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:28:06.496 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:06.496 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:06.497 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:06.497 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:06.498 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:06.498 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:06.499 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:06.499 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:06.500 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:06.500 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:06.501 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:06.501 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:28:06.502 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:06.502 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:06.503 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:06.503 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:06.504 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:06.504 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:06.504 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:06.505 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:06.505 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:06.506 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:06.507 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:28:06.507 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:06.507 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:28:06.508 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:28:06.508 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:28:06.509 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:06.509 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:28:06Z" 2022-09-09T22:28:06.510 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:28:06.511 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662762486.146147_45.1 target= 2022-09-09T22:28:10.088 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2022-09-09T22:28:11.512 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:28:11.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:28:11.514 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:28:11.565 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:28:11.566 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:28:11.637 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:28:11.626+0000 7f6efcc5ddc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:28:11.637 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:28:11.630+0000 7f6efcc5ddc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:28:11.660 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:28:11.661 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:28:11.661 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:28:11.662 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:28:11.663 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:28:11.663 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:28:11.663 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:11.664 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:11.664 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:28:11.664 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:11.664 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:28:11.665 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:11.665 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:11.665 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:11.665 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:11.666 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:28:11.666 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:11.667 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:11.667 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:11.668 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:11.668 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:11.669 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:11.669 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:28:11.670 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:11.670 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:11.671 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:11.671 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:11.672 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:11.672 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:11.673 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:11.673 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:11.674 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:11.674 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:11.675 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:11.675 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:28:11.676 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:11.676 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:11.677 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:11.677 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:11.678 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:11.678 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:11.679 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:11.679 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:11.680 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:11.680 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:11.681 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:11.681 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:28:11.682 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:11.682 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:11.683 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:11.683 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:11.685 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:11.686 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:11.686 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:11.687 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:11.687 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:11.687 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:11.688 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:28:11.688 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:11.688 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:28:11.689 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:28:11.689 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:28:11.689 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:11.689 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:28:11Z" 2022-09-09T22:28:11.690 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:28:11.692 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662762486.146147_45.1 target= 2022-09-09T22:28:16.693 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:28:16.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:28:16.696 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:28:16.745 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:28:16.746 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:28:16.797 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:28:16.794+0000 7f7c0ddd9dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:28:16.799 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:28:16.794+0000 7f7c0ddd9dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:28:16.831 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:28:16.831 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:28:16.832 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:28:16.832 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:28:16.832 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:28:16.833 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:28:16.833 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:16.833 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:16.834 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:28:16.834 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:16.834 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:28:16.835 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:16.835 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:16.835 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:16.835 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:16.836 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:28:16.837 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:16.838 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:16.838 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:16.839 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:16.840 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:16.840 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:16.840 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:28:16.840 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:16.841 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:16.841 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:16.841 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:16.841 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:16.842 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:16.842 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:16.842 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:16.843 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:16.843 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:16.843 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:16.843 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:28:16.843 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:16.844 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:16.844 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:16.845 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:16.845 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:16.846 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:16.846 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:16.846 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:16.846 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:16.847 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:16.852 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:16.853 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:28:16.853 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:16.854 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:16.854 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:16.855 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:16.855 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:16.856 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:16.856 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:16.857 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:16.857 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:16.858 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:16.858 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:28:16.859 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:16.859 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:28:16.859 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:28:16.860 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:28:16.860 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:16.861 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:28:16Z" 2022-09-09T22:28:16.861 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:28:16.862 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662762486.146147_45.1 target= 2022-09-09T22:28:21.865 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:28:21.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:28:21.866 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:28:21.931 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:28:21.931 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:28:21.992 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:28:21.990+0000 7fc0f304cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:28:21.994 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:28:21.990+0000 7fc0f304cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:28:22.033 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:28:22.033 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:28:22.034 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:28:22.034 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:28:22.035 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:28:22.035 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:28:22.036 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:22.037 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:22.037 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:28:22.038 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:22.038 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:28:22.039 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:22.040 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:22.040 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:22.041 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:22.042 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:28:22.043 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:22.043 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:22.044 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:22.044 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:22.046 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:22.047 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:22.047 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:28:22.048 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:22.048 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:22.050 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:22.050 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:22.050 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:22.050 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:22.051 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:22.051 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:22.051 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:22.051 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:22.052 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:22.052 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:28:22.052 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:22.052 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:22.052 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:22.053 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:22.053 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:22.053 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:22.053 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:22.054 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:22.054 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:22.054 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:22.054 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:22.054 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:28:22.055 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:22.055 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:22.055 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:22.055 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:22.056 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:22.056 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:22.056 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:22.056 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:22.056 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:22.057 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:22.057 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:28:22.057 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:22.057 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:28:22.058 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:28:22.058 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:28:22.058 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:22.058 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:28:21Z" 2022-09-09T22:28:22.058 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:28:22.060 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662762486.146147_45.1 target= 2022-09-09T22:28:27.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:28:27.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:28:27.064 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:28:27.112 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:28:27.112 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:28:27.166 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:28:27.162+0000 7fa53b73fdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:28:27.168 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:28:27.166+0000 7fa53b73fdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:28:27.193 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:28:27.194 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:28:27.195 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:28:27.195 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:28:27.196 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:28:27.196 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:28:27.197 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:27.198 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:27.198 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:28:27.199 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:27.199 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:28:27.200 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:27.200 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:27.201 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:27.201 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:27.201 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:28:27.202 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:27.202 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:27.203 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:27.203 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:27.204 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:27.204 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:27.205 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:28:27.205 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:27.206 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:27.207 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:27.207 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:27.208 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:27.208 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:27.209 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:27.209 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:27.209 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:27.211 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:27.211 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:27.212 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:28:27.212 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:27.213 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:27.213 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:27.214 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:27.214 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:27.214 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:27.215 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:27.215 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:27.216 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:27.216 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:27.217 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:27.217 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:28:27.218 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:27.218 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:27.219 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:27.219 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:27.220 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:27.220 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:27.221 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:27.221 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:27.223 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:27.223 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:27.223 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:28:27.224 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:27.225 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:28:27.225 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:28:27.226 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:28:27.226 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:27.227 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:28:27Z" 2022-09-09T22:28:27.227 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:28:27.230 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662762486.146147_45.1 target= 2022-09-09T22:28:32.231 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:28:32.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:28:32.233 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:28:32.295 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:28:32.295 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:28:32.353 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:28:32.350+0000 7fe5229b0dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:28:32.355 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:28:32.350+0000 7fe5229b0dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:28:32.383 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:28:32.383 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:28:32.384 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:28:32.384 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:28:32.385 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:28:32.385 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:28:32.386 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:32.386 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:32.386 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:28:32.387 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:32.387 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:28:32.388 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:32.388 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:32.389 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:32.389 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:32.389 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:28:32.390 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:32.390 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:32.391 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:32.391 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:32.392 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:32.392 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:32.393 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:28:32.393 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:32.393 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:32.394 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:32.394 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:32.395 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:32.395 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:32.396 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:32.396 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:32.397 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:32.397 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:32.398 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:32.398 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:28:32.398 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:32.399 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:32.399 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:32.400 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:32.400 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:32.401 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:32.401 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:32.401 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:32.402 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:32.402 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:32.403 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:32.403 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:28:32.404 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:32.404 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:32.405 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:32.406 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:32.406 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:32.407 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:32.407 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:32.408 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:32.408 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:32.409 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:32.409 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:28:32.410 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:32.410 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:28:32.411 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:28:32.411 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:28:32.412 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:32.412 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:28:32Z" 2022-09-09T22:28:32.413 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:28:32.416 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662762486.146147_45.1 target= 2022-09-09T22:28:37.418 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:28:37.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:28:37.420 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:28:37.467 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:28:37.468 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:28:37.516 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:28:37.510+0000 7f8ccc810dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:28:37.519 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:28:37.514+0000 7f8ccc810dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:28:37.541 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:28:37.542 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:28:37.542 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:28:37.543 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:28:37.543 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:28:37.543 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:28:37.544 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:37.544 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:37.545 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:28:37.545 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:37.545 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:28:37.546 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:37.547 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:37.547 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:37.547 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:37.548 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:28:37.548 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:37.548 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:37.549 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:37.549 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:37.549 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:37.550 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:37.550 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:28:37.550 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:37.551 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:37.552 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:37.552 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:37.552 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:37.553 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:37.553 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:37.554 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:37.554 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:37.555 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:37.555 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:37.555 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:28:37.555 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:37.556 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:37.556 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:37.556 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:37.556 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:37.557 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:37.557 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:37.557 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:37.558 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:37.558 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:37.558 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:37.559 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:28:37.559 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:37.559 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:37.560 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:37.560 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:37.560 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:37.561 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:37.561 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:37.562 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:37.563 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:37.563 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:37.564 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:28:37.564 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:37.564 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:28:37.565 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:28:37.565 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:28:37.566 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:37.566 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:28:37Z" 2022-09-09T22:28:37.567 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:28:37.569 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662762486.146147_45.1 target= 2022-09-09T22:28:42.571 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:28:42.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:28:42.573 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:28:42.621 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:28:42.622 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:28:42.696 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:28:42.690+0000 7ff6a3e7cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:28:42.696 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:28:42.694+0000 7ff6a3e7cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:28:42.727 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:28:42.728 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:28:42.728 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:28:42.728 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:28:42.728 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:28:42.729 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:28:42.729 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:42.730 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:42.730 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:28:42.730 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:42.731 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:28:42.731 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:42.731 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:42.731 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:42.732 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:42.732 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:28:42.732 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:42.732 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:42.732 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:42.733 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:42.733 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:42.733 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:42.733 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:28:42.734 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:42.734 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:42.734 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:42.734 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:42.735 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:42.735 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:42.735 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:42.735 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:42.735 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:42.736 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:42.736 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:42.736 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:28:42.736 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:42.737 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:42.737 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:42.737 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:42.737 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:42.737 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:42.738 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:42.738 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:42.738 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:42.738 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:42.739 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:42.739 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:28:42.739 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:42.740 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:42.740 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:42.740 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:42.740 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:42.741 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:42.741 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:42.741 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:42.741 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:42.742 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:42.742 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:28:42.742 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:42.742 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:28:42.743 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:28:42.743 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:28:42.743 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:42.743 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:28:42Z" 2022-09-09T22:28:42.744 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:28:42.745 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662762486.146147_45.1 target= 2022-09-09T22:28:47.746 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:28:47.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:28:47.748 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:28:47.795 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:28:47.796 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:28:47.863 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:28:47.858+0000 7f4622919dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:28:47.865 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:28:47.862+0000 7f4622919dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:28:47.892 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:28:47.892 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:28:47.893 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:28:47.893 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:28:47.893 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:28:47.893 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:28:47.894 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:47.894 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:47.894 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:28:47.894 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:47.895 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:28:47.895 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:47.895 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:47.895 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:47.896 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:47.896 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:28:47.896 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:47.896 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:47.897 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:47.897 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:47.897 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:47.897 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:47.898 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:28:47.898 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:47.898 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:47.899 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:47.899 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:47.899 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:47.899 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:47.900 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:47.900 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:47.900 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:47.900 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:47.901 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:47.901 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:28:47.901 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:47.901 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:47.901 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:47.902 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:47.902 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:47.902 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:47.902 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:47.903 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:47.903 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:47.903 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:47.903 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:47.904 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:28:47.904 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:47.904 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:47.904 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:47.905 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:47.905 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:47.905 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:47.905 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:47.906 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:47.906 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:47.906 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:47.906 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:28:47.907 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:47.907 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:28:47.907 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:28:47.907 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:28:47.907 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:47.908 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:28:47Z" 2022-09-09T22:28:47.908 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:28:47.909 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662762486.146147_45.1 target= 2022-09-09T22:28:52.910 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:28:52.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:28:52.912 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:28:52.962 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:28:52.962 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:28:53.027 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:28:53.022+0000 7f281a7ebdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:28:53.028 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:28:53.026+0000 7f281a7ebdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:28:53.061 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:28:53.061 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:28:53.061 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:28:53.062 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:28:53.062 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:28:53.062 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:28:53.062 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:53.063 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:53.063 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:28:53.063 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:53.063 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:28:53.064 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:53.064 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:53.064 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:53.064 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:53.065 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:28:53.065 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:53.065 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:53.066 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:53.066 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:53.066 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:53.067 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:53.067 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:28:53.067 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:53.067 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:53.068 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:53.068 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:53.069 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:53.069 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:53.069 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:53.070 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:53.070 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:53.070 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:53.070 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:53.071 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:28:53.071 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:53.071 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:53.071 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:53.072 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:53.072 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:53.072 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:53.072 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:53.072 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:53.073 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:53.073 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:53.073 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:53.073 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:28:53.074 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:53.074 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:53.074 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:53.074 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:53.075 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:53.075 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:53.075 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:53.075 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:53.076 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:53.076 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:53.076 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:28:53.076 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:53.076 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:28:53.077 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:28:53.077 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:28:53.077 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:53.077 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:28:53Z" 2022-09-09T22:28:53.077 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:28:53.079 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662762486.146147_45.1 target= 2022-09-09T22:28:58.081 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:28:58.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:28:58.082 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:28:58.130 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:28:58.131 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:28:58.200 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:28:58.198+0000 7fe726735dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:28:58.202 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:28:58.198+0000 7fe726735dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:28:58.229 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:28:58.229 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:28:58.230 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:28:58.230 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:28:58.230 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:28:58.231 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:28:58.231 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:58.231 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:58.231 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:28:58.232 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:58.232 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:28:58.232 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:58.232 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:58.233 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:58.233 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:58.233 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:28:58.233 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:58.234 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:58.234 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:58.234 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:58.234 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:58.235 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:58.235 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:28:58.235 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:58.235 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:58.236 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:58.236 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:58.236 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:58.237 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:58.237 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:58.238 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:58.238 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:58.238 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:58.239 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:58.239 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:28:58.239 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:58.240 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:58.240 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:58.240 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:58.241 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:58.241 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:58.241 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:58.241 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:58.242 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:58.242 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:58.242 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:28:58.243 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:28:58.243 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:28:58.243 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:28:58.244 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:28:58.244 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:28:58.245 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:28:58.245 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:28:58.246 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:28:58.246 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:28:58.247 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:58.247 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:28:58.248 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:28:58.248 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:58.248 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:28:58.249 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:28:58.249 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:28:58.250 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:28:58.250 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:28:58Z" 2022-09-09T22:28:58.251 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:28:58.253 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662762486.146147_45.1 target= 2022-09-09T22:29:03.254 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:29:03.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:29:03.256 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:29:03.303 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:29:03.303 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:29:03.359 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:29:03.354+0000 7fa36caf7dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:29:03.360 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:29:03.358+0000 7fa36caf7dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:29:03.385 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:29:03.385 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:29:03.386 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:29:03.386 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:29:03.386 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:29:03.386 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:29:03.387 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:03.387 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:03.387 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:29:03.387 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:03.388 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:29:03.388 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:03.388 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:03.388 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:03.389 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:03.389 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:29:03.389 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:03.389 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:03.390 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:03.390 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:03.390 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:03.390 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:03.391 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:29:03.391 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:03.391 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:03.391 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:03.392 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:03.392 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:03.392 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:03.393 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:03.393 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:03.393 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:03.393 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:03.394 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:03.394 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:29:03.394 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:03.394 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:03.395 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:03.395 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:03.395 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:03.395 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:03.395 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:03.396 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:03.396 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:03.396 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:03.396 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:03.397 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:29:03.397 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:03.397 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:03.398 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:03.398 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:03.398 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:03.398 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:03.399 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:03.399 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:03.400 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:03.400 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:03.401 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:29:03.401 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:03.401 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:29:03.402 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:29:03.402 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:29:03.402 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:03.403 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:29:03Z" 2022-09-09T22:29:03.403 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:29:03.405 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662762486.146147_45.1 target= 2022-09-09T22:29:08.407 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:29:08.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:29:08.408 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:29:08.456 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:29:08.458 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:29:08.523 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:29:08.522+0000 7f349e6ccdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:29:08.524 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:29:08.522+0000 7f349e6ccdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:29:08.555 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:29:08.555 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:29:08.555 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:29:08.556 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:29:08.556 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:29:08.557 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:29:08.557 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:08.558 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:08.558 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:29:08.559 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:08.559 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:29:08.559 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:08.559 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:08.560 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:08.560 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:08.560 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:29:08.560 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:08.561 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:08.561 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:08.561 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:08.562 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:08.562 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:08.562 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:29:08.562 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:08.563 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:08.563 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:08.563 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:08.563 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:08.564 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:08.564 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:08.566 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:08.566 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:08.567 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:08.567 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:08.567 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:29:08.567 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:08.568 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:08.568 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:08.568 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:08.568 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:08.569 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:08.569 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:08.570 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:08.570 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:08.570 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:08.570 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:08.571 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:29:08.571 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:08.571 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:08.571 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:08.572 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:08.572 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:08.572 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:08.572 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:08.573 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:08.573 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:08.573 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:08.573 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:29:08.574 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:08.574 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:29:08.574 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:29:08.574 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:29:08.575 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:08.575 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:29:08Z" 2022-09-09T22:29:08.575 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:29:08.576 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662762486.146147_45.1 target= 2022-09-09T22:29:13.578 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:29:13.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:29:13.580 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:29:13.627 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:29:13.627 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:29:13.693 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:29:13.690+0000 7f6fcfc80dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:29:13.695 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:29:13.694+0000 7f6fcfc80dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:29:13.726 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:29:13.727 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:29:13.727 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:29:13.727 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:29:13.728 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:29:13.728 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:29:13.728 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:13.728 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:13.728 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:29:13.729 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:13.729 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:29:13.729 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:13.729 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:13.730 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:13.730 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:13.730 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:29:13.730 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:13.731 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:13.731 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:13.731 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:13.731 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:13.731 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:13.732 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:29:13.732 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:13.732 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:13.732 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:13.733 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:13.733 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:13.733 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:13.733 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:13.734 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:13.734 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:13.734 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:13.734 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:13.735 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:29:13.735 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:13.735 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:13.735 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:13.735 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:13.736 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:13.736 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:13.736 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:13.736 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:13.737 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:13.737 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:13.737 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:13.737 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:29:13.737 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:13.738 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:13.738 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:13.738 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:13.738 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:13.739 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:13.739 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:13.739 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:13.739 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:13.740 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:13.740 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:29:13.740 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:13.740 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:29:13.740 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:29:13.741 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:29:13.741 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:13.741 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:29:13Z" 2022-09-09T22:29:13.741 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:29:13.742 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662762486.146147_45.1 target= 2022-09-09T22:29:18.743 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:29:18.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:29:18.745 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:29:18.794 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:29:18.795 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:29:18.844 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:29:18.842+0000 7fc8d18d6dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:29:18.845 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:29:18.842+0000 7fc8d18d6dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:29:18.871 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:29:18.872 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:29:18.872 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:29:18.872 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:29:18.872 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:29:18.873 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:29:18.873 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:18.873 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:18.873 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:29:18.874 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:18.874 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:29:18.874 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:18.874 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:18.875 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:18.875 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:18.875 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:29:18.875 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:18.875 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:18.876 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:18.876 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:18.876 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:18.876 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:18.877 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:29:18.877 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:18.877 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:18.878 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:18.878 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:18.878 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:18.879 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:18.879 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:18.880 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:18.880 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:18.880 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:18.881 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:18.881 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:29:18.881 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:18.881 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:18.882 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:18.882 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:18.883 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:18.883 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:18.883 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:18.884 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:18.884 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:18.884 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:18.884 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:18.885 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:29:18.885 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:18.885 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:18.885 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:18.885 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:18.886 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:18.886 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:18.886 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:18.886 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:18.887 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:18.887 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:18.887 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:29:18.887 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:18.887 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:29:18.888 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:29:18.888 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:29:18.888 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:18.888 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:29:18Z" 2022-09-09T22:29:18.889 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:29:18.890 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662762486.146147_45.1 target= 2022-09-09T22:29:23.892 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:29:23.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:29:23.893 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:29:23.939 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:29:23.940 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:29:24.014 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:29:24.010+0000 7fb62bbf1dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:29:24.015 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:29:24.014+0000 7fb62bbf1dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:29:24.044 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:29:24.045 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:29:24.045 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:29:24.045 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:29:24.046 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:29:24.046 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:29:24.046 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:24.047 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:24.047 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:29:24.047 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:24.047 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:29:24.047 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:24.048 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:24.048 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:24.048 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:24.048 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:29:24.049 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:24.049 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:24.049 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:24.049 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:24.050 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:24.050 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:24.050 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:29:24.050 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:24.051 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:24.051 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:24.051 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:24.051 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:24.051 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:24.052 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:24.052 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:24.052 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:24.052 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:24.053 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:24.053 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:29:24.053 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:24.053 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:24.054 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:24.054 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:24.054 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:24.054 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:24.054 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:24.055 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:24.055 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:24.055 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:24.055 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:24.056 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:29:24.056 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:24.056 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:24.056 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:24.056 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:24.057 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:24.057 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:24.057 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:24.058 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:24.058 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:24.058 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:24.058 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:29:24.059 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:24.059 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:29:24.059 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:29:24.059 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:29:24.060 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:24.060 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:29:24Z" 2022-09-09T22:29:24.060 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:29:24.062 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662762486.146147_45.1 target= 2022-09-09T22:29:29.063 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:29:29.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:29:29.064 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:29:29.111 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:29:29.112 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:29:29.174 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:29:29.174+0000 7f7830e0cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:29:29.176 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:29:29.174+0000 7f7830e0cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:29:29.200 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:29:29.201 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:29:29.202 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:29:29.202 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:29:29.202 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:29:29.203 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:29:29.203 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:29.204 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:29.204 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:29:29.205 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:29.205 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:29:29.206 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:29.206 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:29.207 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:29.208 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:29.208 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:29:29.210 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:29.211 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:29.212 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:29.212 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:29.213 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:29.213 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:29.213 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:29:29.214 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:29.214 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:29.215 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:29.215 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:29.216 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:29.216 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:29.217 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:29.217 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:29.218 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:29.218 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:29.219 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:29.219 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:29:29.220 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:29.220 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:29.221 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:29.221 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:29.222 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:29.222 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:29.223 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:29.223 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:29.223 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:29.224 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:29.224 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:29.225 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:29:29.225 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:29.226 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:29.226 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:29.227 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:29.227 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:29.228 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:29.228 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:29.229 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:29.229 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:29.230 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:29.230 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:29:29.230 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:29.231 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:29:29.231 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:29:29.232 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:29:29.233 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:29.234 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:29:29Z" 2022-09-09T22:29:29.234 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:29:29.236 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662762486.146147_45.1 target= 2022-09-09T22:29:34.237 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:29:34.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:29:34.239 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:29:34.287 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:29:34.288 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:29:34.348 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:29:34.346+0000 7fb0ce05bdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:29:34.350 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:29:34.346+0000 7fb0ce05bdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:29:34.379 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:29:34.380 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:29:34.380 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:29:34.380 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:29:34.381 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:29:34.381 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:29:34.381 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:34.381 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:34.382 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:29:34.382 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:34.382 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:29:34.382 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:34.383 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:34.383 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:34.383 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:34.383 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:29:34.384 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:34.384 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:34.384 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:34.385 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:34.386 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:34.386 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:34.386 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:29:34.386 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:34.387 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:34.387 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:34.388 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:34.388 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:34.388 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:34.389 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:34.389 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:34.389 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:34.390 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:34.390 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:34.390 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:29:34.391 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:34.391 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:34.392 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:34.392 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:34.392 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:34.393 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:34.393 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:34.394 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:34.394 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:34.395 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:34.395 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:34.396 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:29:34.396 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:34.397 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:34.397 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:34.397 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:34.398 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:34.398 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:34.398 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:34.399 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:34.399 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:34.399 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:34.399 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:29:34.400 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:34.400 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:29:34.400 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:29:34.401 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:29:34.401 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:34.401 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:29:34Z" 2022-09-09T22:29:34.402 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:29:34.403 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662762486.146147_45.1 target= 2022-09-09T22:29:39.405 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:29:39.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:29:39.407 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:29:39.453 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:29:39.454 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:29:39.520 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:29:39.518+0000 7fc9ae496dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:29:39.523 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:29:39.522+0000 7fc9ae496dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:29:39.551 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:29:39.552 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:29:39.554 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:29:39.554 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:29:39.555 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:29:39.555 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:29:39.555 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:39.555 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:39.556 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:29:39.556 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:39.556 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:29:39.556 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:39.556 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:39.557 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:39.557 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:39.557 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:29:39.557 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:39.558 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:39.558 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:39.558 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:39.558 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:39.559 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:39.559 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:29:39.559 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:39.559 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:39.560 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:39.560 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:39.560 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:39.560 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:39.560 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:39.561 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:39.561 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:39.561 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:39.561 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:39.562 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:29:39.562 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:39.562 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:39.562 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:39.563 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:39.563 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:39.563 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:39.563 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:39.564 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:39.564 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:39.564 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:39.564 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:39.564 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:29:39.565 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:39.565 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:39.565 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:39.565 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:39.566 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:39.566 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:39.566 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:39.566 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:39.567 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:39.567 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:39.567 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:29:39.567 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:39.567 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:29:39.568 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:29:39.568 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:29:39.568 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:39.568 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:29:39Z" 2022-09-09T22:29:39.569 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:29:39.570 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662762486.146147_45.1 target= 2022-09-09T22:29:44.572 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:29:44.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:29:44.573 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:29:44.621 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:29:44.622 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:29:44.679 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:29:44.678+0000 7f22610d0dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:29:44.681 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:29:44.678+0000 7f22610d0dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:29:44.708 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:29:44.709 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:29:44.710 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:29:44.710 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:29:44.713 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:29:44.713 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:29:44.713 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:44.714 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:44.714 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:29:44.714 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:44.715 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:29:44.715 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:44.715 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:44.715 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:44.716 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:44.716 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:29:44.716 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:44.716 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:44.717 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:44.717 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:44.717 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:44.717 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:44.717 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:29:44.718 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:44.718 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:44.718 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:44.718 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:44.719 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:44.719 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:44.719 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:44.719 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:44.720 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:44.720 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:44.720 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:44.720 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:29:44.720 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:44.721 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:44.721 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:44.721 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:44.721 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:44.722 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:44.722 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:44.722 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:44.723 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:44.723 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:44.723 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:44.723 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:29:44.724 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:44.724 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:44.724 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:44.724 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:44.725 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:44.725 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:44.725 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:44.725 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:44.725 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:44.726 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:44.726 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:29:44.726 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:44.726 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:29:44.727 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:29:44.727 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:29:44.727 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:44.727 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:29:44Z" 2022-09-09T22:29:44.728 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:29:44.729 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662762486.146147_45.1 target= 2022-09-09T22:29:49.730 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:29:49.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:29:49.731 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:29:49.780 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:29:49.781 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:29:49.846 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:29:49.846+0000 7fba07507dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:29:49.848 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:29:49.846+0000 7fba07507dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:29:49.878 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:29:49.878 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:29:49.879 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:29:49.879 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:29:49.879 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:29:49.879 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:29:49.880 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:49.880 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:49.880 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:29:49.881 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:49.881 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:29:49.881 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:49.882 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:49.882 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:49.883 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:49.883 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:29:49.884 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:49.884 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:49.884 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:49.885 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:49.885 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:49.886 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:49.886 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:29:49.887 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:49.888 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:49.888 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:49.888 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:49.889 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:49.889 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:49.889 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:49.890 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:49.890 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:49.891 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:49.891 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:49.891 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:29:49.892 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:49.892 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:49.893 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:49.894 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:49.894 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:49.894 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:49.895 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:49.895 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:49.896 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:49.896 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:49.896 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:49.897 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:29:49.897 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:49.897 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:49.898 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:49.899 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:49.899 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:49.899 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:49.900 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:49.900 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:49.901 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:49.901 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:49.902 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:29:49.902 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:49.903 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:29:49.904 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:29:49.904 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:29:49.905 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:49.905 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:29:49Z" 2022-09-09T22:29:49.906 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:29:49.908 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662762486.146147_45.1 target= 2022-09-09T22:29:54.909 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:29:54.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:29:54.911 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:29:54.961 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:29:54.962 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:29:55.030 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:29:55.030+0000 7f775935ddc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:29:55.032 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:29:55.030+0000 7f775935ddc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:29:55.058 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:29:55.060 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:29:55.060 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:29:55.060 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:29:55.061 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:29:55.061 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:29:55.061 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:55.061 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:55.062 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:29:55.062 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:55.062 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:29:55.062 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:55.063 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:55.063 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:55.063 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:55.063 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:29:55.064 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:55.064 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:55.064 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:55.064 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:55.065 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:55.065 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:55.065 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:29:55.065 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:55.066 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:55.066 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:55.066 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:55.066 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:55.067 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:55.067 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:55.067 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:55.067 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:55.068 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:55.068 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:55.068 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:29:55.068 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:55.069 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:55.069 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:55.069 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:55.069 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:55.070 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:55.070 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:55.070 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:55.070 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:55.071 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:55.071 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:29:55.071 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:29:55.071 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:29:55.072 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:29:55.072 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:29:55.072 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:29:55.072 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:29:55.073 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:29:55.073 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:29:55.073 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:29:55.073 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:55.073 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:29:55.074 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:29:55.074 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:55.074 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:29:55.075 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:29:55.075 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:29:55.075 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:29:55.076 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:29:55Z" 2022-09-09T22:29:55.076 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:29:55.078 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662762486.146147_45.1 target= 2022-09-09T22:30:00.078 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:00.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:00.080 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:00.130 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:00.131 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:00.202 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:00.190+0000 7f7183887dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:00.203 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:00.194+0000 7f7183887dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:00.223 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:30:00.224 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:30:00.225 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:30:00.225 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:30:00.226 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:30:00.226 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:30:00.226 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:00.226 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:00.227 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:30:00.227 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:00.227 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:30:00.227 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:00.228 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:00.228 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:00.228 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:00.228 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:30:00.229 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:00.229 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:00.229 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:00.229 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:00.230 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:00.230 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:00.230 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:30:00.230 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:00.231 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:00.231 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:00.231 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:00.232 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:00.232 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:00.232 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:00.233 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:00.233 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:00.234 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:00.234 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:00.234 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:30:00.234 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:00.235 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:00.235 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:00.235 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:00.235 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:00.236 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:00.236 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:00.236 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:00.236 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:00.236 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:00.237 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:00.237 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:30:00.237 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:00.237 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:00.238 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:00.238 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:00.238 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:00.238 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:00.238 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:00.239 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:00.239 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:00.239 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:00.239 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:30:00.240 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:00.240 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:30:00.240 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:30:00.240 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:30:00.240 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:00.241 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:30:00Z" 2022-09-09T22:30:00.241 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:30:00.244 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662762486.146147_45.1 target= 2022-09-09T22:30:05.246 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:05.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:05.248 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:05.320 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:05.320 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:05.384 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:05.382+0000 7f6f7daaddc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:05.387 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:05.386+0000 7f6f7daaddc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:05.419 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:30:05.419 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:30:05.420 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:30:05.420 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:30:05.421 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:30:05.421 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:30:05.422 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:05.422 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:05.423 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:30:05.423 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:05.424 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:30:05.424 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:05.425 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:05.425 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:05.426 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:05.427 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:30:05.427 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:05.428 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:30:05.428 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:05.429 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:05.429 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:05.430 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:05.430 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:30:05.431 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:05.431 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:05.432 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:05.432 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:05.432 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:05.433 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:05.433 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:05.433 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:05.434 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:05.434 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:05.434 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:05.435 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:30:05.435 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:05.435 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:05.435 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:05.436 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:05.436 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:05.436 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:05.437 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:05.438 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:05.438 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:05.438 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:05.438 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:05.439 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:30:05.439 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:05.439 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:05.439 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:05.439 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:05.440 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:05.440 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:05.440 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:05.440 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:05.441 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:05.441 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:05.441 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:30:05.442 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:05.442 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:30:05.442 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:30:05.443 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:30:05.443 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:05.443 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:30:05Z" 2022-09-09T22:30:05.443 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:30:05.444 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-09T22:30:05.444 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-09T22:30:05.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:05.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:05.445 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:05.509 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:05.509 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:05.565 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:30:05.563+0000 7feeeb318dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:05.566 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:30:05.563+0000 7feeeb318dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:05.589 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:30:05.589 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T22:30:05.589 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T22:30:05.590 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T22:30:05.590 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T22:30:05.590 INFO:teuthology.orchestra.run.smithi005.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:30:05.590 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:05.591 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:05.591 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T22:30:05.591 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:05.591 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T22:30:05.592 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:05.592 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:05.592 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:05.592 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:05.593 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 1, 2022-09-09T22:30:05.593 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:05.593 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:30:05.593 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:05.594 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:05.594 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:05.594 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:05.594 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T22:30:05.595 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:05.595 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:05.595 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:30:05.595 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:05.596 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:30:05.596 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:05.597 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:30:05.597 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:05.598 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:05.598 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:05.599 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:05.599 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T22:30:05.599 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:05.600 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:05.600 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:30:05.600 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:05.601 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:30:05.601 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:05.602 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:30:05.602 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:05.603 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:05.603 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:05.604 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:05.604 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T22:30:05.604 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:05.605 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:05.605 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:30:05.606 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:05.606 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:30:05.606 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:05.607 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:30:05.608 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:05.609 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:05.609 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:05.609 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:30:05.610 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:05.610 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T22:30:05.610 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 1, 2022-09-09T22:30:05.611 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 1 2022-09-09T22:30:05.611 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:05.612 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T22:30:05Z" 2022-09-09T22:30:05.612 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:30:05.614 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-09T22:30:05.614 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-09T22:30:05.615 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:05.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:05.617 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:05.660 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:05.660 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:05.712 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:05.710+0000 7f722ab8cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:05.714 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:05.714+0000 7f722ab8cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:05.738 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:30:05.738 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:30:05.738 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:30:05.742 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:30:05.742 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:30:05.743 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:30:05.743 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:05.743 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:05.744 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:30:05.744 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:05.745 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:30:05.746 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:05.747 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:05.747 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:05.748 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:05.748 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:30:05.749 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:05.749 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:30:05.750 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:05.750 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:05.751 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:05.751 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:05.752 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:30:05.752 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:05.753 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:05.753 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:05.754 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:05.755 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:05.755 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:05.756 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:05.756 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:05.757 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:05.758 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:05.758 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:05.759 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:30:05.759 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:05.760 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:05.761 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:05.761 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:05.761 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:05.762 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:05.762 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:05.763 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:05.764 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:05.764 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:05.764 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:05.765 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:30:05.765 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:05.766 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:05.766 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:05.767 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:05.768 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:05.768 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:05.769 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:05.769 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:05.770 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:05.770 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:05.771 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:30:05.771 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:05.771 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:30:05.772 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:30:05.772 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:30:05.773 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:05.773 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:30:05Z" 2022-09-09T22:30:05.774 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:30:05.777 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-09T22:30:05.778 INFO:rgw_multi.tests:meta checkpoint 2022-09-09T22:30:05.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:05.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:05.780 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:05.828 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:05.828 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:05.883 INFO:teuthology.orchestra.run.smithi005.stderr:No --period given, using current period=503b1d61-4513-48c8-a308-17b76219610c 2022-09-09T22:30:05.909 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T22:30:05.910 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:05.910 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:05.911 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:28:06.146147Z" 2022-09-09T22:30:05.911 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:05.911 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:05.912 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:30:05.912 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:30:05.912 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:05.913 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:05.913 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:30:05.914 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:30:05.914 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:05.915 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:05.915 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:30:05.915 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:30:05.916 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:05.916 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T22:30:05.918 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-09T22:30:05.919 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:05.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:05.922 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:05.971 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:05.972 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:06.037 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:06.034+0000 7f7ff4c2adc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:06.042 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:06.038+0000 7f7ff4c2adc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:06.064 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:30:06.064 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:30:06.065 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:30:06.065 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:30:06.066 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:30:06.066 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:30:06.066 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:06.067 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:06.067 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:30:06.067 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:06.068 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:30:06.068 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:06.068 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:06.069 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:06.069 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:06.070 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:30:06.070 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:06.071 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:30:06.071 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:06.071 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:06.072 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:06.072 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:06.072 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:30:06.072 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:06.073 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:06.073 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:06.073 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:06.073 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:06.074 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:06.074 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:06.075 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:06.075 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:06.075 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:06.075 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:06.076 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:30:06.076 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:06.076 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:06.077 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:06.077 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:06.078 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:06.078 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:06.078 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:06.079 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:06.079 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:06.080 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:06.080 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:06.081 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:30:06.081 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:06.081 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:06.082 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:06.082 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:06.083 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:06.083 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:06.084 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:06.084 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:06.084 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:06.085 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:06.085 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:30:06.086 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:06.086 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:30:06.086 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:30:06.087 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:30:06.087 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:06.087 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:30:06Z" 2022-09-09T22:30:06.088 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:30:06.089 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-09T22:30:06.090 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-09T22:30:06.091 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:06.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:06.092 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:06.141 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:06.142 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:06.197 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:30:06.195+0000 7fbc53432dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:06.198 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:30:06.199+0000 7fbc53432dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:06.223 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:30:06.224 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T22:30:06.226 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T22:30:06.227 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T22:30:06.228 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T22:30:06.229 INFO:teuthology.orchestra.run.smithi005.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:30:06.230 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:06.230 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:06.231 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T22:30:06.231 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:06.232 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T22:30:06.233 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:06.233 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:06.234 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:06.234 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:06.235 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 1, 2022-09-09T22:30:06.235 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:06.236 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:30:06.236 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:06.237 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:06.237 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:06.237 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:06.238 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T22:30:06.239 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:06.240 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:06.240 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:30:06.240 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:06.241 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:30:06.241 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:06.242 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:30:06.242 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:06.242 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:06.243 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:06.243 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:06.244 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T22:30:06.244 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:06.244 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:06.245 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:30:06.245 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:06.245 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:30:06.246 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:06.246 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:30:06.247 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:06.247 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:06.248 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:06.249 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:06.249 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T22:30:06.249 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:06.251 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:06.251 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:30:06.252 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:06.252 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:30:06.253 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:06.253 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:30:06.254 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:06.254 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:06.255 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:06.255 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:30:06.255 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:06.256 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T22:30:06.256 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 1, 2022-09-09T22:30:06.257 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 1 2022-09-09T22:30:06.257 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:06.257 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T22:30:06Z" 2022-09-09T22:30:06.258 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:30:06.260 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-09T22:30:06.261 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-09T22:30:06.262 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:06.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:06.263 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:06.310 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:06.310 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:06.377 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:06.374+0000 7f989d73ddc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:06.379 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:06.378+0000 7f989d73ddc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:06.404 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:30:06.405 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:30:06.405 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:30:06.405 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:30:06.405 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:30:06.406 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:30:06.406 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:06.406 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:06.407 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:30:06.407 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:06.407 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:30:06.407 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:06.408 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:06.408 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:06.408 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:06.409 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:30:06.410 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:06.410 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:30:06.411 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:06.411 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:06.412 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:06.412 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:06.412 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:30:06.413 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:06.413 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:06.413 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:06.413 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:06.414 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:06.414 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:06.414 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:06.414 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:06.415 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:06.415 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:06.415 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:06.416 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:30:06.416 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:06.416 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:06.416 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:06.417 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:06.417 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:06.417 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:06.417 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:06.418 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:06.418 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:06.418 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:06.418 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:06.419 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:30:06.420 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:06.420 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:06.420 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:06.421 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:06.421 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:06.421 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:06.422 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:06.422 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:06.423 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:06.423 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:06.423 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:30:06.424 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:06.424 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:30:06.425 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:30:06.425 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:30:06.425 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:06.426 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:30:06Z" 2022-09-09T22:30:06.426 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:30:06.428 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-09T22:30:06.428 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/git.ceph.com_ceph-c_28758235ba2ab1c1d8afa0f88d00f07b5687a51e/qa/../src/test/rgw/rgw_multi' with args=['tests.py', 'tests_ps.py'] 2022-09-09T22:30:06.522 DEBUG:boto:Using access key provided by client. 2022-09-09T22:30:06.536 DEBUG:boto:Using secret key provided by client. 2022-09-09T22:30:06.537 DEBUG:boto:Using access key provided by client. 2022-09-09T22:30:06.540 DEBUG:boto:Using secret key provided by client. 2022-09-09T22:30:06.541 DEBUG:boto:Using access key provided by client. 2022-09-09T22:30:06.542 DEBUG:boto:Using secret key provided by client. 2022-09-09T22:30:06.543 DEBUG:boto:Using access key provided by client. 2022-09-09T22:30:06.544 DEBUG:boto:Using secret key provided by client. 2022-09-09T22:30:06.545 DEBUG:boto:Using access key provided by client. 2022-09-09T22:30:06.546 DEBUG:boto:Using secret key provided by client. 2022-09-09T22:30:06.547 DEBUG:boto:Using access key provided by client. 2022-09-09T22:30:06.549 DEBUG:boto:Using secret key provided by client. 2022-09-09T22:30:06.550 DEBUG:boto:Using access key provided by client. 2022-09-09T22:30:06.551 DEBUG:boto:Using secret key provided by client. 2022-09-09T22:30:06.553 DEBUG:boto:Using access key provided by client. 2022-09-09T22:30:06.554 DEBUG:boto:Using secret key provided by client. 2022-09-09T22:30:06.555 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jkcgbj-1 2022-09-09T22:30:06.556 DEBUG:boto:path=/jkcgbj-1/ 2022-09-09T22:30:06.557 DEBUG:boto:auth_path=/jkcgbj-1/ 2022-09-09T22:30:06.558 DEBUG:boto:Method: PUT 2022-09-09T22:30:06.558 DEBUG:boto:Path: /jkcgbj-1/ 2022-09-09T22:30:06.559 DEBUG:boto:Data: 2022-09-09T22:30:06.560 DEBUG:boto:Headers: {} 2022-09-09T22:30:06.560 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:06.561 DEBUG:boto:Port: 8000 2022-09-09T22:30:06.561 DEBUG:boto:Params: {} 2022-09-09T22:30:06.562 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T22:30:06.563 DEBUG:boto:Token: None 2022-09-09T22:30:06.564 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:30:06 GMT /jkcgbj-1/ 2022-09-09T22:30:06.566 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:yWeJMErsgRxfyeg/C8f+9xerjNg= 2022-09-09T22:30:06.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:yWeJMErsgRxfyeg/C8f+9xerjNg=', 'Content-Length': '0'} 2022-09-09T22:30:09.640 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004a945ac6f957659e-00631bbe6e-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:09.641 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jkcgbj-2 2022-09-09T22:30:09.641 DEBUG:boto:path=/jkcgbj-2/ 2022-09-09T22:30:09.642 DEBUG:boto:auth_path=/jkcgbj-2/ 2022-09-09T22:30:09.642 DEBUG:boto:Method: PUT 2022-09-09T22:30:09.643 DEBUG:boto:Path: /jkcgbj-2/ 2022-09-09T22:30:09.645 DEBUG:boto:Data: 2022-09-09T22:30:09.645 DEBUG:boto:Headers: {} 2022-09-09T22:30:09.646 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:09.646 DEBUG:boto:Port: 8000 2022-09-09T22:30:09.647 DEBUG:boto:Params: {} 2022-09-09T22:30:09.647 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T22:30:09.648 DEBUG:boto:Token: None 2022-09-09T22:30:09.648 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:30:09 GMT /jkcgbj-2/ 2022-09-09T22:30:09.649 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9XogAhbeVwk+Oz7fhzpSKlwxakE= 2022-09-09T22:30:09.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9XogAhbeVwk+Oz7fhzpSKlwxakE=', 'Content-Length': '0'} 2022-09-09T22:30:12.723 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000013d1bf888fa48cef-00631bbe71-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:12.724 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jkcgbj-3 2022-09-09T22:30:12.724 DEBUG:boto:path=/jkcgbj-3/ 2022-09-09T22:30:12.725 DEBUG:boto:auth_path=/jkcgbj-3/ 2022-09-09T22:30:12.726 DEBUG:boto:Method: PUT 2022-09-09T22:30:12.727 DEBUG:boto:Path: /jkcgbj-3/ 2022-09-09T22:30:12.728 DEBUG:boto:Data: 2022-09-09T22:30:12.729 DEBUG:boto:Headers: {} 2022-09-09T22:30:12.731 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:12.732 DEBUG:boto:Port: 8001 2022-09-09T22:30:12.733 DEBUG:boto:Params: {} 2022-09-09T22:30:12.734 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-09T22:30:12.736 DEBUG:boto:Token: None 2022-09-09T22:30:12.737 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:30:12 GMT /jkcgbj-3/ 2022-09-09T22:30:12.737 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0vc8bz/JImBKdgfAjNADJaje0ME= 2022-09-09T22:30:12.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0vc8bz/JImBKdgfAjNADJaje0ME=', 'Content-Length': '0'} 2022-09-09T22:30:12.784 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000acb83a0ab2f6e4e3-00631bbe74-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:12.785 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:12.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:12.786 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:12.832 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:12.833 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:12.881 INFO:teuthology.orchestra.run.smithi005.stderr:No --period given, using current period=503b1d61-4513-48c8-a308-17b76219610c 2022-09-09T22:30:12.908 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T22:30:12.908 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:12.908 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:12.909 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:28:06.146147Z" 2022-09-09T22:30:12.909 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:12.909 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:12.909 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:30:12.910 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:30:12.910 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:12.910 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:12.910 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762612.770414_49.1", 2022-09-09T22:30:12.911 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:30:12.770414Z" 2022-09-09T22:30:12.911 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:12.911 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:12.911 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:30:12.912 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:30:12.912 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:12.912 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T22:30:12.913 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762612.770414_49.1', 3: ''} 2022-09-09T22:30:12.914 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-09T22:30:12.915 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:12.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:12.916 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:12.964 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:12.964 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:13.020 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:13.018+0000 7f6efba7edc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:13.023 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:13.022+0000 7f6efba7edc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:13.051 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:30:13.052 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:30:13.052 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:30:13.052 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:30:13.052 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:30:13.053 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:30:13.053 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:13.053 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:13.054 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:30:13.054 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:13.054 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:30:13.055 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:13.055 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:13.055 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:13.055 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:13.056 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:30:13.056 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:13.056 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:30:13.057 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:13.057 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:13.057 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:13.058 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:13.058 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:30:13.059 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:13.059 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:13.059 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:13.060 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:13.060 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:13.060 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:13.061 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:13.062 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:13.063 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:13.063 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:13.064 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:13.064 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:30:13.065 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:13.065 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:13.065 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762609.632258_38.1", 2022-09-09T22:30:13.066 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:13.066 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:13.066 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:13.066 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:09.632258Z", 2022-09-09T22:30:13.067 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:13.067 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:13.067 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:13.068 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:13.068 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:30:13.068 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:13.068 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:13.069 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:13.069 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:13.069 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:13.070 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:13.070 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:13.070 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:13.071 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:13.071 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:13.071 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:30:13.071 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:13.072 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:30:13.072 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:30:13.072 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:30:13.072 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:13.073 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:30:13Z" 2022-09-09T22:30:13.073 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:30:13.075 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762609.632258_38.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:09.632258Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:30:13Z" } 2022-09-09T22:30:13.075 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762609.632258_38.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:09.632258Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2022-09-09T22:30:13.076 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762612.770414_49.1', 3: ''} 2022-09-09T22:30:13.077 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762609.632258_38.1', 3: ''} 2022-09-09T22:30:13.077 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662762612.770414_49.1 target=1_1662762609.632258_38.1 2022-09-09T22:30:18.078 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:18.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:18.080 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:18.128 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:18.129 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:18.183 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:18.182+0000 7fcaba8a1dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:18.185 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:18.186+0000 7fcaba8a1dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:18.210 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:30:18.210 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:30:18.210 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:30:18.211 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:30:18.211 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:30:18.211 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:30:18.212 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:18.212 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:18.212 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:30:18.213 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:18.213 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:30:18.213 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:18.214 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:18.214 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:18.214 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:18.215 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:30:18.215 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:18.216 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:30:18.216 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:18.216 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:18.217 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:18.217 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:18.217 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:30:18.218 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:18.218 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:18.218 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:18.219 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:18.219 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:18.219 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:18.220 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:18.220 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:18.220 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:18.220 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:18.221 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:18.221 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:30:18.221 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:18.222 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:18.222 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762612.770414_49.1", 2022-09-09T22:30:18.222 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:18.223 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:18.223 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:18.223 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:12.770414Z", 2022-09-09T22:30:18.223 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:18.224 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:18.224 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:18.224 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:18.225 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:30:18.225 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:18.225 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:18.226 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:18.227 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:18.227 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:18.228 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:18.228 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:18.228 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:18.228 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:18.229 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:18.229 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:30:18.229 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:18.230 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:30:18.230 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:30:18.231 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:30:18.231 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:18.231 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:30:18Z" 2022-09-09T22:30:18.232 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:30:18.233 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762612.770414_49.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:12.770414Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:30:18Z" } 2022-09-09T22:30:18.234 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762612.770414_49.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:12.770414Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2022-09-09T22:30:18.235 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762612.770414_49.1', 3: ''} 2022-09-09T22:30:18.235 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762612.770414_49.1', 3: ''} 2022-09-09T22:30:18.236 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-09T22:30:18.236 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-09T22:30:18.237 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:18.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:18.238 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:18.287 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:18.288 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:18.345 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:30:18.343+0000 7f0264456dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:18.347 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:30:18.347+0000 7f0264456dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:18.371 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:30:18.371 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T22:30:18.371 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T22:30:18.372 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T22:30:18.372 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T22:30:18.372 INFO:teuthology.orchestra.run.smithi005.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:30:18.373 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:18.373 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:18.373 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T22:30:18.373 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:18.374 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T22:30:18.375 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:18.375 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:18.376 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:18.376 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:18.377 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 1, 2022-09-09T22:30:18.377 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:18.378 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:30:18.378 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:18.379 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:18.379 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:18.380 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:18.381 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T22:30:18.381 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:18.382 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:18.382 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:30:18.383 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:18.383 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:30:18.384 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:18.384 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:30:18.385 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:18.385 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:18.386 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:18.386 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:18.387 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T22:30:18.387 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:18.388 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:18.388 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762612.770414_49.1", 2022-09-09T22:30:18.389 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:18.389 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:30:18.390 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:18.390 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:12.770414Z", 2022-09-09T22:30:18.390 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:18.390 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:18.391 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:18.392 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:18.392 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T22:30:18.392 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:18.393 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:18.393 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:30:18.393 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:18.393 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:30:18.394 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:18.394 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:30:18.394 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:18.395 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:18.395 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:18.395 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:30:18.396 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:18.396 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T22:30:18.396 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 1, 2022-09-09T22:30:18.397 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 1 2022-09-09T22:30:18.397 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:18.397 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T22:30:18Z" 2022-09-09T22:30:18.397 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:30:18.399 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762612.770414_49.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:12.770414Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:30:18Z" } 2022-09-09T22:30:18.400 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762612.770414_49.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:12.770414Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2022-09-09T22:30:18.400 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762612.770414_49.1', 3: ''} 2022-09-09T22:30:18.401 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762612.770414_49.1', 3: ''} 2022-09-09T22:30:18.401 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-09T22:30:18.402 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-09T22:30:18.403 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:18.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:18.405 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:18.455 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:18.456 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:18.510 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:18.510+0000 7f3e2493cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:18.513 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:18.510+0000 7f3e2493cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:18.540 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:30:18.541 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:30:18.541 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:30:18.541 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:30:18.541 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:30:18.542 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:30:18.542 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:18.542 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:18.543 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:30:18.543 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:18.543 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:30:18.543 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:18.544 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:18.544 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:18.544 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:18.545 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:30:18.545 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:18.545 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:30:18.545 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:18.546 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:18.546 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:18.546 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:18.547 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:30:18.547 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:18.547 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:18.547 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:18.548 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:18.548 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:18.548 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:18.549 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:18.549 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:18.550 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:18.550 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:18.551 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:18.553 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:30:18.553 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:18.554 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:18.554 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762612.770414_49.1", 2022-09-09T22:30:18.555 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:18.555 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:18.555 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:18.556 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:12.770414Z", 2022-09-09T22:30:18.556 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:18.557 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:18.557 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:18.558 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:18.558 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:30:18.559 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:18.559 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:18.559 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:18.559 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:18.560 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:18.560 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:18.560 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:18.561 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:18.561 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:18.561 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:18.562 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:30:18.562 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:18.562 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:30:18.563 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:30:18.563 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:30:18.563 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:18.564 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:30:18Z" 2022-09-09T22:30:18.564 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:30:18.565 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762612.770414_49.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:12.770414Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:30:18Z" } 2022-09-09T22:30:18.566 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762612.770414_49.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:12.770414Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2022-09-09T22:30:18.567 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762612.770414_49.1', 3: ''} 2022-09-09T22:30:18.567 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762612.770414_49.1', 3: ''} 2022-09-09T22:30:18.568 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-09T22:30:18.568 DEBUG:boto:path=/jkcgbj-1/ 2022-09-09T22:30:18.569 DEBUG:boto:auth_path=/jkcgbj-1/ 2022-09-09T22:30:18.572 DEBUG:boto:Method: HEAD 2022-09-09T22:30:18.573 DEBUG:boto:Path: /jkcgbj-1/ 2022-09-09T22:30:18.574 DEBUG:boto:Data: 2022-09-09T22:30:18.576 DEBUG:boto:Headers: {} 2022-09-09T22:30:18.577 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:18.577 DEBUG:boto:Port: 8000 2022-09-09T22:30:18.578 DEBUG:boto:Params: {} 2022-09-09T22:30:18.580 DEBUG:boto:Token: None 2022-09-09T22:30:18.581 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:18 GMT /jkcgbj-1/ 2022-09-09T22:30:18.581 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:a8YiXYggTcFRFUn1HbWhls505VE= 2022-09-09T22:30:18.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:a8YiXYggTcFRFUn1HbWhls505VE=', 'Content-Length': '0'} 2022-09-09T22:30:18.588 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e66d95f4af80b1f-00631bbe7a-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:18.589 DEBUG:boto:path=/jkcgbj-2/ 2022-09-09T22:30:18.590 DEBUG:boto:auth_path=/jkcgbj-2/ 2022-09-09T22:30:18.591 DEBUG:boto:Method: HEAD 2022-09-09T22:30:18.591 DEBUG:boto:Path: /jkcgbj-2/ 2022-09-09T22:30:18.592 DEBUG:boto:Data: 2022-09-09T22:30:18.593 DEBUG:boto:Headers: {} 2022-09-09T22:30:18.593 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:18.594 DEBUG:boto:Port: 8000 2022-09-09T22:30:18.594 DEBUG:boto:Params: {} 2022-09-09T22:30:18.595 DEBUG:boto:Token: None 2022-09-09T22:30:18.595 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:18 GMT /jkcgbj-2/ 2022-09-09T22:30:18.596 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sU2a/QZw/GWXMnuArovW4RHnDr0= 2022-09-09T22:30:18.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sU2a/QZw/GWXMnuArovW4RHnDr0=', 'Content-Length': '0'} 2022-09-09T22:30:18.601 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b9b8251cea568bc-00631bbe7a-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:18.602 DEBUG:boto:path=/jkcgbj-3/ 2022-09-09T22:30:18.602 DEBUG:boto:auth_path=/jkcgbj-3/ 2022-09-09T22:30:18.603 DEBUG:boto:Method: HEAD 2022-09-09T22:30:18.604 DEBUG:boto:Path: /jkcgbj-3/ 2022-09-09T22:30:18.604 DEBUG:boto:Data: 2022-09-09T22:30:18.605 DEBUG:boto:Headers: {} 2022-09-09T22:30:18.605 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:18.606 DEBUG:boto:Port: 8000 2022-09-09T22:30:18.606 DEBUG:boto:Params: {} 2022-09-09T22:30:18.607 DEBUG:boto:Token: None 2022-09-09T22:30:18.608 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:18 GMT /jkcgbj-3/ 2022-09-09T22:30:18.608 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:89+JVYMR5wa9ZpLGCiunQ4soE5I= 2022-09-09T22:30:18.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:89+JVYMR5wa9ZpLGCiunQ4soE5I=', 'Content-Length': '0'} 2022-09-09T22:30:18.613 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069274963ffeb1702-00631bbe7a-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:18.614 DEBUG:boto:path=/jkcgbj-1/ 2022-09-09T22:30:18.615 DEBUG:boto:auth_path=/jkcgbj-1/ 2022-09-09T22:30:18.615 DEBUG:boto:Method: HEAD 2022-09-09T22:30:18.616 DEBUG:boto:Path: /jkcgbj-1/ 2022-09-09T22:30:18.616 DEBUG:boto:Data: 2022-09-09T22:30:18.617 DEBUG:boto:Headers: {} 2022-09-09T22:30:18.618 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:18.618 DEBUG:boto:Port: 8000 2022-09-09T22:30:18.619 DEBUG:boto:Params: {} 2022-09-09T22:30:18.619 DEBUG:boto:Token: None 2022-09-09T22:30:18.620 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:18 GMT /jkcgbj-1/ 2022-09-09T22:30:18.621 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:a8YiXYggTcFRFUn1HbWhls505VE= 2022-09-09T22:30:18.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:a8YiXYggTcFRFUn1HbWhls505VE=', 'Content-Length': '0'} 2022-09-09T22:30:18.626 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bca7339c17c0cf70-00631bbe7a-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:18.628 DEBUG:boto:path=/jkcgbj-2/ 2022-09-09T22:30:18.629 DEBUG:boto:auth_path=/jkcgbj-2/ 2022-09-09T22:30:18.630 DEBUG:boto:Method: HEAD 2022-09-09T22:30:18.631 DEBUG:boto:Path: /jkcgbj-2/ 2022-09-09T22:30:18.632 DEBUG:boto:Data: 2022-09-09T22:30:18.633 DEBUG:boto:Headers: {} 2022-09-09T22:30:18.634 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:18.635 DEBUG:boto:Port: 8000 2022-09-09T22:30:18.637 DEBUG:boto:Params: {} 2022-09-09T22:30:18.638 DEBUG:boto:Token: None 2022-09-09T22:30:18.639 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:18 GMT /jkcgbj-2/ 2022-09-09T22:30:18.641 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sU2a/QZw/GWXMnuArovW4RHnDr0= 2022-09-09T22:30:18.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sU2a/QZw/GWXMnuArovW4RHnDr0=', 'Content-Length': '0'} 2022-09-09T22:30:18.648 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ead696b3b1292d67-00631bbe7a-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:18.650 DEBUG:boto:path=/jkcgbj-3/ 2022-09-09T22:30:18.651 DEBUG:boto:auth_path=/jkcgbj-3/ 2022-09-09T22:30:18.652 DEBUG:boto:Method: HEAD 2022-09-09T22:30:18.652 DEBUG:boto:Path: /jkcgbj-3/ 2022-09-09T22:30:18.653 DEBUG:boto:Data: 2022-09-09T22:30:18.654 DEBUG:boto:Headers: {} 2022-09-09T22:30:18.655 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:18.656 DEBUG:boto:Port: 8000 2022-09-09T22:30:18.657 DEBUG:boto:Params: {} 2022-09-09T22:30:18.658 DEBUG:boto:Token: None 2022-09-09T22:30:18.659 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:18 GMT /jkcgbj-3/ 2022-09-09T22:30:18.660 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:89+JVYMR5wa9ZpLGCiunQ4soE5I= 2022-09-09T22:30:18.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:89+JVYMR5wa9ZpLGCiunQ4soE5I=', 'Content-Length': '0'} 2022-09-09T22:30:18.667 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b896a025ef808783-00631bbe7a-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:18.668 DEBUG:boto:path=/jkcgbj-1/ 2022-09-09T22:30:18.669 DEBUG:boto:auth_path=/jkcgbj-1/ 2022-09-09T22:30:18.670 DEBUG:boto:Method: HEAD 2022-09-09T22:30:18.672 DEBUG:boto:Path: /jkcgbj-1/ 2022-09-09T22:30:18.673 DEBUG:boto:Data: 2022-09-09T22:30:18.673 DEBUG:boto:Headers: {} 2022-09-09T22:30:18.674 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:18.675 DEBUG:boto:Port: 8001 2022-09-09T22:30:18.676 DEBUG:boto:Params: {} 2022-09-09T22:30:18.677 DEBUG:boto:Token: None 2022-09-09T22:30:18.679 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:18 GMT /jkcgbj-1/ 2022-09-09T22:30:18.680 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:a8YiXYggTcFRFUn1HbWhls505VE= 2022-09-09T22:30:18.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:a8YiXYggTcFRFUn1HbWhls505VE=', 'Content-Length': '0'} 2022-09-09T22:30:18.685 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000552f59dedaae756f-00631bbe7a-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:18.686 DEBUG:boto:path=/jkcgbj-2/ 2022-09-09T22:30:18.687 DEBUG:boto:auth_path=/jkcgbj-2/ 2022-09-09T22:30:18.687 DEBUG:boto:Method: HEAD 2022-09-09T22:30:18.688 DEBUG:boto:Path: /jkcgbj-2/ 2022-09-09T22:30:18.688 DEBUG:boto:Data: 2022-09-09T22:30:18.689 DEBUG:boto:Headers: {} 2022-09-09T22:30:18.689 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:18.690 DEBUG:boto:Port: 8001 2022-09-09T22:30:18.690 DEBUG:boto:Params: {} 2022-09-09T22:30:18.691 DEBUG:boto:Token: None 2022-09-09T22:30:18.692 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:18 GMT /jkcgbj-2/ 2022-09-09T22:30:18.692 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sU2a/QZw/GWXMnuArovW4RHnDr0= 2022-09-09T22:30:18.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sU2a/QZw/GWXMnuArovW4RHnDr0=', 'Content-Length': '0'} 2022-09-09T22:30:18.698 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000581a9dc796701322-00631bbe7a-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:18.699 DEBUG:boto:path=/jkcgbj-3/ 2022-09-09T22:30:18.699 DEBUG:boto:auth_path=/jkcgbj-3/ 2022-09-09T22:30:18.700 DEBUG:boto:Method: HEAD 2022-09-09T22:30:18.701 DEBUG:boto:Path: /jkcgbj-3/ 2022-09-09T22:30:18.701 DEBUG:boto:Data: 2022-09-09T22:30:18.702 DEBUG:boto:Headers: {} 2022-09-09T22:30:18.703 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:18.704 DEBUG:boto:Port: 8001 2022-09-09T22:30:18.705 DEBUG:boto:Params: {} 2022-09-09T22:30:18.706 DEBUG:boto:Token: None 2022-09-09T22:30:18.707 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:18 GMT /jkcgbj-3/ 2022-09-09T22:30:18.708 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:89+JVYMR5wa9ZpLGCiunQ4soE5I= 2022-09-09T22:30:18.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:89+JVYMR5wa9ZpLGCiunQ4soE5I=', 'Content-Length': '0'} 2022-09-09T22:30:18.714 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000410e0ee1ffaf3ca7-00631bbe7a-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:18.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2022-09-09T22:30:18.719 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jkcgbj-4 2022-09-09T22:30:18.719 DEBUG:boto:path=/jkcgbj-4/ 2022-09-09T22:30:18.720 DEBUG:boto:auth_path=/jkcgbj-4/ 2022-09-09T22:30:18.720 DEBUG:boto:Method: PUT 2022-09-09T22:30:18.721 DEBUG:boto:Path: /jkcgbj-4/ 2022-09-09T22:30:18.721 DEBUG:boto:Data: 2022-09-09T22:30:18.722 DEBUG:boto:Headers: {} 2022-09-09T22:30:18.723 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:18.723 DEBUG:boto:Port: 8000 2022-09-09T22:30:18.724 DEBUG:boto:Params: {} 2022-09-09T22:30:18.724 DEBUG:boto:Token: None 2022-09-09T22:30:18.725 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:30:18 GMT /jkcgbj-4/ 2022-09-09T22:30:18.725 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qlUsArz4t2f13svfjmnCsFunWNg= 2022-09-09T22:30:18.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qlUsArz4t2f13svfjmnCsFunWNg=', 'Content-Length': '0'} 2022-09-09T22:30:18.752 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b186de70d4d2e2f3-00631bbe7a-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:18.753 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jkcgbj-5 2022-09-09T22:30:18.753 DEBUG:boto:path=/jkcgbj-5/ 2022-09-09T22:30:18.754 DEBUG:boto:auth_path=/jkcgbj-5/ 2022-09-09T22:30:18.755 DEBUG:boto:Method: PUT 2022-09-09T22:30:18.756 DEBUG:boto:Path: /jkcgbj-5/ 2022-09-09T22:30:18.756 DEBUG:boto:Data: 2022-09-09T22:30:18.757 DEBUG:boto:Headers: {} 2022-09-09T22:30:18.759 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:18.760 DEBUG:boto:Port: 8000 2022-09-09T22:30:18.761 DEBUG:boto:Params: {} 2022-09-09T22:30:18.762 DEBUG:boto:Token: None 2022-09-09T22:30:18.763 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:30:18 GMT /jkcgbj-5/ 2022-09-09T22:30:18.764 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:BxtApLIveVYTWH26KZ8W5Hav3v8= 2022-09-09T22:30:18.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:BxtApLIveVYTWH26KZ8W5Hav3v8=', 'Content-Length': '0'} 2022-09-09T22:30:18.824 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000030678ccf7afd976d-00631bbe7a-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:18.825 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jkcgbj-6 2022-09-09T22:30:18.826 DEBUG:boto:path=/jkcgbj-6/ 2022-09-09T22:30:18.826 DEBUG:boto:auth_path=/jkcgbj-6/ 2022-09-09T22:30:18.827 DEBUG:boto:Method: PUT 2022-09-09T22:30:18.827 DEBUG:boto:Path: /jkcgbj-6/ 2022-09-09T22:30:18.828 DEBUG:boto:Data: 2022-09-09T22:30:18.828 DEBUG:boto:Headers: {} 2022-09-09T22:30:18.829 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:18.829 DEBUG:boto:Port: 8001 2022-09-09T22:30:18.830 DEBUG:boto:Params: {} 2022-09-09T22:30:18.831 DEBUG:boto:Token: None 2022-09-09T22:30:18.831 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:30:18 GMT /jkcgbj-6/ 2022-09-09T22:30:18.832 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wOLG6wgOOWPiChhyU8MDRBG5Gqw= 2022-09-09T22:30:18.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wOLG6wgOOWPiChhyU8MDRBG5Gqw=', 'Content-Length': '0'} 2022-09-09T22:30:18.873 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cfe808e0d5a119c9-00631bbe7a-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:18.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:18.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:18.875 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:18.921 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:18.922 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:18.975 INFO:teuthology.orchestra.run.smithi005.stderr:No --period given, using current period=503b1d61-4513-48c8-a308-17b76219610c 2022-09-09T22:30:19.012 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T22:30:19.013 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:19.013 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:19.014 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:28:06.146147Z" 2022-09-09T22:30:19.014 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:19.015 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:19.015 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:30:19.016 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:30:19.016 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:19.017 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:19.017 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762618.860801_61.1", 2022-09-09T22:30:19.018 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:30:18.860801Z" 2022-09-09T22:30:19.018 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:19.019 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:19.021 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:30:19.021 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:30:19.021 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:19.022 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T22:30:19.025 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762618.860801_61.1', 3: ''} 2022-09-09T22:30:19.026 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-09T22:30:19.027 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:19.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:19.028 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:19.072 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:19.072 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:19.129 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:19.126+0000 7f27f4133dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:19.130 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:19.130+0000 7f27f4133dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:19.156 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:30:19.157 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:30:19.158 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:30:19.158 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:30:19.159 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:30:19.159 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:30:19.160 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:19.161 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:19.162 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:30:19.162 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:19.163 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:30:19.164 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:19.164 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:19.165 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:19.166 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:19.167 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:30:19.167 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:19.168 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:30:19.168 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:19.169 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:19.169 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:19.173 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:19.173 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:30:19.173 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:19.174 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:19.174 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:19.175 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:19.176 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:19.177 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:19.178 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:19.179 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:19.179 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:19.180 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:19.180 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:19.181 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:30:19.181 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:19.182 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:19.182 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762618.860801_61.1", 2022-09-09T22:30:19.183 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:19.183 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:19.184 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:19.185 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:18.860801Z", 2022-09-09T22:30:19.186 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:19.186 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:19.187 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:19.187 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:19.189 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:30:19.189 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:19.190 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:19.190 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:19.191 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:19.191 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:19.192 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:19.192 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:19.193 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:19.193 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:19.194 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:19.194 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:30:19.196 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:19.196 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:30:19.197 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:30:19.197 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:30:19.198 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:19.198 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:30:19Z" 2022-09-09T22:30:19.199 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:30:19.201 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762618.860801_61.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:18.860801Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:30:19Z" } 2022-09-09T22:30:19.202 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762618.860801_61.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:18.860801Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2022-09-09T22:30:19.203 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762618.860801_61.1', 3: ''} 2022-09-09T22:30:19.204 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762618.860801_61.1', 3: ''} 2022-09-09T22:30:19.205 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-09T22:30:19.207 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-09T22:30:19.208 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:19.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:19.211 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:19.258 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:19.259 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:19.312 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:30:19.311+0000 7f1290177dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:19.314 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:30:19.315+0000 7f1290177dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:19.339 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:30:19.340 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T22:30:19.340 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T22:30:19.341 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T22:30:19.341 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T22:30:19.342 INFO:teuthology.orchestra.run.smithi005.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:30:19.343 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:19.343 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:19.343 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T22:30:19.344 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:19.344 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T22:30:19.345 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:19.345 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:19.346 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:19.347 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:19.347 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 1, 2022-09-09T22:30:19.348 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:19.348 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:30:19.349 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:19.349 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:19.350 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:19.350 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:19.352 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T22:30:19.353 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:19.353 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:19.354 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:30:19.354 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:19.355 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:30:19.356 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:19.356 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:30:19.357 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:19.357 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:19.360 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:19.361 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:19.361 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T22:30:19.361 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:19.362 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:19.370 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762618.860801_61.1", 2022-09-09T22:30:19.370 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:19.371 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:30:19.371 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:19.372 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:18.860801Z", 2022-09-09T22:30:19.373 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:19.373 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:19.373 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:19.374 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:19.375 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T22:30:19.375 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:19.375 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:19.377 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:30:19.377 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:19.378 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:30:19.379 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:19.379 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:30:19.380 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:19.380 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:19.381 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:19.381 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:30:19.382 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:19.382 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T22:30:19.383 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 1, 2022-09-09T22:30:19.383 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 1 2022-09-09T22:30:19.384 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:19.384 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T22:30:19Z" 2022-09-09T22:30:19.384 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:30:19.387 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762618.860801_61.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:18.860801Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:30:19Z" } 2022-09-09T22:30:19.388 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762618.860801_61.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:18.860801Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2022-09-09T22:30:19.389 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762618.860801_61.1', 3: ''} 2022-09-09T22:30:19.390 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762618.860801_61.1', 3: ''} 2022-09-09T22:30:19.390 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-09T22:30:19.391 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-09T22:30:19.391 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:19.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:19.393 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:19.441 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:19.443 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:19.508 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:19.506+0000 7fea6998fdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:19.509 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:19.510+0000 7fea6998fdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:19.535 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:30:19.536 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:30:19.536 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:30:19.537 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:30:19.537 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:30:19.538 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:30:19.538 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:19.539 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:19.539 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:30:19.540 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:19.540 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:30:19.540 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:19.541 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:19.542 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:19.542 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:19.543 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:30:19.543 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:19.544 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:30:19.544 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:19.545 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:19.545 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:19.546 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:19.546 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:30:19.547 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:19.550 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:19.550 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:19.551 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:19.551 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:19.552 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:19.552 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:19.553 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:19.553 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:19.554 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:19.555 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:19.555 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:30:19.555 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:19.556 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:19.556 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762618.860801_61.1", 2022-09-09T22:30:19.557 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:19.557 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:19.557 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:19.557 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:18.860801Z", 2022-09-09T22:30:19.558 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:19.558 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:19.558 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:19.559 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:19.559 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:30:19.559 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:19.560 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:19.560 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:19.561 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:19.561 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:19.562 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:19.563 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:19.563 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:19.564 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:19.564 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:19.565 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:30:19.565 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:19.566 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:30:19.566 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:30:19.567 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:30:19.567 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:19.568 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:30:19Z" 2022-09-09T22:30:19.568 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:30:19.571 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762618.860801_61.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:18.860801Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:30:19Z" } 2022-09-09T22:30:19.574 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762618.860801_61.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:18.860801Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2022-09-09T22:30:19.575 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762618.860801_61.1', 3: ''} 2022-09-09T22:30:19.576 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762618.860801_61.1', 3: ''} 2022-09-09T22:30:19.577 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-09T22:30:19.578 DEBUG:boto:path=/jkcgbj-4/ 2022-09-09T22:30:19.578 DEBUG:boto:auth_path=/jkcgbj-4/ 2022-09-09T22:30:19.579 DEBUG:boto:Method: HEAD 2022-09-09T22:30:19.580 DEBUG:boto:Path: /jkcgbj-4/ 2022-09-09T22:30:19.582 DEBUG:boto:Data: 2022-09-09T22:30:19.582 DEBUG:boto:Headers: {} 2022-09-09T22:30:19.583 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:19.583 DEBUG:boto:Port: 8000 2022-09-09T22:30:19.584 DEBUG:boto:Params: {} 2022-09-09T22:30:19.585 DEBUG:boto:Token: None 2022-09-09T22:30:19.585 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:19 GMT /jkcgbj-4/ 2022-09-09T22:30:19.586 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jLHnyvXD/EDlvACoTA/OcjTjYH4= 2022-09-09T22:30:19.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jLHnyvXD/EDlvACoTA/OcjTjYH4=', 'Content-Length': '0'} 2022-09-09T22:30:19.594 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f243d09b3b5daa16-00631bbe7b-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:19.596 DEBUG:boto:path=/jkcgbj-5/ 2022-09-09T22:30:19.597 DEBUG:boto:auth_path=/jkcgbj-5/ 2022-09-09T22:30:19.598 DEBUG:boto:Method: HEAD 2022-09-09T22:30:19.599 DEBUG:boto:Path: /jkcgbj-5/ 2022-09-09T22:30:19.600 DEBUG:boto:Data: 2022-09-09T22:30:19.601 DEBUG:boto:Headers: {} 2022-09-09T22:30:19.605 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:19.606 DEBUG:boto:Port: 8000 2022-09-09T22:30:19.607 DEBUG:boto:Params: {} 2022-09-09T22:30:19.607 DEBUG:boto:Token: None 2022-09-09T22:30:19.608 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:19 GMT /jkcgbj-5/ 2022-09-09T22:30:19.608 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jUBdEGcEGcvsCLjfLSewrYifLBU= 2022-09-09T22:30:19.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jUBdEGcEGcvsCLjfLSewrYifLBU=', 'Content-Length': '0'} 2022-09-09T22:30:19.615 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b55f0bcf10871e10-00631bbe7b-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:19.616 DEBUG:boto:path=/jkcgbj-6/ 2022-09-09T22:30:19.616 DEBUG:boto:auth_path=/jkcgbj-6/ 2022-09-09T22:30:19.617 DEBUG:boto:Method: HEAD 2022-09-09T22:30:19.617 DEBUG:boto:Path: /jkcgbj-6/ 2022-09-09T22:30:19.618 DEBUG:boto:Data: 2022-09-09T22:30:19.619 DEBUG:boto:Headers: {} 2022-09-09T22:30:19.619 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:19.620 DEBUG:boto:Port: 8000 2022-09-09T22:30:19.620 DEBUG:boto:Params: {} 2022-09-09T22:30:19.621 DEBUG:boto:Token: None 2022-09-09T22:30:19.622 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:19 GMT /jkcgbj-6/ 2022-09-09T22:30:19.623 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KjqMbf/Lh3VL6DVWKvB0yHUzErY= 2022-09-09T22:30:19.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KjqMbf/Lh3VL6DVWKvB0yHUzErY=', 'Content-Length': '0'} 2022-09-09T22:30:19.630 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a3be9939dda5878-00631bbe7b-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:19.631 DEBUG:boto:path=/jkcgbj-4/ 2022-09-09T22:30:19.632 DEBUG:boto:auth_path=/jkcgbj-4/ 2022-09-09T22:30:19.633 DEBUG:boto:Method: HEAD 2022-09-09T22:30:19.634 DEBUG:boto:Path: /jkcgbj-4/ 2022-09-09T22:30:19.637 DEBUG:boto:Data: 2022-09-09T22:30:19.638 DEBUG:boto:Headers: {} 2022-09-09T22:30:19.639 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:19.640 DEBUG:boto:Port: 8000 2022-09-09T22:30:19.641 DEBUG:boto:Params: {} 2022-09-09T22:30:19.642 DEBUG:boto:Token: None 2022-09-09T22:30:19.643 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:19 GMT /jkcgbj-4/ 2022-09-09T22:30:19.645 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jLHnyvXD/EDlvACoTA/OcjTjYH4= 2022-09-09T22:30:19.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jLHnyvXD/EDlvACoTA/OcjTjYH4=', 'Content-Length': '0'} 2022-09-09T22:30:19.652 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ad5861fc55be4e5-00631bbe7b-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:19.653 DEBUG:boto:path=/jkcgbj-5/ 2022-09-09T22:30:19.653 DEBUG:boto:auth_path=/jkcgbj-5/ 2022-09-09T22:30:19.656 DEBUG:boto:Method: HEAD 2022-09-09T22:30:19.656 DEBUG:boto:Path: /jkcgbj-5/ 2022-09-09T22:30:19.657 DEBUG:boto:Data: 2022-09-09T22:30:19.658 DEBUG:boto:Headers: {} 2022-09-09T22:30:19.659 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:19.661 DEBUG:boto:Port: 8000 2022-09-09T22:30:19.661 DEBUG:boto:Params: {} 2022-09-09T22:30:19.662 DEBUG:boto:Token: None 2022-09-09T22:30:19.663 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:19 GMT /jkcgbj-5/ 2022-09-09T22:30:19.663 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jUBdEGcEGcvsCLjfLSewrYifLBU= 2022-09-09T22:30:19.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jUBdEGcEGcvsCLjfLSewrYifLBU=', 'Content-Length': '0'} 2022-09-09T22:30:19.673 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc4c23c7e45584bd-00631bbe7b-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:19.674 DEBUG:boto:path=/jkcgbj-6/ 2022-09-09T22:30:19.675 DEBUG:boto:auth_path=/jkcgbj-6/ 2022-09-09T22:30:19.677 DEBUG:boto:Method: HEAD 2022-09-09T22:30:19.677 DEBUG:boto:Path: /jkcgbj-6/ 2022-09-09T22:30:19.678 DEBUG:boto:Data: 2022-09-09T22:30:19.679 DEBUG:boto:Headers: {} 2022-09-09T22:30:19.679 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:19.679 DEBUG:boto:Port: 8000 2022-09-09T22:30:19.680 DEBUG:boto:Params: {} 2022-09-09T22:30:19.681 DEBUG:boto:Token: None 2022-09-09T22:30:19.681 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:19 GMT /jkcgbj-6/ 2022-09-09T22:30:19.682 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KjqMbf/Lh3VL6DVWKvB0yHUzErY= 2022-09-09T22:30:19.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KjqMbf/Lh3VL6DVWKvB0yHUzErY=', 'Content-Length': '0'} 2022-09-09T22:30:19.690 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e125b9f16311cbbd-00631bbe7b-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:19.691 DEBUG:boto:path=/jkcgbj-4/ 2022-09-09T22:30:19.691 DEBUG:boto:auth_path=/jkcgbj-4/ 2022-09-09T22:30:19.692 DEBUG:boto:Method: HEAD 2022-09-09T22:30:19.692 DEBUG:boto:Path: /jkcgbj-4/ 2022-09-09T22:30:19.693 DEBUG:boto:Data: 2022-09-09T22:30:19.694 DEBUG:boto:Headers: {} 2022-09-09T22:30:19.695 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:19.695 DEBUG:boto:Port: 8001 2022-09-09T22:30:19.696 DEBUG:boto:Params: {} 2022-09-09T22:30:19.697 DEBUG:boto:Token: None 2022-09-09T22:30:19.698 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:19 GMT /jkcgbj-4/ 2022-09-09T22:30:19.699 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jLHnyvXD/EDlvACoTA/OcjTjYH4= 2022-09-09T22:30:19.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jLHnyvXD/EDlvACoTA/OcjTjYH4=', 'Content-Length': '0'} 2022-09-09T22:30:19.707 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f309dc884cbe928-00631bbe7b-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:19.709 DEBUG:boto:path=/jkcgbj-5/ 2022-09-09T22:30:19.710 DEBUG:boto:auth_path=/jkcgbj-5/ 2022-09-09T22:30:19.711 DEBUG:boto:Method: HEAD 2022-09-09T22:30:19.711 DEBUG:boto:Path: /jkcgbj-5/ 2022-09-09T22:30:19.712 DEBUG:boto:Data: 2022-09-09T22:30:19.713 DEBUG:boto:Headers: {} 2022-09-09T22:30:19.714 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:19.715 DEBUG:boto:Port: 8001 2022-09-09T22:30:19.716 DEBUG:boto:Params: {} 2022-09-09T22:30:19.717 DEBUG:boto:Token: None 2022-09-09T22:30:19.718 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:19 GMT /jkcgbj-5/ 2022-09-09T22:30:19.719 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jUBdEGcEGcvsCLjfLSewrYifLBU= 2022-09-09T22:30:19.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jUBdEGcEGcvsCLjfLSewrYifLBU=', 'Content-Length': '0'} 2022-09-09T22:30:19.725 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000527044fcc7c2c34a-00631bbe7b-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:19.727 DEBUG:boto:path=/jkcgbj-6/ 2022-09-09T22:30:19.728 DEBUG:boto:auth_path=/jkcgbj-6/ 2022-09-09T22:30:19.729 DEBUG:boto:Method: HEAD 2022-09-09T22:30:19.729 DEBUG:boto:Path: /jkcgbj-6/ 2022-09-09T22:30:19.730 DEBUG:boto:Data: 2022-09-09T22:30:19.731 DEBUG:boto:Headers: {} 2022-09-09T22:30:19.732 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:19.733 DEBUG:boto:Port: 8001 2022-09-09T22:30:19.734 DEBUG:boto:Params: {} 2022-09-09T22:30:19.735 DEBUG:boto:Token: None 2022-09-09T22:30:19.736 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:19 GMT /jkcgbj-6/ 2022-09-09T22:30:19.737 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KjqMbf/Lh3VL6DVWKvB0yHUzErY= 2022-09-09T22:30:19.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KjqMbf/Lh3VL6DVWKvB0yHUzErY=', 'Content-Length': '0'} 2022-09-09T22:30:19.745 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000033215a5058dc0483-00631bbe7b-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:19.746 DEBUG:boto:path=/jkcgbj-4/ 2022-09-09T22:30:19.747 DEBUG:boto:auth_path=/jkcgbj-4/ 2022-09-09T22:30:19.748 DEBUG:boto:Method: PUT 2022-09-09T22:30:19.749 DEBUG:boto:Path: /jkcgbj-4/ 2022-09-09T22:30:19.750 DEBUG:boto:Data: 2022-09-09T22:30:19.751 DEBUG:boto:Headers: {} 2022-09-09T22:30:19.752 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:19.752 DEBUG:boto:Port: 8000 2022-09-09T22:30:19.753 DEBUG:boto:Params: {} 2022-09-09T22:30:19.755 DEBUG:boto:Token: None 2022-09-09T22:30:19.756 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:30:19 GMT /jkcgbj-4/ 2022-09-09T22:30:19.756 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:kb6bpC0T4GdGfDa6qLEZiM5dCwA= 2022-09-09T22:30:19.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:kb6bpC0T4GdGfDa6qLEZiM5dCwA=', 'Content-Length': '0'} 2022-09-09T22:30:19.772 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002118d49345f550bc-00631bbe7b-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:19.775 DEBUG:boto:path=/jkcgbj-5/ 2022-09-09T22:30:19.776 DEBUG:boto:auth_path=/jkcgbj-5/ 2022-09-09T22:30:19.777 DEBUG:boto:Method: PUT 2022-09-09T22:30:19.778 DEBUG:boto:Path: /jkcgbj-5/ 2022-09-09T22:30:19.779 DEBUG:boto:Data: 2022-09-09T22:30:19.780 DEBUG:boto:Headers: {} 2022-09-09T22:30:19.781 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:19.782 DEBUG:boto:Port: 8000 2022-09-09T22:30:19.783 DEBUG:boto:Params: {} 2022-09-09T22:30:19.784 DEBUG:boto:Token: None 2022-09-09T22:30:19.786 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:30:19 GMT /jkcgbj-5/ 2022-09-09T22:30:19.787 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8y9vEfA2YIbhQBqUXqx+poFlyzg= 2022-09-09T22:30:19.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8y9vEfA2YIbhQBqUXqx+poFlyzg=', 'Content-Length': '0'} 2022-09-09T22:30:19.800 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000227aec86de921ab-00631bbe7b-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:19.801 DEBUG:boto:path=/jkcgbj-6/ 2022-09-09T22:30:19.801 DEBUG:boto:auth_path=/jkcgbj-6/ 2022-09-09T22:30:19.802 DEBUG:boto:Method: PUT 2022-09-09T22:30:19.803 DEBUG:boto:Path: /jkcgbj-6/ 2022-09-09T22:30:19.803 DEBUG:boto:Data: 2022-09-09T22:30:19.803 DEBUG:boto:Headers: {} 2022-09-09T22:30:19.804 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:19.804 DEBUG:boto:Port: 8000 2022-09-09T22:30:19.805 DEBUG:boto:Params: {} 2022-09-09T22:30:19.806 DEBUG:boto:Token: None 2022-09-09T22:30:19.814 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:30:19 GMT /jkcgbj-6/ 2022-09-09T22:30:19.815 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:DBBEeuHaUb5FQw7uOfiMpNWAkbA= 2022-09-09T22:30:19.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:DBBEeuHaUb5FQw7uOfiMpNWAkbA=', 'Content-Length': '0'} 2022-09-09T22:30:19.822 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000071e72023b24331d6-00631bbe7b-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:19.823 DEBUG:boto:path=/jkcgbj-4/ 2022-09-09T22:30:19.824 DEBUG:boto:auth_path=/jkcgbj-4/ 2022-09-09T22:30:19.824 DEBUG:boto:Method: PUT 2022-09-09T22:30:19.825 DEBUG:boto:Path: /jkcgbj-4/ 2022-09-09T22:30:19.825 DEBUG:boto:Data: 2022-09-09T22:30:19.834 DEBUG:boto:Headers: {} 2022-09-09T22:30:19.834 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:19.835 DEBUG:boto:Port: 8000 2022-09-09T22:30:19.835 DEBUG:boto:Params: {} 2022-09-09T22:30:19.836 DEBUG:boto:Token: None 2022-09-09T22:30:19.836 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:30:19 GMT /jkcgbj-4/ 2022-09-09T22:30:19.837 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:kb6bpC0T4GdGfDa6qLEZiM5dCwA= 2022-09-09T22:30:19.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:kb6bpC0T4GdGfDa6qLEZiM5dCwA=', 'Content-Length': '0'} 2022-09-09T22:30:19.852 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004c78e146763986ea-00631bbe7b-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:19.852 DEBUG:boto:path=/jkcgbj-5/ 2022-09-09T22:30:19.853 DEBUG:boto:auth_path=/jkcgbj-5/ 2022-09-09T22:30:19.853 DEBUG:boto:Method: PUT 2022-09-09T22:30:19.854 DEBUG:boto:Path: /jkcgbj-5/ 2022-09-09T22:30:19.854 DEBUG:boto:Data: 2022-09-09T22:30:19.855 DEBUG:boto:Headers: {} 2022-09-09T22:30:19.855 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:19.856 DEBUG:boto:Port: 8000 2022-09-09T22:30:19.856 DEBUG:boto:Params: {} 2022-09-09T22:30:19.857 DEBUG:boto:Token: None 2022-09-09T22:30:19.858 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:30:19 GMT /jkcgbj-5/ 2022-09-09T22:30:19.858 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8y9vEfA2YIbhQBqUXqx+poFlyzg= 2022-09-09T22:30:19.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8y9vEfA2YIbhQBqUXqx+poFlyzg=', 'Content-Length': '0'} 2022-09-09T22:30:19.873 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f0ac7f2ed6f59997-00631bbe7b-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:19.873 DEBUG:boto:path=/jkcgbj-6/ 2022-09-09T22:30:19.874 DEBUG:boto:auth_path=/jkcgbj-6/ 2022-09-09T22:30:19.875 DEBUG:boto:Method: PUT 2022-09-09T22:30:19.875 DEBUG:boto:Path: /jkcgbj-6/ 2022-09-09T22:30:19.876 DEBUG:boto:Data: 2022-09-09T22:30:19.877 DEBUG:boto:Headers: {} 2022-09-09T22:30:19.878 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:19.879 DEBUG:boto:Port: 8000 2022-09-09T22:30:19.879 DEBUG:boto:Params: {} 2022-09-09T22:30:19.880 DEBUG:boto:Token: None 2022-09-09T22:30:19.885 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:30:19 GMT /jkcgbj-6/ 2022-09-09T22:30:19.886 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:DBBEeuHaUb5FQw7uOfiMpNWAkbA= 2022-09-09T22:30:19.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:DBBEeuHaUb5FQw7uOfiMpNWAkbA=', 'Content-Length': '0'} 2022-09-09T22:30:19.909 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e8dc74249f2c010a-00631bbe7b-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:19.911 DEBUG:boto:path=/jkcgbj-4/ 2022-09-09T22:30:19.912 DEBUG:boto:auth_path=/jkcgbj-4/ 2022-09-09T22:30:19.913 DEBUG:boto:Method: PUT 2022-09-09T22:30:19.914 DEBUG:boto:Path: /jkcgbj-4/ 2022-09-09T22:30:19.914 DEBUG:boto:Data: 2022-09-09T22:30:19.915 DEBUG:boto:Headers: {} 2022-09-09T22:30:19.922 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:19.923 DEBUG:boto:Port: 8001 2022-09-09T22:30:19.924 DEBUG:boto:Params: {} 2022-09-09T22:30:19.925 DEBUG:boto:Token: None 2022-09-09T22:30:19.927 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:30:19 GMT /jkcgbj-4/ 2022-09-09T22:30:19.928 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:kb6bpC0T4GdGfDa6qLEZiM5dCwA= 2022-09-09T22:30:19.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:kb6bpC0T4GdGfDa6qLEZiM5dCwA=', 'Content-Length': '0'} 2022-09-09T22:30:19.941 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000348f800ca0036a9-00631bbe7b-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:19.942 DEBUG:boto:path=/jkcgbj-5/ 2022-09-09T22:30:19.943 DEBUG:boto:auth_path=/jkcgbj-5/ 2022-09-09T22:30:19.943 DEBUG:boto:Method: PUT 2022-09-09T22:30:19.944 DEBUG:boto:Path: /jkcgbj-5/ 2022-09-09T22:30:19.945 DEBUG:boto:Data: 2022-09-09T22:30:19.945 DEBUG:boto:Headers: {} 2022-09-09T22:30:19.946 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:19.946 DEBUG:boto:Port: 8001 2022-09-09T22:30:19.947 DEBUG:boto:Params: {} 2022-09-09T22:30:19.948 DEBUG:boto:Token: None 2022-09-09T22:30:19.948 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:30:19 GMT /jkcgbj-5/ 2022-09-09T22:30:19.949 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8y9vEfA2YIbhQBqUXqx+poFlyzg= 2022-09-09T22:30:19.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8y9vEfA2YIbhQBqUXqx+poFlyzg=', 'Content-Length': '0'} 2022-09-09T22:30:19.966 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b52ba5dd3481118d-00631bbe7b-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:19.969 DEBUG:boto:path=/jkcgbj-6/ 2022-09-09T22:30:19.970 DEBUG:boto:auth_path=/jkcgbj-6/ 2022-09-09T22:30:19.971 DEBUG:boto:Method: PUT 2022-09-09T22:30:19.972 DEBUG:boto:Path: /jkcgbj-6/ 2022-09-09T22:30:19.973 DEBUG:boto:Data: 2022-09-09T22:30:19.974 DEBUG:boto:Headers: {} 2022-09-09T22:30:19.975 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:19.976 DEBUG:boto:Port: 8001 2022-09-09T22:30:19.977 DEBUG:boto:Params: {} 2022-09-09T22:30:19.978 DEBUG:boto:Token: None 2022-09-09T22:30:19.980 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:30:19 GMT /jkcgbj-6/ 2022-09-09T22:30:19.981 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:DBBEeuHaUb5FQw7uOfiMpNWAkbA= 2022-09-09T22:30:19.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:DBBEeuHaUb5FQw7uOfiMpNWAkbA=', 'Content-Length': '0'} 2022-09-09T22:30:19.997 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000409e8d6a50336784-00631bbe7b-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:19.998 DEBUG:boto:path=/jkcgbj-4/ 2022-09-09T22:30:19.999 DEBUG:boto:auth_path=/jkcgbj-4/ 2022-09-09T22:30:20.000 DEBUG:boto:Method: HEAD 2022-09-09T22:30:20.001 DEBUG:boto:Path: /jkcgbj-4/ 2022-09-09T22:30:20.002 DEBUG:boto:Data: 2022-09-09T22:30:20.003 DEBUG:boto:Headers: {} 2022-09-09T22:30:20.004 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:20.005 DEBUG:boto:Port: 8000 2022-09-09T22:30:20.006 DEBUG:boto:Params: {} 2022-09-09T22:30:20.007 DEBUG:boto:Token: None 2022-09-09T22:30:20.011 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:20 GMT /jkcgbj-4/ 2022-09-09T22:30:20.013 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:NuNL5T1qMO2EOOiJPd32e58S07U= 2022-09-09T22:30:20.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:NuNL5T1qMO2EOOiJPd32e58S07U=', 'Content-Length': '0'} 2022-09-09T22:30:20.020 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c520c7a641a702ac-00631bbe7c-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:20.022 DEBUG:boto:path=/jkcgbj-5/ 2022-09-09T22:30:20.023 DEBUG:boto:auth_path=/jkcgbj-5/ 2022-09-09T22:30:20.024 DEBUG:boto:Method: HEAD 2022-09-09T22:30:20.025 DEBUG:boto:Path: /jkcgbj-5/ 2022-09-09T22:30:20.026 DEBUG:boto:Data: 2022-09-09T22:30:20.027 DEBUG:boto:Headers: {} 2022-09-09T22:30:20.028 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:20.029 DEBUG:boto:Port: 8000 2022-09-09T22:30:20.030 DEBUG:boto:Params: {} 2022-09-09T22:30:20.031 DEBUG:boto:Token: None 2022-09-09T22:30:20.032 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:20 GMT /jkcgbj-5/ 2022-09-09T22:30:20.034 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bmHJbAeUy9cnfWthIRPp6BcEDrs= 2022-09-09T22:30:20.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bmHJbAeUy9cnfWthIRPp6BcEDrs=', 'Content-Length': '0'} 2022-09-09T22:30:20.040 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000053424b297da1527e-00631bbe7c-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:20.041 DEBUG:boto:path=/jkcgbj-6/ 2022-09-09T22:30:20.041 DEBUG:boto:auth_path=/jkcgbj-6/ 2022-09-09T22:30:20.042 DEBUG:boto:Method: HEAD 2022-09-09T22:30:20.043 DEBUG:boto:Path: /jkcgbj-6/ 2022-09-09T22:30:20.044 DEBUG:boto:Data: 2022-09-09T22:30:20.044 DEBUG:boto:Headers: {} 2022-09-09T22:30:20.044 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:20.046 DEBUG:boto:Port: 8000 2022-09-09T22:30:20.047 DEBUG:boto:Params: {} 2022-09-09T22:30:20.049 DEBUG:boto:Token: None 2022-09-09T22:30:20.050 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:20 GMT /jkcgbj-6/ 2022-09-09T22:30:20.051 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0TYC+bNJRocPO3AfwdL23tbqoBw= 2022-09-09T22:30:20.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0TYC+bNJRocPO3AfwdL23tbqoBw=', 'Content-Length': '0'} 2022-09-09T22:30:20.057 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc20c0e0242d708e-00631bbe7c-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:20.058 DEBUG:boto:path=/jkcgbj-4/ 2022-09-09T22:30:20.059 DEBUG:boto:auth_path=/jkcgbj-4/ 2022-09-09T22:30:20.060 DEBUG:boto:Method: HEAD 2022-09-09T22:30:20.061 DEBUG:boto:Path: /jkcgbj-4/ 2022-09-09T22:30:20.062 DEBUG:boto:Data: 2022-09-09T22:30:20.063 DEBUG:boto:Headers: {} 2022-09-09T22:30:20.064 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:20.065 DEBUG:boto:Port: 8000 2022-09-09T22:30:20.066 DEBUG:boto:Params: {} 2022-09-09T22:30:20.067 DEBUG:boto:Token: None 2022-09-09T22:30:20.068 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:20 GMT /jkcgbj-4/ 2022-09-09T22:30:20.069 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:NuNL5T1qMO2EOOiJPd32e58S07U= 2022-09-09T22:30:20.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:NuNL5T1qMO2EOOiJPd32e58S07U=', 'Content-Length': '0'} 2022-09-09T22:30:20.076 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005c911f71050b4f35-00631bbe7c-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:20.077 DEBUG:boto:path=/jkcgbj-5/ 2022-09-09T22:30:20.078 DEBUG:boto:auth_path=/jkcgbj-5/ 2022-09-09T22:30:20.079 DEBUG:boto:Method: HEAD 2022-09-09T22:30:20.080 DEBUG:boto:Path: /jkcgbj-5/ 2022-09-09T22:30:20.081 DEBUG:boto:Data: 2022-09-09T22:30:20.082 DEBUG:boto:Headers: {} 2022-09-09T22:30:20.083 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:20.084 DEBUG:boto:Port: 8000 2022-09-09T22:30:20.084 DEBUG:boto:Params: {} 2022-09-09T22:30:20.086 DEBUG:boto:Token: None 2022-09-09T22:30:20.087 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:20 GMT /jkcgbj-5/ 2022-09-09T22:30:20.088 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bmHJbAeUy9cnfWthIRPp6BcEDrs= 2022-09-09T22:30:20.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bmHJbAeUy9cnfWthIRPp6BcEDrs=', 'Content-Length': '0'} 2022-09-09T22:30:20.095 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000458a9ccd084f882f-00631bbe7c-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:20.096 DEBUG:boto:path=/jkcgbj-6/ 2022-09-09T22:30:20.097 DEBUG:boto:auth_path=/jkcgbj-6/ 2022-09-09T22:30:20.098 DEBUG:boto:Method: HEAD 2022-09-09T22:30:20.099 DEBUG:boto:Path: /jkcgbj-6/ 2022-09-09T22:30:20.100 DEBUG:boto:Data: 2022-09-09T22:30:20.101 DEBUG:boto:Headers: {} 2022-09-09T22:30:20.102 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:20.103 DEBUG:boto:Port: 8000 2022-09-09T22:30:20.104 DEBUG:boto:Params: {} 2022-09-09T22:30:20.105 DEBUG:boto:Token: None 2022-09-09T22:30:20.106 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:20 GMT /jkcgbj-6/ 2022-09-09T22:30:20.107 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0TYC+bNJRocPO3AfwdL23tbqoBw= 2022-09-09T22:30:20.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0TYC+bNJRocPO3AfwdL23tbqoBw=', 'Content-Length': '0'} 2022-09-09T22:30:20.114 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cd27eee6bcce650d-00631bbe7c-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:20.115 DEBUG:boto:path=/jkcgbj-4/ 2022-09-09T22:30:20.118 DEBUG:boto:auth_path=/jkcgbj-4/ 2022-09-09T22:30:20.119 DEBUG:boto:Method: HEAD 2022-09-09T22:30:20.119 DEBUG:boto:Path: /jkcgbj-4/ 2022-09-09T22:30:20.121 DEBUG:boto:Data: 2022-09-09T22:30:20.122 DEBUG:boto:Headers: {} 2022-09-09T22:30:20.123 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:20.124 DEBUG:boto:Port: 8001 2022-09-09T22:30:20.125 DEBUG:boto:Params: {} 2022-09-09T22:30:20.126 DEBUG:boto:Token: None 2022-09-09T22:30:20.127 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:20 GMT /jkcgbj-4/ 2022-09-09T22:30:20.128 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:NuNL5T1qMO2EOOiJPd32e58S07U= 2022-09-09T22:30:20.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:NuNL5T1qMO2EOOiJPd32e58S07U=', 'Content-Length': '0'} 2022-09-09T22:30:20.135 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089fd2bcf709c3190-00631bbe7c-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:20.136 DEBUG:boto:path=/jkcgbj-5/ 2022-09-09T22:30:20.137 DEBUG:boto:auth_path=/jkcgbj-5/ 2022-09-09T22:30:20.138 DEBUG:boto:Method: HEAD 2022-09-09T22:30:20.139 DEBUG:boto:Path: /jkcgbj-5/ 2022-09-09T22:30:20.140 DEBUG:boto:Data: 2022-09-09T22:30:20.141 DEBUG:boto:Headers: {} 2022-09-09T22:30:20.142 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:20.144 DEBUG:boto:Port: 8001 2022-09-09T22:30:20.145 DEBUG:boto:Params: {} 2022-09-09T22:30:20.146 DEBUG:boto:Token: None 2022-09-09T22:30:20.147 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:20 GMT /jkcgbj-5/ 2022-09-09T22:30:20.147 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bmHJbAeUy9cnfWthIRPp6BcEDrs= 2022-09-09T22:30:20.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bmHJbAeUy9cnfWthIRPp6BcEDrs=', 'Content-Length': '0'} 2022-09-09T22:30:20.153 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000017cc89038b3cbf92-00631bbe7c-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:20.154 DEBUG:boto:path=/jkcgbj-6/ 2022-09-09T22:30:20.154 DEBUG:boto:auth_path=/jkcgbj-6/ 2022-09-09T22:30:20.155 DEBUG:boto:Method: HEAD 2022-09-09T22:30:20.156 DEBUG:boto:Path: /jkcgbj-6/ 2022-09-09T22:30:20.156 DEBUG:boto:Data: 2022-09-09T22:30:20.157 DEBUG:boto:Headers: {} 2022-09-09T22:30:20.157 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:20.158 DEBUG:boto:Port: 8001 2022-09-09T22:30:20.158 DEBUG:boto:Params: {} 2022-09-09T22:30:20.159 DEBUG:boto:Token: None 2022-09-09T22:30:20.160 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:20 GMT /jkcgbj-6/ 2022-09-09T22:30:20.160 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0TYC+bNJRocPO3AfwdL23tbqoBw= 2022-09-09T22:30:20.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0TYC+bNJRocPO3AfwdL23tbqoBw=', 'Content-Length': '0'} 2022-09-09T22:30:20.166 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f9ac90b9f9f4d600-00631bbe7c-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:20.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:20.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:20.169 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:20.229 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:20.230 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:20.274 INFO:teuthology.orchestra.run.smithi005.stderr:No --period given, using current period=503b1d61-4513-48c8-a308-17b76219610c 2022-09-09T22:30:20.300 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T22:30:20.300 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:20.301 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:20.301 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:28:06.146147Z" 2022-09-09T22:30:20.302 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:20.302 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:20.303 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:30:20.303 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:30:20.304 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:20.304 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:20.304 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762618.860801_61.1", 2022-09-09T22:30:20.305 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:30:18.860801Z" 2022-09-09T22:30:20.305 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:20.306 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:20.306 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:30:20.307 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:30:20.307 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:20.308 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T22:30:20.310 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762618.860801_61.1', 3: ''} 2022-09-09T22:30:20.311 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-09T22:30:20.311 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:20.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:20.313 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:20.362 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:20.363 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:20.420 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:20.418+0000 7fe1a3ccbdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:20.422 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:20.422+0000 7fe1a3ccbdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:20.454 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:30:20.455 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:30:20.456 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:30:20.456 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:30:20.457 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:30:20.458 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:30:20.459 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:20.459 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:20.459 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:30:20.459 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:20.460 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:30:20.460 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:20.460 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:20.460 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:20.461 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:20.463 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:30:20.463 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:20.463 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:30:20.463 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:20.464 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:20.464 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:20.464 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:20.464 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:30:20.465 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:20.465 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:20.466 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:20.466 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:20.466 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:20.467 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:20.467 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:20.468 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:20.468 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:20.469 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:20.469 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:20.469 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:30:20.470 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:20.476 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:20.476 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762618.860801_61.1", 2022-09-09T22:30:20.476 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:20.477 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:20.477 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:20.477 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:18.860801Z", 2022-09-09T22:30:20.478 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:20.478 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:20.479 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:20.479 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:20.479 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:30:20.480 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:20.480 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:20.480 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:20.481 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:20.481 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:20.481 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:20.481 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:20.482 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:20.482 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:20.482 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:20.483 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:30:20.483 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:20.484 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:30:20.484 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:30:20.484 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:30:20.485 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:20.485 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:30:20Z" 2022-09-09T22:30:20.485 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:30:20.487 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762618.860801_61.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:18.860801Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:30:20Z" } 2022-09-09T22:30:20.487 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762618.860801_61.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:18.860801Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2022-09-09T22:30:20.488 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762618.860801_61.1', 3: ''} 2022-09-09T22:30:20.488 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762618.860801_61.1', 3: ''} 2022-09-09T22:30:20.489 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-09T22:30:20.489 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-09T22:30:20.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:20.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:20.491 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:20.541 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:20.542 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:20.596 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:30:20.595+0000 7f554f630dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:20.597 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:30:20.599+0000 7f554f630dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:20.622 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:30:20.623 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T22:30:20.623 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T22:30:20.624 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T22:30:20.624 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T22:30:20.625 INFO:teuthology.orchestra.run.smithi005.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:30:20.626 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:20.628 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:20.628 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T22:30:20.629 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:20.629 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T22:30:20.630 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:20.630 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:20.631 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:20.631 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:20.632 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 1, 2022-09-09T22:30:20.632 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:20.632 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:30:20.633 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:20.633 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:20.633 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:20.634 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:20.634 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T22:30:20.635 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:20.635 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:20.636 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:30:20.636 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:20.636 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:30:20.637 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:20.642 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:30:20.643 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:20.643 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:20.643 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:20.644 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:20.644 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T22:30:20.645 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:20.645 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:20.645 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762618.860801_61.1", 2022-09-09T22:30:20.646 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:20.646 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:30:20.646 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:20.647 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:18.860801Z", 2022-09-09T22:30:20.647 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:20.647 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:20.647 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:20.648 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:20.648 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T22:30:20.648 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:20.648 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:20.649 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:30:20.649 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:20.649 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:30:20.650 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:20.652 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:30:20.652 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:20.653 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:20.653 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:20.654 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:30:20.655 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:20.655 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T22:30:20.656 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 1, 2022-09-09T22:30:20.656 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 1 2022-09-09T22:30:20.657 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:20.657 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T22:30:20Z" 2022-09-09T22:30:20.658 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:30:20.660 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762618.860801_61.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:18.860801Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:30:20Z" } 2022-09-09T22:30:20.662 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762618.860801_61.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:18.860801Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2022-09-09T22:30:20.663 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762618.860801_61.1', 3: ''} 2022-09-09T22:30:20.664 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762618.860801_61.1', 3: ''} 2022-09-09T22:30:20.664 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-09T22:30:20.665 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-09T22:30:20.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:20.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:20.668 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:20.715 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:20.717 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:20.790 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:20.790+0000 7f07c349adc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:20.792 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:20.790+0000 7f07c349adc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:20.818 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:30:20.819 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:30:20.819 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:30:20.819 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:30:20.820 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:30:20.820 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:30:20.820 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:20.820 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:20.821 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:30:20.821 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:20.821 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:30:20.821 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:20.822 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:20.822 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:20.822 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:20.822 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:30:20.823 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:20.823 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:30:20.823 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:20.824 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:20.824 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:20.824 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:20.824 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:30:20.825 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:20.825 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:20.825 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:20.825 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:20.826 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:20.826 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:20.826 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:20.826 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:20.827 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:20.827 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:20.827 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:20.827 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:30:20.828 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:20.828 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:20.828 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762618.860801_61.1", 2022-09-09T22:30:20.828 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:20.829 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:20.829 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:20.829 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:18.860801Z", 2022-09-09T22:30:20.829 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:20.830 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:20.830 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:20.830 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:20.831 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:30:20.831 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:20.831 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:20.832 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:20.833 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:20.833 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:20.833 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:20.834 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:20.834 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:20.834 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:20.834 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:20.835 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:30:20.835 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:20.835 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:30:20.836 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:30:20.836 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:30:20.836 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:20.837 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:30:20Z" 2022-09-09T22:30:20.837 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:30:20.838 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762618.860801_61.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:18.860801Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:30:20Z" } 2022-09-09T22:30:20.839 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762618.860801_61.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:18.860801Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2022-09-09T22:30:20.840 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762618.860801_61.1', 3: ''} 2022-09-09T22:30:20.841 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762618.860801_61.1', 3: ''} 2022-09-09T22:30:20.841 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-09T22:30:20.842 DEBUG:boto:path=/jkcgbj-4/ 2022-09-09T22:30:20.842 DEBUG:boto:auth_path=/jkcgbj-4/ 2022-09-09T22:30:20.843 DEBUG:boto:Method: HEAD 2022-09-09T22:30:20.844 DEBUG:boto:Path: /jkcgbj-4/ 2022-09-09T22:30:20.844 DEBUG:boto:Data: 2022-09-09T22:30:20.845 DEBUG:boto:Headers: {} 2022-09-09T22:30:20.845 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:20.846 DEBUG:boto:Port: 8000 2022-09-09T22:30:20.846 DEBUG:boto:Params: {} 2022-09-09T22:30:20.850 DEBUG:boto:Token: None 2022-09-09T22:30:20.850 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:20 GMT /jkcgbj-4/ 2022-09-09T22:30:20.851 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:NuNL5T1qMO2EOOiJPd32e58S07U= 2022-09-09T22:30:20.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:NuNL5T1qMO2EOOiJPd32e58S07U=', 'Content-Length': '0'} 2022-09-09T22:30:20.857 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d082f0cf472965e-00631bbe7c-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:20.858 DEBUG:boto:path=/jkcgbj-5/ 2022-09-09T22:30:20.862 DEBUG:boto:auth_path=/jkcgbj-5/ 2022-09-09T22:30:20.866 DEBUG:boto:Method: HEAD 2022-09-09T22:30:20.867 DEBUG:boto:Path: /jkcgbj-5/ 2022-09-09T22:30:20.868 DEBUG:boto:Data: 2022-09-09T22:30:20.868 DEBUG:boto:Headers: {} 2022-09-09T22:30:20.869 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:20.869 DEBUG:boto:Port: 8000 2022-09-09T22:30:20.870 DEBUG:boto:Params: {} 2022-09-09T22:30:20.870 DEBUG:boto:Token: None 2022-09-09T22:30:20.871 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:20 GMT /jkcgbj-5/ 2022-09-09T22:30:20.871 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bmHJbAeUy9cnfWthIRPp6BcEDrs= 2022-09-09T22:30:20.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bmHJbAeUy9cnfWthIRPp6BcEDrs=', 'Content-Length': '0'} 2022-09-09T22:30:20.877 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be68c49226e29a59-00631bbe7c-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:20.878 DEBUG:boto:path=/jkcgbj-6/ 2022-09-09T22:30:20.878 DEBUG:boto:auth_path=/jkcgbj-6/ 2022-09-09T22:30:20.879 DEBUG:boto:Method: HEAD 2022-09-09T22:30:20.879 DEBUG:boto:Path: /jkcgbj-6/ 2022-09-09T22:30:20.880 DEBUG:boto:Data: 2022-09-09T22:30:20.881 DEBUG:boto:Headers: {} 2022-09-09T22:30:20.881 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:20.882 DEBUG:boto:Port: 8000 2022-09-09T22:30:20.884 DEBUG:boto:Params: {} 2022-09-09T22:30:20.885 DEBUG:boto:Token: None 2022-09-09T22:30:20.886 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:20 GMT /jkcgbj-6/ 2022-09-09T22:30:20.887 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0TYC+bNJRocPO3AfwdL23tbqoBw= 2022-09-09T22:30:20.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0TYC+bNJRocPO3AfwdL23tbqoBw=', 'Content-Length': '0'} 2022-09-09T22:30:20.894 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009af5287e3e98da8e-00631bbe7c-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:20.895 DEBUG:boto:path=/jkcgbj-4/ 2022-09-09T22:30:20.896 DEBUG:boto:auth_path=/jkcgbj-4/ 2022-09-09T22:30:20.896 DEBUG:boto:Method: HEAD 2022-09-09T22:30:20.897 DEBUG:boto:Path: /jkcgbj-4/ 2022-09-09T22:30:20.897 DEBUG:boto:Data: 2022-09-09T22:30:20.898 DEBUG:boto:Headers: {} 2022-09-09T22:30:20.899 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:20.900 DEBUG:boto:Port: 8000 2022-09-09T22:30:20.900 DEBUG:boto:Params: {} 2022-09-09T22:30:20.902 DEBUG:boto:Token: None 2022-09-09T22:30:20.903 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:20 GMT /jkcgbj-4/ 2022-09-09T22:30:20.903 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:NuNL5T1qMO2EOOiJPd32e58S07U= 2022-09-09T22:30:20.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:NuNL5T1qMO2EOOiJPd32e58S07U=', 'Content-Length': '0'} 2022-09-09T22:30:20.909 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060f4a2e93615fe0b-00631bbe7c-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:20.910 DEBUG:boto:path=/jkcgbj-5/ 2022-09-09T22:30:20.911 DEBUG:boto:auth_path=/jkcgbj-5/ 2022-09-09T22:30:20.912 DEBUG:boto:Method: HEAD 2022-09-09T22:30:20.912 DEBUG:boto:Path: /jkcgbj-5/ 2022-09-09T22:30:20.913 DEBUG:boto:Data: 2022-09-09T22:30:20.914 DEBUG:boto:Headers: {} 2022-09-09T22:30:20.914 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:20.915 DEBUG:boto:Port: 8000 2022-09-09T22:30:20.915 DEBUG:boto:Params: {} 2022-09-09T22:30:20.916 DEBUG:boto:Token: None 2022-09-09T22:30:20.916 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:20 GMT /jkcgbj-5/ 2022-09-09T22:30:20.917 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bmHJbAeUy9cnfWthIRPp6BcEDrs= 2022-09-09T22:30:20.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bmHJbAeUy9cnfWthIRPp6BcEDrs=', 'Content-Length': '0'} 2022-09-09T22:30:20.922 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e4a4af865d7881c-00631bbe7c-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:20.922 DEBUG:boto:path=/jkcgbj-6/ 2022-09-09T22:30:20.923 DEBUG:boto:auth_path=/jkcgbj-6/ 2022-09-09T22:30:20.924 DEBUG:boto:Method: HEAD 2022-09-09T22:30:20.925 DEBUG:boto:Path: /jkcgbj-6/ 2022-09-09T22:30:20.926 DEBUG:boto:Data: 2022-09-09T22:30:20.927 DEBUG:boto:Headers: {} 2022-09-09T22:30:20.927 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:20.928 DEBUG:boto:Port: 8000 2022-09-09T22:30:20.929 DEBUG:boto:Params: {} 2022-09-09T22:30:20.930 DEBUG:boto:Token: None 2022-09-09T22:30:20.930 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:20 GMT /jkcgbj-6/ 2022-09-09T22:30:20.931 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0TYC+bNJRocPO3AfwdL23tbqoBw= 2022-09-09T22:30:20.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0TYC+bNJRocPO3AfwdL23tbqoBw=', 'Content-Length': '0'} 2022-09-09T22:30:20.936 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093b9686d940a9b38-00631bbe7c-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:20.937 DEBUG:boto:path=/jkcgbj-4/ 2022-09-09T22:30:20.938 DEBUG:boto:auth_path=/jkcgbj-4/ 2022-09-09T22:30:20.939 DEBUG:boto:Method: HEAD 2022-09-09T22:30:20.939 DEBUG:boto:Path: /jkcgbj-4/ 2022-09-09T22:30:20.940 DEBUG:boto:Data: 2022-09-09T22:30:20.941 DEBUG:boto:Headers: {} 2022-09-09T22:30:20.942 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:20.943 DEBUG:boto:Port: 8001 2022-09-09T22:30:20.943 DEBUG:boto:Params: {} 2022-09-09T22:30:20.944 DEBUG:boto:Token: None 2022-09-09T22:30:20.945 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:20 GMT /jkcgbj-4/ 2022-09-09T22:30:20.946 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:NuNL5T1qMO2EOOiJPd32e58S07U= 2022-09-09T22:30:20.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:NuNL5T1qMO2EOOiJPd32e58S07U=', 'Content-Length': '0'} 2022-09-09T22:30:20.951 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000049d56e4f0f25a22a-00631bbe7c-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:20.952 DEBUG:boto:path=/jkcgbj-5/ 2022-09-09T22:30:20.953 DEBUG:boto:auth_path=/jkcgbj-5/ 2022-09-09T22:30:20.953 DEBUG:boto:Method: HEAD 2022-09-09T22:30:20.954 DEBUG:boto:Path: /jkcgbj-5/ 2022-09-09T22:30:20.954 DEBUG:boto:Data: 2022-09-09T22:30:20.955 DEBUG:boto:Headers: {} 2022-09-09T22:30:20.955 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:20.955 DEBUG:boto:Port: 8001 2022-09-09T22:30:20.956 DEBUG:boto:Params: {} 2022-09-09T22:30:20.957 DEBUG:boto:Token: None 2022-09-09T22:30:20.957 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:20 GMT /jkcgbj-5/ 2022-09-09T22:30:20.958 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bmHJbAeUy9cnfWthIRPp6BcEDrs= 2022-09-09T22:30:20.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bmHJbAeUy9cnfWthIRPp6BcEDrs=', 'Content-Length': '0'} 2022-09-09T22:30:20.963 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ca836861c8757e5-00631bbe7c-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:20.964 DEBUG:boto:path=/jkcgbj-6/ 2022-09-09T22:30:20.964 DEBUG:boto:auth_path=/jkcgbj-6/ 2022-09-09T22:30:20.965 DEBUG:boto:Method: HEAD 2022-09-09T22:30:20.965 DEBUG:boto:Path: /jkcgbj-6/ 2022-09-09T22:30:20.966 DEBUG:boto:Data: 2022-09-09T22:30:20.966 DEBUG:boto:Headers: {} 2022-09-09T22:30:20.967 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:20.967 DEBUG:boto:Port: 8001 2022-09-09T22:30:20.968 DEBUG:boto:Params: {} 2022-09-09T22:30:20.968 DEBUG:boto:Token: None 2022-09-09T22:30:20.969 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:20 GMT /jkcgbj-6/ 2022-09-09T22:30:20.970 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0TYC+bNJRocPO3AfwdL23tbqoBw= 2022-09-09T22:30:20.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0TYC+bNJRocPO3AfwdL23tbqoBw=', 'Content-Length': '0'} 2022-09-09T22:30:20.974 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e5af687d2754949-00631bbe7c-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:20.976 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2022-09-09T22:30:20.977 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jkcgbj-7 2022-09-09T22:30:20.978 DEBUG:boto:path=/jkcgbj-7/ 2022-09-09T22:30:20.978 DEBUG:boto:auth_path=/jkcgbj-7/ 2022-09-09T22:30:20.979 DEBUG:boto:Method: PUT 2022-09-09T22:30:20.979 DEBUG:boto:Path: /jkcgbj-7/ 2022-09-09T22:30:20.980 DEBUG:boto:Data: 2022-09-09T22:30:20.980 DEBUG:boto:Headers: {} 2022-09-09T22:30:20.981 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:20.981 DEBUG:boto:Port: 8000 2022-09-09T22:30:20.982 DEBUG:boto:Params: {} 2022-09-09T22:30:20.982 DEBUG:boto:Token: None 2022-09-09T22:30:20.983 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:30:20 GMT /jkcgbj-7/ 2022-09-09T22:30:20.983 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Pms/21+d7oJ3domMec68ZLzfu+E= 2022-09-09T22:30:20.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Pms/21+d7oJ3domMec68ZLzfu+E=', 'Content-Length': '0'} 2022-09-09T22:30:21.008 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f3fb9793454e1789-00631bbe7c-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:21.009 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jkcgbj-8 2022-09-09T22:30:21.010 DEBUG:boto:path=/jkcgbj-8/ 2022-09-09T22:30:21.010 DEBUG:boto:auth_path=/jkcgbj-8/ 2022-09-09T22:30:21.011 DEBUG:boto:Method: PUT 2022-09-09T22:30:21.011 DEBUG:boto:Path: /jkcgbj-8/ 2022-09-09T22:30:21.012 DEBUG:boto:Data: 2022-09-09T22:30:21.012 DEBUG:boto:Headers: {} 2022-09-09T22:30:21.013 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:21.015 DEBUG:boto:Port: 8000 2022-09-09T22:30:21.016 DEBUG:boto:Params: {} 2022-09-09T22:30:21.017 DEBUG:boto:Token: None 2022-09-09T22:30:21.018 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:30:21 GMT /jkcgbj-8/ 2022-09-09T22:30:21.019 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:lWIbbyknWCRDsnOxa/J76l+VDaI= 2022-09-09T22:30:21.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:lWIbbyknWCRDsnOxa/J76l+VDaI=', 'Content-Length': '0'} 2022-09-09T22:30:21.064 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d28f6611beb21fac-00631bbe7d-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:21.065 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jkcgbj-9 2022-09-09T22:30:21.066 DEBUG:boto:path=/jkcgbj-9/ 2022-09-09T22:30:21.067 DEBUG:boto:auth_path=/jkcgbj-9/ 2022-09-09T22:30:21.068 DEBUG:boto:Method: PUT 2022-09-09T22:30:21.069 DEBUG:boto:Path: /jkcgbj-9/ 2022-09-09T22:30:21.070 DEBUG:boto:Data: 2022-09-09T22:30:21.071 DEBUG:boto:Headers: {} 2022-09-09T22:30:21.072 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:21.073 DEBUG:boto:Port: 8001 2022-09-09T22:30:21.074 DEBUG:boto:Params: {} 2022-09-09T22:30:21.077 DEBUG:boto:Token: None 2022-09-09T22:30:21.079 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:30:21 GMT /jkcgbj-9/ 2022-09-09T22:30:21.080 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:MoJZf+otgQKWhNZo8glyS+cRRpo= 2022-09-09T22:30:21.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:MoJZf+otgQKWhNZo8glyS+cRRpo=', 'Content-Length': '0'} 2022-09-09T22:30:21.124 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000042ee882ee8a47430-00631bbe7d-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:21.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:21.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:21.129 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:21.177 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:21.178 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:21.236 INFO:teuthology.orchestra.run.smithi005.stderr:No --period given, using current period=503b1d61-4513-48c8-a308-17b76219610c 2022-09-09T22:30:21.254 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T22:30:21.255 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:21.255 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:21.256 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:28:06.146147Z" 2022-09-09T22:30:21.256 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:21.256 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:21.256 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:30:21.257 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:30:21.257 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:21.257 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:21.258 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762621.110791_73.1", 2022-09-09T22:30:21.258 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:30:21.110791Z" 2022-09-09T22:30:21.258 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:21.259 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:21.259 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:30:21.259 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:30:21.259 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:21.260 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T22:30:21.262 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762621.110791_73.1', 3: ''} 2022-09-09T22:30:21.262 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-09T22:30:21.263 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:21.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:21.264 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:21.313 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:21.314 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:21.382 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:21.382+0000 7fd7af241dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:21.384 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:21.382+0000 7fd7af241dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:21.415 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:30:21.415 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:30:21.416 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:30:21.416 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:30:21.417 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:30:21.417 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:30:21.417 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:21.417 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:21.418 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:30:21.418 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:21.418 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:30:21.418 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:21.419 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:21.419 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:21.419 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:21.420 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:30:21.420 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:21.420 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:30:21.420 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:21.421 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:21.421 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:21.421 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:21.421 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:30:21.422 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:21.422 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:21.422 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:21.422 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:21.423 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:21.423 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:21.423 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:21.423 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:21.424 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:21.424 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:21.424 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:21.424 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:30:21.425 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:21.425 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:21.425 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762621.105043_71.1", 2022-09-09T22:30:21.425 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:21.426 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:21.426 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:21.426 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:21.105043Z", 2022-09-09T22:30:21.427 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:21.427 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:21.427 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:21.427 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:21.428 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:30:21.428 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:21.428 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:21.429 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:21.429 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:21.429 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:21.429 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:21.430 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:21.430 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:21.430 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:21.430 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:21.431 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:30:21.431 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:21.431 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:30:21.432 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:30:21.432 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:30:21.433 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:21.433 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:30:21Z" 2022-09-09T22:30:21.434 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:30:21.436 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762621.105043_71.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:21.105043Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:30:21Z" } 2022-09-09T22:30:21.436 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762621.105043_71.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:21.105043Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2022-09-09T22:30:21.437 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762621.110791_73.1', 3: ''} 2022-09-09T22:30:21.437 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762621.105043_71.1', 3: ''} 2022-09-09T22:30:21.438 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662762621.110791_73.1 target=1_1662762621.105043_71.1 2022-09-09T22:30:26.439 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:26.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:26.440 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:26.490 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:26.491 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:26.546 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:26.546+0000 7fe986494dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:26.548 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:26.546+0000 7fe986494dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:26.573 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:30:26.573 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:30:26.573 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:30:26.574 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:30:26.574 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:30:26.575 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:30:26.575 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:26.575 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:26.576 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:30:26.576 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:26.576 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:30:26.577 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:26.577 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:26.577 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:26.577 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:26.578 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:30:26.578 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:26.578 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:30:26.579 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:26.579 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:26.579 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:26.579 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:26.580 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:30:26.580 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:26.580 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:26.581 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:26.581 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:26.581 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:26.581 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:26.582 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:26.582 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:26.582 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:26.583 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:26.583 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:26.584 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:30:26.584 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:26.584 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:26.585 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762621.105043_71.1", 2022-09-09T22:30:26.585 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:26.585 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:26.585 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:26.586 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:21.105043Z", 2022-09-09T22:30:26.586 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:26.586 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:26.587 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:26.587 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:26.587 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:30:26.587 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:26.588 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:26.588 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:26.588 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:26.588 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:26.589 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:26.589 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:26.589 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:26.590 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:26.590 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:26.590 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:30:26.590 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:26.591 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:30:26.591 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:30:26.591 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:30:26.592 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:26.592 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:30:26Z" 2022-09-09T22:30:26.592 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:30:26.594 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762621.105043_71.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:21.105043Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:30:26Z" } 2022-09-09T22:30:26.594 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762621.105043_71.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:21.105043Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2022-09-09T22:30:26.595 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762621.110791_73.1', 3: ''} 2022-09-09T22:30:26.595 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762621.105043_71.1', 3: ''} 2022-09-09T22:30:26.596 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662762621.110791_73.1 target=1_1662762621.105043_71.1 2022-09-09T22:30:31.597 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:31.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:31.600 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:31.648 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:31.648 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:31.707 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:31.706+0000 7f90daae4dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:31.709 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:31.710+0000 7f90daae4dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:31.736 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:30:31.737 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:30:31.737 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:30:31.738 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:30:31.738 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:30:31.739 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:30:31.739 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:31.740 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:31.740 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:30:31.741 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:31.741 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:30:31.741 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:31.741 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:31.742 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:31.742 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:31.742 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:30:31.743 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:31.743 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:30:31.743 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:31.743 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:31.744 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:31.744 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:31.744 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:30:31.744 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:31.745 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:31.745 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:31.745 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:31.746 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:31.746 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:31.747 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:31.747 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:31.747 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:31.748 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:31.748 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:31.749 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:30:31.749 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:31.749 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:31.750 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762621.105043_71.1", 2022-09-09T22:30:31.750 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:31.751 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:31.751 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:31.751 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:21.105043Z", 2022-09-09T22:30:31.754 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:31.754 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:31.755 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:31.755 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:31.756 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:30:31.756 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:31.756 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:31.757 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:31.757 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:31.758 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:31.758 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:31.759 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:31.759 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:31.760 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:31.760 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:31.761 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:30:31.761 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:31.761 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:30:31.762 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:30:31.762 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:30:31.762 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:31.763 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:30:31Z" 2022-09-09T22:30:31.764 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:30:31.766 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762621.105043_71.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:21.105043Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:30:31Z" } 2022-09-09T22:30:31.767 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762621.105043_71.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:21.105043Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2022-09-09T22:30:31.767 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762621.110791_73.1', 3: ''} 2022-09-09T22:30:31.768 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762621.105043_71.1', 3: ''} 2022-09-09T22:30:31.769 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662762621.110791_73.1 target=1_1662762621.105043_71.1 2022-09-09T22:30:36.772 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:36.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:36.774 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:36.821 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:36.822 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:36.884 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:36.882+0000 7fe8a8f00dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:36.886 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:36.886+0000 7fe8a8f00dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:36.913 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:30:36.913 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:30:36.914 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:30:36.914 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:30:36.915 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:30:36.916 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:30:36.917 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:36.917 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:36.917 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:30:36.918 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:36.921 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:30:36.921 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:36.922 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:36.922 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:36.923 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:36.923 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:30:36.924 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:36.924 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:30:36.925 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:36.925 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:36.926 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:36.926 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:36.928 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:30:36.928 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:36.928 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:36.928 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:36.929 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:36.929 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:36.929 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:36.930 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:36.930 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:36.930 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:36.930 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:36.931 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:36.931 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:30:36.931 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:36.931 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:36.932 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762621.110791_73.1", 2022-09-09T22:30:36.932 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:36.932 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:36.933 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:36.933 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:21.110791Z", 2022-09-09T22:30:36.933 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:36.933 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:36.934 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:36.934 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:36.934 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:30:36.934 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:36.935 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:36.935 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:36.935 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:36.935 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:36.936 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:36.936 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:36.936 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:36.936 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:36.937 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:36.937 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:30:36.937 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:36.937 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:30:36.938 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:30:36.938 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:30:36.938 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:36.938 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:30:36Z" 2022-09-09T22:30:36.939 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:30:36.940 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762621.110791_73.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:21.110791Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:30:36Z" } 2022-09-09T22:30:36.941 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762621.110791_73.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:21.110791Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2022-09-09T22:30:36.941 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762621.110791_73.1', 3: ''} 2022-09-09T22:30:36.942 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762621.110791_73.1', 3: ''} 2022-09-09T22:30:36.942 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-09T22:30:36.943 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-09T22:30:36.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:36.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:36.945 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:36.995 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:36.996 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:37.044 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:30:37.043+0000 7f85f5964dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:37.046 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:30:37.047+0000 7f85f5964dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:37.069 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:30:37.070 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T22:30:37.070 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T22:30:37.070 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T22:30:37.071 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T22:30:37.071 INFO:teuthology.orchestra.run.smithi005.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:30:37.071 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:37.071 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:37.072 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T22:30:37.072 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:37.072 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T22:30:37.072 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:37.073 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:37.073 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:37.073 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:37.073 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 1, 2022-09-09T22:30:37.074 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:37.074 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:30:37.074 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:37.075 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:37.075 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:37.075 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:37.075 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T22:30:37.075 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:37.076 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:37.076 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:30:37.076 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:37.076 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:30:37.078 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:37.079 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:30:37.079 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:37.080 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:37.080 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:37.081 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:37.081 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T22:30:37.082 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:37.083 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:37.084 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762621.110791_73.1", 2022-09-09T22:30:37.084 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:37.085 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:30:37.085 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:37.086 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:21.110791Z", 2022-09-09T22:30:37.086 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:37.087 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:37.087 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:37.090 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:37.091 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T22:30:37.095 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:37.095 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:37.096 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:30:37.096 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:37.097 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:30:37.098 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:37.098 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:30:37.099 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:37.099 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:37.100 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:37.100 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:30:37.100 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:37.101 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T22:30:37.101 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 1, 2022-09-09T22:30:37.102 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 1 2022-09-09T22:30:37.102 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:37.103 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T22:30:37Z" 2022-09-09T22:30:37.103 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:30:37.105 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762621.110791_73.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:21.110791Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:30:37Z" } 2022-09-09T22:30:37.106 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762621.110791_73.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:21.110791Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2022-09-09T22:30:37.108 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762621.110791_73.1', 3: ''} 2022-09-09T22:30:37.108 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762621.110791_73.1', 3: ''} 2022-09-09T22:30:37.109 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-09T22:30:37.110 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-09T22:30:37.111 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:37.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:37.113 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:37.162 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:37.163 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:37.215 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:37.214+0000 7fea4b957dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:37.217 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:37.218+0000 7fea4b957dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:37.244 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:30:37.245 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:30:37.245 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:30:37.245 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:30:37.249 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:30:37.250 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:30:37.250 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:37.251 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:37.251 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:30:37.251 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:37.251 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:30:37.252 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:37.252 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:37.252 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:37.253 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:37.254 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:30:37.254 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:37.254 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:30:37.255 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:37.255 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:37.255 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:37.255 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:37.256 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:30:37.256 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:37.256 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:37.257 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:37.257 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:37.257 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:37.257 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:37.258 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:37.258 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:37.258 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:37.259 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:37.259 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:37.259 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:30:37.259 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:37.260 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:37.260 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762621.110791_73.1", 2022-09-09T22:30:37.260 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:37.261 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:37.261 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:37.261 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:21.110791Z", 2022-09-09T22:30:37.262 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:37.262 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:37.262 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:37.263 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:37.263 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:30:37.264 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:37.264 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:37.264 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:37.265 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:37.265 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:37.265 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:37.266 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:37.266 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:37.266 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:37.266 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:37.267 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:30:37.267 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:37.267 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:30:37.267 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:30:37.268 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:30:37.268 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:37.268 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:30:37Z" 2022-09-09T22:30:37.268 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:30:37.269 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762621.110791_73.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:21.110791Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:30:37Z" } 2022-09-09T22:30:37.270 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762621.110791_73.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:21.110791Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2022-09-09T22:30:37.270 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762621.110791_73.1', 3: ''} 2022-09-09T22:30:37.271 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762621.110791_73.1', 3: ''} 2022-09-09T22:30:37.271 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-09T22:30:37.272 DEBUG:boto:path=/jkcgbj-7/ 2022-09-09T22:30:37.272 DEBUG:boto:auth_path=/jkcgbj-7/ 2022-09-09T22:30:37.273 DEBUG:boto:Method: HEAD 2022-09-09T22:30:37.274 DEBUG:boto:Path: /jkcgbj-7/ 2022-09-09T22:30:37.275 DEBUG:boto:Data: 2022-09-09T22:30:37.275 DEBUG:boto:Headers: {} 2022-09-09T22:30:37.277 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:37.277 DEBUG:boto:Port: 8000 2022-09-09T22:30:37.278 DEBUG:boto:Params: {} 2022-09-09T22:30:37.279 DEBUG:boto:Token: None 2022-09-09T22:30:37.279 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:37 GMT /jkcgbj-7/ 2022-09-09T22:30:37.280 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:36lzwI6FqQTYCDeO4tKf8++buSo= 2022-09-09T22:30:37.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:36lzwI6FqQTYCDeO4tKf8++buSo=', 'Content-Length': '0'} 2022-09-09T22:30:37.286 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002642478f79c7c885-00631bbe8d-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:37.288 DEBUG:boto:path=/jkcgbj-8/ 2022-09-09T22:30:37.288 DEBUG:boto:auth_path=/jkcgbj-8/ 2022-09-09T22:30:37.289 DEBUG:boto:Method: HEAD 2022-09-09T22:30:37.292 DEBUG:boto:Path: /jkcgbj-8/ 2022-09-09T22:30:37.293 DEBUG:boto:Data: 2022-09-09T22:30:37.294 DEBUG:boto:Headers: {} 2022-09-09T22:30:37.294 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:37.295 DEBUG:boto:Port: 8000 2022-09-09T22:30:37.295 DEBUG:boto:Params: {} 2022-09-09T22:30:37.296 DEBUG:boto:Token: None 2022-09-09T22:30:37.296 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:37 GMT /jkcgbj-8/ 2022-09-09T22:30:37.297 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:XDSRCU8XCLr4ybkaLhXHuvlLvL8= 2022-09-09T22:30:37.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:XDSRCU8XCLr4ybkaLhXHuvlLvL8=', 'Content-Length': '0'} 2022-09-09T22:30:37.303 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009660a0e2ea451068-00631bbe8d-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:37.304 DEBUG:boto:path=/jkcgbj-9/ 2022-09-09T22:30:37.305 DEBUG:boto:auth_path=/jkcgbj-9/ 2022-09-09T22:30:37.306 DEBUG:boto:Method: HEAD 2022-09-09T22:30:37.308 DEBUG:boto:Path: /jkcgbj-9/ 2022-09-09T22:30:37.309 DEBUG:boto:Data: 2022-09-09T22:30:37.310 DEBUG:boto:Headers: {} 2022-09-09T22:30:37.313 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:37.313 DEBUG:boto:Port: 8000 2022-09-09T22:30:37.314 DEBUG:boto:Params: {} 2022-09-09T22:30:37.315 DEBUG:boto:Token: None 2022-09-09T22:30:37.315 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:37 GMT /jkcgbj-9/ 2022-09-09T22:30:37.316 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:3je3/n+SHlfq6/ahjiVL+P76L0E= 2022-09-09T22:30:37.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:3je3/n+SHlfq6/ahjiVL+P76L0E=', 'Content-Length': '0'} 2022-09-09T22:30:37.322 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d7b1c882d3026c9a-00631bbe8d-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:37.324 DEBUG:boto:path=/jkcgbj-7/ 2022-09-09T22:30:37.325 DEBUG:boto:auth_path=/jkcgbj-7/ 2022-09-09T22:30:37.326 DEBUG:boto:Method: HEAD 2022-09-09T22:30:37.327 DEBUG:boto:Path: /jkcgbj-7/ 2022-09-09T22:30:37.327 DEBUG:boto:Data: 2022-09-09T22:30:37.328 DEBUG:boto:Headers: {} 2022-09-09T22:30:37.329 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:37.330 DEBUG:boto:Port: 8000 2022-09-09T22:30:37.331 DEBUG:boto:Params: {} 2022-09-09T22:30:37.332 DEBUG:boto:Token: None 2022-09-09T22:30:37.334 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:37 GMT /jkcgbj-7/ 2022-09-09T22:30:37.335 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:36lzwI6FqQTYCDeO4tKf8++buSo= 2022-09-09T22:30:37.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:36lzwI6FqQTYCDeO4tKf8++buSo=', 'Content-Length': '0'} 2022-09-09T22:30:37.342 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c7a28e97e401153-00631bbe8d-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:37.342 DEBUG:boto:path=/jkcgbj-8/ 2022-09-09T22:30:37.343 DEBUG:boto:auth_path=/jkcgbj-8/ 2022-09-09T22:30:37.343 DEBUG:boto:Method: HEAD 2022-09-09T22:30:37.344 DEBUG:boto:Path: /jkcgbj-8/ 2022-09-09T22:30:37.345 DEBUG:boto:Data: 2022-09-09T22:30:37.345 DEBUG:boto:Headers: {} 2022-09-09T22:30:37.346 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:37.347 DEBUG:boto:Port: 8000 2022-09-09T22:30:37.348 DEBUG:boto:Params: {} 2022-09-09T22:30:37.349 DEBUG:boto:Token: None 2022-09-09T22:30:37.350 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:37 GMT /jkcgbj-8/ 2022-09-09T22:30:37.351 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:XDSRCU8XCLr4ybkaLhXHuvlLvL8= 2022-09-09T22:30:37.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:XDSRCU8XCLr4ybkaLhXHuvlLvL8=', 'Content-Length': '0'} 2022-09-09T22:30:37.357 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1f7e043096c843a-00631bbe8d-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:37.358 DEBUG:boto:path=/jkcgbj-9/ 2022-09-09T22:30:37.359 DEBUG:boto:auth_path=/jkcgbj-9/ 2022-09-09T22:30:37.360 DEBUG:boto:Method: HEAD 2022-09-09T22:30:37.361 DEBUG:boto:Path: /jkcgbj-9/ 2022-09-09T22:30:37.362 DEBUG:boto:Data: 2022-09-09T22:30:37.363 DEBUG:boto:Headers: {} 2022-09-09T22:30:37.364 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:37.365 DEBUG:boto:Port: 8000 2022-09-09T22:30:37.366 DEBUG:boto:Params: {} 2022-09-09T22:30:37.367 DEBUG:boto:Token: None 2022-09-09T22:30:37.368 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:37 GMT /jkcgbj-9/ 2022-09-09T22:30:37.369 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:3je3/n+SHlfq6/ahjiVL+P76L0E= 2022-09-09T22:30:37.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:3je3/n+SHlfq6/ahjiVL+P76L0E=', 'Content-Length': '0'} 2022-09-09T22:30:37.376 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000081904d34cb3a7f1d-00631bbe8d-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:37.377 DEBUG:boto:path=/jkcgbj-7/ 2022-09-09T22:30:37.378 DEBUG:boto:auth_path=/jkcgbj-7/ 2022-09-09T22:30:37.379 DEBUG:boto:Method: HEAD 2022-09-09T22:30:37.382 DEBUG:boto:Path: /jkcgbj-7/ 2022-09-09T22:30:37.382 DEBUG:boto:Data: 2022-09-09T22:30:37.383 DEBUG:boto:Headers: {} 2022-09-09T22:30:37.384 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:37.385 DEBUG:boto:Port: 8001 2022-09-09T22:30:37.386 DEBUG:boto:Params: {} 2022-09-09T22:30:37.387 DEBUG:boto:Token: None 2022-09-09T22:30:37.388 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:37 GMT /jkcgbj-7/ 2022-09-09T22:30:37.388 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:36lzwI6FqQTYCDeO4tKf8++buSo= 2022-09-09T22:30:37.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:36lzwI6FqQTYCDeO4tKf8++buSo=', 'Content-Length': '0'} 2022-09-09T22:30:37.395 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006273f28e4aa47446-00631bbe8d-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:37.396 DEBUG:boto:path=/jkcgbj-8/ 2022-09-09T22:30:37.396 DEBUG:boto:auth_path=/jkcgbj-8/ 2022-09-09T22:30:37.397 DEBUG:boto:Method: HEAD 2022-09-09T22:30:37.397 DEBUG:boto:Path: /jkcgbj-8/ 2022-09-09T22:30:37.398 DEBUG:boto:Data: 2022-09-09T22:30:37.399 DEBUG:boto:Headers: {} 2022-09-09T22:30:37.400 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:37.401 DEBUG:boto:Port: 8001 2022-09-09T22:30:37.402 DEBUG:boto:Params: {} 2022-09-09T22:30:37.403 DEBUG:boto:Token: None 2022-09-09T22:30:37.404 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:37 GMT /jkcgbj-8/ 2022-09-09T22:30:37.405 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:XDSRCU8XCLr4ybkaLhXHuvlLvL8= 2022-09-09T22:30:37.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:XDSRCU8XCLr4ybkaLhXHuvlLvL8=', 'Content-Length': '0'} 2022-09-09T22:30:37.410 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fdcda7fb9a6c5951-00631bbe8d-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:37.411 DEBUG:boto:path=/jkcgbj-9/ 2022-09-09T22:30:37.412 DEBUG:boto:auth_path=/jkcgbj-9/ 2022-09-09T22:30:37.412 DEBUG:boto:Method: HEAD 2022-09-09T22:30:37.413 DEBUG:boto:Path: /jkcgbj-9/ 2022-09-09T22:30:37.413 DEBUG:boto:Data: 2022-09-09T22:30:37.414 DEBUG:boto:Headers: {} 2022-09-09T22:30:37.414 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:37.415 DEBUG:boto:Port: 8001 2022-09-09T22:30:37.415 DEBUG:boto:Params: {} 2022-09-09T22:30:37.416 DEBUG:boto:Token: None 2022-09-09T22:30:37.417 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:37 GMT /jkcgbj-9/ 2022-09-09T22:30:37.417 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:3je3/n+SHlfq6/ahjiVL+P76L0E= 2022-09-09T22:30:37.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:3je3/n+SHlfq6/ahjiVL+P76L0E=', 'Content-Length': '0'} 2022-09-09T22:30:37.423 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fb88feaf4ff14160-00631bbe8d-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:37.425 DEBUG:boto:path=/jkcgbj-7/ 2022-09-09T22:30:37.425 DEBUG:boto:auth_path=/jkcgbj-7/ 2022-09-09T22:30:37.427 DEBUG:boto:Method: DELETE 2022-09-09T22:30:37.428 DEBUG:boto:Path: /jkcgbj-7/ 2022-09-09T22:30:37.429 DEBUG:boto:Data: 2022-09-09T22:30:37.429 DEBUG:boto:Headers: {} 2022-09-09T22:30:37.430 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:37.431 DEBUG:boto:Port: 8000 2022-09-09T22:30:37.432 DEBUG:boto:Params: {} 2022-09-09T22:30:37.433 DEBUG:boto:Token: None 2022-09-09T22:30:37.434 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:30:37 GMT /jkcgbj-7/ 2022-09-09T22:30:37.436 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:m7ZWU9Cnwggx3+pXxax0WLGQ1uo= 2022-09-09T22:30:37.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:m7ZWU9Cnwggx3+pXxax0WLGQ1uo=', 'Content-Length': '0'} 2022-09-09T22:30:37.517 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000751dce1b1b17252d-00631bbe8d-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:30:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:37.518 DEBUG:boto:path=/jkcgbj-8/ 2022-09-09T22:30:37.519 DEBUG:boto:auth_path=/jkcgbj-8/ 2022-09-09T22:30:37.519 DEBUG:boto:Method: DELETE 2022-09-09T22:30:37.520 DEBUG:boto:Path: /jkcgbj-8/ 2022-09-09T22:30:37.520 DEBUG:boto:Data: 2022-09-09T22:30:37.520 DEBUG:boto:Headers: {} 2022-09-09T22:30:37.521 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:37.522 DEBUG:boto:Port: 8000 2022-09-09T22:30:37.522 DEBUG:boto:Params: {} 2022-09-09T22:30:37.523 DEBUG:boto:Token: None 2022-09-09T22:30:37.523 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:30:37 GMT /jkcgbj-8/ 2022-09-09T22:30:37.524 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JfDvHFcIw2/YNS/CjYDoQcyPgg0= 2022-09-09T22:30:37.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JfDvHFcIw2/YNS/CjYDoQcyPgg0=', 'Content-Length': '0'} 2022-09-09T22:30:37.667 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000098aa93ab6ae25d8-00631bbe8d-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:30:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:37.668 DEBUG:boto:path=/jkcgbj-9/ 2022-09-09T22:30:37.669 DEBUG:boto:auth_path=/jkcgbj-9/ 2022-09-09T22:30:37.670 DEBUG:boto:Method: DELETE 2022-09-09T22:30:37.670 DEBUG:boto:Path: /jkcgbj-9/ 2022-09-09T22:30:37.671 DEBUG:boto:Data: 2022-09-09T22:30:37.672 DEBUG:boto:Headers: {} 2022-09-09T22:30:37.672 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:37.673 DEBUG:boto:Port: 8001 2022-09-09T22:30:37.676 DEBUG:boto:Params: {} 2022-09-09T22:30:37.677 DEBUG:boto:Token: None 2022-09-09T22:30:37.678 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:30:37 GMT /jkcgbj-9/ 2022-09-09T22:30:37.679 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:p7rSM2aWaECk9KPwB1CPTS7434g= 2022-09-09T22:30:37.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:p7rSM2aWaECk9KPwB1CPTS7434g=', 'Content-Length': '0'} 2022-09-09T22:30:37.821 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000087a021e4a5a41035-00631bbe8d-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:30:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:37.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:37.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:37.825 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:37.874 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:37.880 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:37.920 INFO:teuthology.orchestra.run.smithi005.stderr:No --period given, using current period=503b1d61-4513-48c8-a308-17b76219610c 2022-09-09T22:30:37.946 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T22:30:37.947 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:37.948 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:37.948 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:28:06.146147Z" 2022-09-09T22:30:37.950 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:37.950 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:37.951 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:30:37.951 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:30:37.951 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:37.952 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:37.952 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:30:37.952 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:30:37.757014Z" 2022-09-09T22:30:37.953 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:37.953 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:37.953 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:30:37.953 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:30:37.954 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:37.954 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T22:30:37.955 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: ''} 2022-09-09T22:30:37.956 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-09T22:30:37.956 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:37.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:37.958 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:38.005 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:38.006 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:38.067 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:38.066+0000 7efda5696dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:38.070 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:38.070+0000 7efda5696dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:38.094 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:30:38.094 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:30:38.094 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:30:38.095 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:30:38.095 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:30:38.095 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:30:38.096 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:38.096 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:38.096 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:30:38.096 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:38.097 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:30:38.097 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:38.097 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:38.097 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:38.098 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:38.098 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:30:38.098 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:38.098 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:30:38.101 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:38.101 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:38.101 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:38.102 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:38.102 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:30:38.102 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:38.103 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:38.103 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:38.103 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:38.103 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:38.103 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:38.104 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:38.104 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:38.104 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:38.104 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:38.105 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:38.105 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:30:38.105 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:38.105 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:38.106 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.476803_74.1", 2022-09-09T22:30:38.106 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:38.106 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:38.106 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:38.107 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.476803Z", 2022-09-09T22:30:38.107 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:38.107 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:38.107 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:38.108 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:38.108 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:30:38.108 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:38.108 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:38.109 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:38.109 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:38.109 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:38.109 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:38.109 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:38.110 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:38.110 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:38.110 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:38.110 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:30:38.111 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:38.111 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:30:38.111 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:30:38.111 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:30:38.112 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:38.112 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:30:38Z" 2022-09-09T22:30:38.112 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:30:38.113 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.476803_74.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.476803Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:30:38Z" } 2022-09-09T22:30:38.114 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.476803_74.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.476803Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2022-09-09T22:30:38.115 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: ''} 2022-09-09T22:30:38.115 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.476803_74.1', 3: ''} 2022-09-09T22:30:38.115 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662762637.757014_85.1 target=1_1662762637.476803_74.1 2022-09-09T22:30:43.117 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:43.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:43.119 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:43.170 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:43.171 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:43.246 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:43.242+0000 7f6401a26dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:43.247 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:43.246+0000 7f6401a26dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:43.282 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:30:43.283 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:30:43.283 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:30:43.284 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:30:43.284 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:30:43.285 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:30:43.285 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:43.286 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:43.287 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:30:43.287 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:43.288 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:30:43.288 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:43.289 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:43.289 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:43.290 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:43.290 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:30:43.291 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:43.291 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:30:43.291 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:43.292 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:43.292 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:43.293 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:43.293 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:30:43.293 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:43.294 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:43.294 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:43.295 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:43.295 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:43.296 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:43.296 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:43.296 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:43.297 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:43.297 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:43.298 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:43.298 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:30:43.299 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:43.299 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:43.300 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:30:43.301 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:43.302 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:43.302 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:43.302 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:30:43.303 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:43.303 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:43.303 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:43.303 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:43.304 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:30:43.304 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:43.304 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:43.304 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:43.305 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:43.305 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:43.305 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:43.305 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:43.306 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:43.306 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:43.306 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:43.306 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:30:43.307 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:43.307 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:30:43.307 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:30:43.307 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:30:43.308 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:43.308 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:30:43Z" 2022-09-09T22:30:43.308 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:30:43.309 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:30:43Z" } 2022-09-09T22:30:43.309 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2022-09-09T22:30:43.310 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: ''} 2022-09-09T22:30:43.311 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: ''} 2022-09-09T22:30:43.312 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-09T22:30:43.312 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-09T22:30:43.313 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:43.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:43.314 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:43.363 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:43.364 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:43.419 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:30:43.415+0000 7fed21683dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:43.420 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:30:43.419+0000 7fed21683dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:43.445 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:30:43.446 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T22:30:43.446 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T22:30:43.449 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T22:30:43.449 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T22:30:43.449 INFO:teuthology.orchestra.run.smithi005.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:30:43.449 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:43.450 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:43.450 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T22:30:43.450 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:43.451 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T22:30:43.451 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:43.451 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:43.451 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:43.452 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:43.452 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 1, 2022-09-09T22:30:43.452 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:43.452 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:30:43.453 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:43.453 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:43.453 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:43.453 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:43.454 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T22:30:43.454 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:43.455 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:43.455 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:30:43.455 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:43.456 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:30:43.456 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:43.456 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:30:43.456 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:43.457 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:43.457 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:43.457 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:43.458 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T22:30:43.458 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:43.458 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:43.458 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:30:43.459 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:43.459 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:30:43.459 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:43.460 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:30:43.460 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:43.460 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:43.461 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:43.461 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:43.461 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T22:30:43.461 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:43.462 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:43.462 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:30:43.462 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:43.462 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:30:43.463 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:43.463 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:30:43.463 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:43.464 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:43.464 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:43.465 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:30:43.465 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:43.466 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T22:30:43.466 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 1, 2022-09-09T22:30:43.466 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 1 2022-09-09T22:30:43.466 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:43.467 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T22:30:43Z" 2022-09-09T22:30:43.467 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:30:43.469 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:30:43Z" } 2022-09-09T22:30:43.469 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2022-09-09T22:30:43.470 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: ''} 2022-09-09T22:30:43.470 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: ''} 2022-09-09T22:30:43.471 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-09T22:30:43.472 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-09T22:30:43.472 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:43.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:43.473 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:43.521 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:43.522 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:43.587 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:43.586+0000 7fa71788cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:43.588 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:43.590+0000 7fa71788cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:43.616 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:30:43.616 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:30:43.617 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:30:43.617 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:30:43.617 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:30:43.617 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:30:43.618 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:43.618 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:43.618 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:30:43.619 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:43.619 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:30:43.619 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:43.619 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:43.620 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:43.620 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:43.620 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:30:43.621 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:43.621 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:30:43.621 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:43.621 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:43.622 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:43.622 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:43.622 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:30:43.622 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:43.623 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:43.623 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:43.623 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:43.623 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:43.624 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:43.624 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:43.624 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:43.624 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:43.625 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:43.625 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:43.625 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:30:43.625 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:43.625 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:43.626 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:30:43.626 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:43.627 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:43.627 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:43.627 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:30:43.628 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:43.628 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:43.628 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:43.628 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:43.629 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:30:43.629 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:43.629 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:43.629 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:43.630 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:43.630 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:43.630 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:43.630 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:43.631 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:43.631 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:43.631 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:43.632 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:30:43.632 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:43.632 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:30:43.633 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:30:43.633 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:30:43.633 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:43.633 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:30:43Z" 2022-09-09T22:30:43.634 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:30:43.636 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:30:43Z" } 2022-09-09T22:30:43.637 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2022-09-09T22:30:43.638 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: ''} 2022-09-09T22:30:43.639 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: ''} 2022-09-09T22:30:43.640 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-09T22:30:43.641 DEBUG:boto:path=/jkcgbj-7/ 2022-09-09T22:30:43.642 DEBUG:boto:auth_path=/jkcgbj-7/ 2022-09-09T22:30:43.644 DEBUG:boto:Method: HEAD 2022-09-09T22:30:43.646 DEBUG:boto:Path: /jkcgbj-7/ 2022-09-09T22:30:43.647 DEBUG:boto:Data: 2022-09-09T22:30:43.647 DEBUG:boto:Headers: {} 2022-09-09T22:30:43.648 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:43.649 DEBUG:boto:Port: 8000 2022-09-09T22:30:43.649 DEBUG:boto:Params: {} 2022-09-09T22:30:43.650 DEBUG:boto:Token: None 2022-09-09T22:30:43.651 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:43 GMT /jkcgbj-7/ 2022-09-09T22:30:43.652 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jvqleyVg5ZWHGRGud2feuHhRMvo= 2022-09-09T22:30:43.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jvqleyVg5ZWHGRGud2feuHhRMvo=', 'Content-Length': '0'} 2022-09-09T22:30:43.656 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx0000057c6e893288415c5-00631bbe93-10ff-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:43.657 DEBUG:boto:path=/jkcgbj-8/ 2022-09-09T22:30:43.658 DEBUG:boto:auth_path=/jkcgbj-8/ 2022-09-09T22:30:43.658 DEBUG:boto:Method: HEAD 2022-09-09T22:30:43.659 DEBUG:boto:Path: /jkcgbj-8/ 2022-09-09T22:30:43.659 DEBUG:boto:Data: 2022-09-09T22:30:43.660 DEBUG:boto:Headers: {} 2022-09-09T22:30:43.661 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:43.662 DEBUG:boto:Port: 8000 2022-09-09T22:30:43.663 DEBUG:boto:Params: {} 2022-09-09T22:30:43.664 DEBUG:boto:Token: None 2022-09-09T22:30:43.665 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:43 GMT /jkcgbj-8/ 2022-09-09T22:30:43.667 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ShE479ddeH7ICO3RjELGZUVMb+Y= 2022-09-09T22:30:43.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ShE479ddeH7ICO3RjELGZUVMb+Y=', 'Content-Length': '0'} 2022-09-09T22:30:43.670 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx000004fb05912c68fd0d4-00631bbe93-10ff-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:43.671 DEBUG:boto:path=/jkcgbj-9/ 2022-09-09T22:30:43.673 DEBUG:boto:auth_path=/jkcgbj-9/ 2022-09-09T22:30:43.674 DEBUG:boto:Method: HEAD 2022-09-09T22:30:43.675 DEBUG:boto:Path: /jkcgbj-9/ 2022-09-09T22:30:43.676 DEBUG:boto:Data: 2022-09-09T22:30:43.676 DEBUG:boto:Headers: {} 2022-09-09T22:30:43.677 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:43.678 DEBUG:boto:Port: 8000 2022-09-09T22:30:43.679 DEBUG:boto:Params: {} 2022-09-09T22:30:43.679 DEBUG:boto:Token: None 2022-09-09T22:30:43.680 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:43 GMT /jkcgbj-9/ 2022-09-09T22:30:43.680 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nS/fknL55lfdRBJfGlwQN07iJwU= 2022-09-09T22:30:43.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nS/fknL55lfdRBJfGlwQN07iJwU=', 'Content-Length': '0'} 2022-09-09T22:30:43.683 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx0000098b93d57eff22ba2-00631bbe93-10ff-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:43.684 DEBUG:boto:path=/jkcgbj-7/ 2022-09-09T22:30:43.685 DEBUG:boto:auth_path=/jkcgbj-7/ 2022-09-09T22:30:43.685 DEBUG:boto:Method: HEAD 2022-09-09T22:30:43.686 DEBUG:boto:Path: /jkcgbj-7/ 2022-09-09T22:30:43.686 DEBUG:boto:Data: 2022-09-09T22:30:43.687 DEBUG:boto:Headers: {} 2022-09-09T22:30:43.687 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:43.688 DEBUG:boto:Port: 8000 2022-09-09T22:30:43.688 DEBUG:boto:Params: {} 2022-09-09T22:30:43.689 DEBUG:boto:Token: None 2022-09-09T22:30:43.690 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:43 GMT /jkcgbj-7/ 2022-09-09T22:30:43.690 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jvqleyVg5ZWHGRGud2feuHhRMvo= 2022-09-09T22:30:43.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jvqleyVg5ZWHGRGud2feuHhRMvo=', 'Content-Length': '0'} 2022-09-09T22:30:43.697 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx000005406da10cb9318b3-00631bbe93-10e8-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:43.699 DEBUG:boto:path=/jkcgbj-8/ 2022-09-09T22:30:43.699 DEBUG:boto:auth_path=/jkcgbj-8/ 2022-09-09T22:30:43.700 DEBUG:boto:Method: HEAD 2022-09-09T22:30:43.701 DEBUG:boto:Path: /jkcgbj-8/ 2022-09-09T22:30:43.702 DEBUG:boto:Data: 2022-09-09T22:30:43.703 DEBUG:boto:Headers: {} 2022-09-09T22:30:43.704 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:43.705 DEBUG:boto:Port: 8000 2022-09-09T22:30:43.706 DEBUG:boto:Params: {} 2022-09-09T22:30:43.707 DEBUG:boto:Token: None 2022-09-09T22:30:43.708 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:43 GMT /jkcgbj-8/ 2022-09-09T22:30:43.709 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ShE479ddeH7ICO3RjELGZUVMb+Y= 2022-09-09T22:30:43.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ShE479ddeH7ICO3RjELGZUVMb+Y=', 'Content-Length': '0'} 2022-09-09T22:30:43.726 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx000003ad7969bd4641cbf-00631bbe93-10e8-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:43.727 DEBUG:boto:path=/jkcgbj-9/ 2022-09-09T22:30:43.729 DEBUG:boto:auth_path=/jkcgbj-9/ 2022-09-09T22:30:43.730 DEBUG:boto:Method: HEAD 2022-09-09T22:30:43.731 DEBUG:boto:Path: /jkcgbj-9/ 2022-09-09T22:30:43.732 DEBUG:boto:Data: 2022-09-09T22:30:43.734 DEBUG:boto:Headers: {} 2022-09-09T22:30:43.735 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:43.736 DEBUG:boto:Port: 8000 2022-09-09T22:30:43.737 DEBUG:boto:Params: {} 2022-09-09T22:30:43.738 DEBUG:boto:Token: None 2022-09-09T22:30:43.739 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:43 GMT /jkcgbj-9/ 2022-09-09T22:30:43.740 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nS/fknL55lfdRBJfGlwQN07iJwU= 2022-09-09T22:30:43.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nS/fknL55lfdRBJfGlwQN07iJwU=', 'Content-Length': '0'} 2022-09-09T22:30:43.746 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx000000c4b24a00176c482-00631bbe93-10e8-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:43.749 DEBUG:boto:path=/jkcgbj-7/ 2022-09-09T22:30:43.750 DEBUG:boto:auth_path=/jkcgbj-7/ 2022-09-09T22:30:43.751 DEBUG:boto:Method: HEAD 2022-09-09T22:30:43.752 DEBUG:boto:Path: /jkcgbj-7/ 2022-09-09T22:30:43.753 DEBUG:boto:Data: 2022-09-09T22:30:43.754 DEBUG:boto:Headers: {} 2022-09-09T22:30:43.755 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:43.756 DEBUG:boto:Port: 8001 2022-09-09T22:30:43.757 DEBUG:boto:Params: {} 2022-09-09T22:30:43.759 DEBUG:boto:Token: None 2022-09-09T22:30:43.760 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:43 GMT /jkcgbj-7/ 2022-09-09T22:30:43.761 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jvqleyVg5ZWHGRGud2feuHhRMvo= 2022-09-09T22:30:43.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jvqleyVg5ZWHGRGud2feuHhRMvo=', 'Content-Length': '0'} 2022-09-09T22:30:43.766 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx00000f27992c2cd420038-00631bbe93-1103-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:43.767 DEBUG:boto:path=/jkcgbj-8/ 2022-09-09T22:30:43.768 DEBUG:boto:auth_path=/jkcgbj-8/ 2022-09-09T22:30:43.769 DEBUG:boto:Method: HEAD 2022-09-09T22:30:43.770 DEBUG:boto:Path: /jkcgbj-8/ 2022-09-09T22:30:43.771 DEBUG:boto:Data: 2022-09-09T22:30:43.771 DEBUG:boto:Headers: {} 2022-09-09T22:30:43.772 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:43.773 DEBUG:boto:Port: 8001 2022-09-09T22:30:43.774 DEBUG:boto:Params: {} 2022-09-09T22:30:43.775 DEBUG:boto:Token: None 2022-09-09T22:30:43.776 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:43 GMT /jkcgbj-8/ 2022-09-09T22:30:43.777 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ShE479ddeH7ICO3RjELGZUVMb+Y= 2022-09-09T22:30:43.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ShE479ddeH7ICO3RjELGZUVMb+Y=', 'Content-Length': '0'} 2022-09-09T22:30:43.783 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx00000faf129bdf2161e75-00631bbe93-1103-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:43.784 DEBUG:boto:path=/jkcgbj-9/ 2022-09-09T22:30:43.785 DEBUG:boto:auth_path=/jkcgbj-9/ 2022-09-09T22:30:43.785 DEBUG:boto:Method: HEAD 2022-09-09T22:30:43.786 DEBUG:boto:Path: /jkcgbj-9/ 2022-09-09T22:30:43.787 DEBUG:boto:Data: 2022-09-09T22:30:43.787 DEBUG:boto:Headers: {} 2022-09-09T22:30:43.789 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:43.789 DEBUG:boto:Port: 8001 2022-09-09T22:30:43.790 DEBUG:boto:Params: {} 2022-09-09T22:30:43.792 DEBUG:boto:Token: None 2022-09-09T22:30:43.793 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:43 GMT /jkcgbj-9/ 2022-09-09T22:30:43.794 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nS/fknL55lfdRBJfGlwQN07iJwU= 2022-09-09T22:30:43.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nS/fknL55lfdRBJfGlwQN07iJwU=', 'Content-Length': '0'} 2022-09-09T22:30:43.797 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx00000333c35b29ee294f2-00631bbe93-1103-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:43.798 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2022-09-09T22:30:43.799 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jkcgbj-10 2022-09-09T22:30:43.800 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:30:43.800 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:30:43.801 DEBUG:boto:Method: PUT 2022-09-09T22:30:43.802 DEBUG:boto:Path: /jkcgbj-10/ 2022-09-09T22:30:43.803 DEBUG:boto:Data: 2022-09-09T22:30:43.804 DEBUG:boto:Headers: {} 2022-09-09T22:30:43.804 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:43.805 DEBUG:boto:Port: 8000 2022-09-09T22:30:43.805 DEBUG:boto:Params: {} 2022-09-09T22:30:43.806 DEBUG:boto:Token: None 2022-09-09T22:30:43.807 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:30:43 GMT /jkcgbj-10/ 2022-09-09T22:30:43.808 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:MDqBcWp9m/HvP7OB9SOPspRN9BA= 2022-09-09T22:30:43.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:MDqBcWp9m/HvP7OB9SOPspRN9BA=', 'Content-Length': '0'} 2022-09-09T22:30:43.836 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000023ab8a69176a653c-00631bbe93-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:43.837 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jkcgbj-11 2022-09-09T22:30:43.837 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:30:43.838 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:30:43.838 DEBUG:boto:Method: PUT 2022-09-09T22:30:43.839 DEBUG:boto:Path: /jkcgbj-11/ 2022-09-09T22:30:43.839 DEBUG:boto:Data: 2022-09-09T22:30:43.840 DEBUG:boto:Headers: {} 2022-09-09T22:30:43.840 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:43.841 DEBUG:boto:Port: 8000 2022-09-09T22:30:43.841 DEBUG:boto:Params: {} 2022-09-09T22:30:43.842 DEBUG:boto:Token: None 2022-09-09T22:30:43.843 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:30:43 GMT /jkcgbj-11/ 2022-09-09T22:30:43.843 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:uEp+rU79unqMXM0A7MeRz5Mixn8= 2022-09-09T22:30:43.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:uEp+rU79unqMXM0A7MeRz5Mixn8=', 'Content-Length': '0'} 2022-09-09T22:30:43.890 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ec5ca7a6bc966e35-00631bbe93-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:43.891 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jkcgbj-12 2022-09-09T22:30:43.892 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:30:43.893 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:30:43.894 DEBUG:boto:Method: PUT 2022-09-09T22:30:43.894 DEBUG:boto:Path: /jkcgbj-12/ 2022-09-09T22:30:43.895 DEBUG:boto:Data: 2022-09-09T22:30:43.896 DEBUG:boto:Headers: {} 2022-09-09T22:30:43.897 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:43.898 DEBUG:boto:Port: 8001 2022-09-09T22:30:43.898 DEBUG:boto:Params: {} 2022-09-09T22:30:43.899 DEBUG:boto:Token: None 2022-09-09T22:30:43.902 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:30:43 GMT /jkcgbj-12/ 2022-09-09T22:30:43.903 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Ez/L3+2TrRGMoPiO1LzBZqB16P0= 2022-09-09T22:30:43.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Ez/L3+2TrRGMoPiO1LzBZqB16P0=', 'Content-Length': '0'} 2022-09-09T22:30:43.946 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ab9f9f32ca9d9953-00631bbe93-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:43.947 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:30:43.948 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:30:43.949 DEBUG:boto:Method: HEAD 2022-09-09T22:30:43.950 DEBUG:boto:Path: /jkcgbj-10/ 2022-09-09T22:30:43.951 DEBUG:boto:Data: 2022-09-09T22:30:43.952 DEBUG:boto:Headers: {} 2022-09-09T22:30:43.953 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:43.954 DEBUG:boto:Port: 8000 2022-09-09T22:30:43.955 DEBUG:boto:Params: {} 2022-09-09T22:30:43.956 DEBUG:boto:Token: None 2022-09-09T22:30:43.957 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:43 GMT /jkcgbj-10/ 2022-09-09T22:30:43.958 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:3e2afOmUefW/KNobFNMrft2sfRQ= 2022-09-09T22:30:43.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:3e2afOmUefW/KNobFNMrft2sfRQ=', 'Content-Length': '0'} 2022-09-09T22:30:43.964 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000eefce8e8804f1a9-00631bbe93-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:43.965 DEBUG:boto:path=/jkcgbj-10/myobj 2022-09-09T22:30:43.969 DEBUG:boto:auth_path=/jkcgbj-10/myobj 2022-09-09T22:30:43.969 DEBUG:boto:Method: PUT 2022-09-09T22:30:43.970 DEBUG:boto:Path: /jkcgbj-10/myobj 2022-09-09T22:30:43.971 DEBUG:boto:Data: 2022-09-09T22:30:43.971 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-09T22:30:43.972 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:43.972 DEBUG:boto:Port: 8000 2022-09-09T22:30:43.974 DEBUG:boto:Params: {} 2022-09-09T22:30:43.975 DEBUG:boto:Token: None 2022-09-09T22:30:43.976 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 09 Sep 2022 22:30:43 GMT /jkcgbj-10/myobj 2022-09-09T22:30:43.976 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9/uj8y5yPLeHInndesypKhSX3Mo= 2022-09-09T22:30:43.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:30:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9/uj8y5yPLeHInndesypKhSX3Mo='} 2022-09-09T22:30:44.003 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f025f07555076aee-00631bbe93-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:44.004 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:30:44.005 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:30:44.006 DEBUG:boto:Method: HEAD 2022-09-09T22:30:44.008 DEBUG:boto:Path: /jkcgbj-10/ 2022-09-09T22:30:44.009 DEBUG:boto:Data: 2022-09-09T22:30:44.010 DEBUG:boto:Headers: {} 2022-09-09T22:30:44.011 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:44.012 DEBUG:boto:Port: 8000 2022-09-09T22:30:44.012 DEBUG:boto:Params: {} 2022-09-09T22:30:44.013 DEBUG:boto:Token: None 2022-09-09T22:30:44.014 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:44 GMT /jkcgbj-10/ 2022-09-09T22:30:44.015 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:mtDhnkSHq2POy3gt4gGwTZM74wI= 2022-09-09T22:30:44.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:mtDhnkSHq2POy3gt4gGwTZM74wI=', 'Content-Length': '0'} 2022-09-09T22:30:44.026 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc332dee574fc4cc-00631bbe94-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:44.028 DEBUG:boto:path=/jkcgbj-10/_myobj 2022-09-09T22:30:44.029 DEBUG:boto:auth_path=/jkcgbj-10/_myobj 2022-09-09T22:30:44.030 DEBUG:boto:Method: PUT 2022-09-09T22:30:44.031 DEBUG:boto:Path: /jkcgbj-10/_myobj 2022-09-09T22:30:44.032 DEBUG:boto:Data: 2022-09-09T22:30:44.033 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-09T22:30:44.034 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:44.035 DEBUG:boto:Port: 8000 2022-09-09T22:30:44.036 DEBUG:boto:Params: {} 2022-09-09T22:30:44.038 DEBUG:boto:Token: None 2022-09-09T22:30:44.042 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 09 Sep 2022 22:30:44 GMT /jkcgbj-10/_myobj 2022-09-09T22:30:44.043 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+8ARD+d8mMCPjEQ0XaROSyTXf2g= 2022-09-09T22:30:44.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:30:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+8ARD+d8mMCPjEQ0XaROSyTXf2g='} 2022-09-09T22:30:44.088 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002ceb8b538462b202-00631bbe94-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:44.090 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:30:44.093 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:30:44.095 DEBUG:boto:Method: HEAD 2022-09-09T22:30:44.096 DEBUG:boto:Path: /jkcgbj-10/ 2022-09-09T22:30:44.097 DEBUG:boto:Data: 2022-09-09T22:30:44.098 DEBUG:boto:Headers: {} 2022-09-09T22:30:44.099 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:44.100 DEBUG:boto:Port: 8000 2022-09-09T22:30:44.101 DEBUG:boto:Params: {} 2022-09-09T22:30:44.102 DEBUG:boto:Token: None 2022-09-09T22:30:44.103 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:44 GMT /jkcgbj-10/ 2022-09-09T22:30:44.110 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:mtDhnkSHq2POy3gt4gGwTZM74wI= 2022-09-09T22:30:44.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:mtDhnkSHq2POy3gt4gGwTZM74wI=', 'Content-Length': '0'} 2022-09-09T22:30:44.127 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e0bb1566b20b5bc7-00631bbe94-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:44.129 DEBUG:boto:path=/jkcgbj-10/%3A 2022-09-09T22:30:44.130 DEBUG:boto:auth_path=/jkcgbj-10/%3A 2022-09-09T22:30:44.131 DEBUG:boto:Method: PUT 2022-09-09T22:30:44.132 DEBUG:boto:Path: /jkcgbj-10/%3A 2022-09-09T22:30:44.133 DEBUG:boto:Data: 2022-09-09T22:30:44.134 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-09T22:30:44.135 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:44.136 DEBUG:boto:Port: 8000 2022-09-09T22:30:44.138 DEBUG:boto:Params: {} 2022-09-09T22:30:44.139 DEBUG:boto:Token: None 2022-09-09T22:30:44.140 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 09 Sep 2022 22:30:44 GMT /jkcgbj-10/%3A 2022-09-09T22:30:44.141 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:HvBp1ofgpkv9Uq2hBbvmVu1e8ys= 2022-09-09T22:30:44.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:30:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:HvBp1ofgpkv9Uq2hBbvmVu1e8ys='} 2022-09-09T22:30:44.169 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fc1a3972cafe48f8-00631bbe94-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:44.171 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:30:44.172 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:30:44.173 DEBUG:boto:Method: HEAD 2022-09-09T22:30:44.174 DEBUG:boto:Path: /jkcgbj-10/ 2022-09-09T22:30:44.175 DEBUG:boto:Data: 2022-09-09T22:30:44.176 DEBUG:boto:Headers: {} 2022-09-09T22:30:44.178 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:44.179 DEBUG:boto:Port: 8000 2022-09-09T22:30:44.180 DEBUG:boto:Params: {} 2022-09-09T22:30:44.182 DEBUG:boto:Token: None 2022-09-09T22:30:44.183 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:44 GMT /jkcgbj-10/ 2022-09-09T22:30:44.184 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:mtDhnkSHq2POy3gt4gGwTZM74wI= 2022-09-09T22:30:44.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:mtDhnkSHq2POy3gt4gGwTZM74wI=', 'Content-Length': '0'} 2022-09-09T22:30:44.198 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cfdc86446e774300-00631bbe94-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:44.203 DEBUG:boto:path=/jkcgbj-10/%26 2022-09-09T22:30:44.203 DEBUG:boto:auth_path=/jkcgbj-10/%26 2022-09-09T22:30:44.204 DEBUG:boto:Method: PUT 2022-09-09T22:30:44.204 DEBUG:boto:Path: /jkcgbj-10/%26 2022-09-09T22:30:44.205 DEBUG:boto:Data: 2022-09-09T22:30:44.206 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-09T22:30:44.207 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:30:44.207 DEBUG:boto:Port: 8000 2022-09-09T22:30:44.208 DEBUG:boto:Params: {} 2022-09-09T22:30:44.208 DEBUG:boto:Token: None 2022-09-09T22:30:44.209 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 09 Sep 2022 22:30:44 GMT /jkcgbj-10/%26 2022-09-09T22:30:44.210 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Hwm/H5SfXQM4pI9j5PR/eu7Uzzg= 2022-09-09T22:30:44.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:30:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Hwm/H5SfXQM4pI9j5PR/eu7Uzzg='} 2022-09-09T22:30:44.258 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f0da96dd7a66bdb7-00631bbe94-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:44.259 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:30:44.260 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:30:44.260 DEBUG:boto:Method: HEAD 2022-09-09T22:30:44.261 DEBUG:boto:Path: /jkcgbj-11/ 2022-09-09T22:30:44.262 DEBUG:boto:Data: 2022-09-09T22:30:44.263 DEBUG:boto:Headers: {} 2022-09-09T22:30:44.263 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:44.265 DEBUG:boto:Port: 8000 2022-09-09T22:30:44.266 DEBUG:boto:Params: {} 2022-09-09T22:30:44.267 DEBUG:boto:Token: None 2022-09-09T22:30:44.268 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:44 GMT /jkcgbj-11/ 2022-09-09T22:30:44.269 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:WS92voawp/IycMzl3nJMekqFYEE= 2022-09-09T22:30:44.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:WS92voawp/IycMzl3nJMekqFYEE=', 'Content-Length': '0'} 2022-09-09T22:30:44.276 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005dd22c652b7a2cac-00631bbe94-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:44.278 DEBUG:boto:path=/jkcgbj-11/myobj 2022-09-09T22:30:44.279 DEBUG:boto:auth_path=/jkcgbj-11/myobj 2022-09-09T22:30:44.280 DEBUG:boto:Method: PUT 2022-09-09T22:30:44.281 DEBUG:boto:Path: /jkcgbj-11/myobj 2022-09-09T22:30:44.283 DEBUG:boto:Data: 2022-09-09T22:30:44.284 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-09T22:30:44.285 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:44.286 DEBUG:boto:Port: 8000 2022-09-09T22:30:44.287 DEBUG:boto:Params: {} 2022-09-09T22:30:44.288 DEBUG:boto:Token: None 2022-09-09T22:30:44.289 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 09 Sep 2022 22:30:44 GMT /jkcgbj-11/myobj 2022-09-09T22:30:44.290 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:3lEM9XPFgNiSmvJp23PEYEwvDfU= 2022-09-09T22:30:44.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:30:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:3lEM9XPFgNiSmvJp23PEYEwvDfU='} 2022-09-09T22:30:44.319 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000286ca74dc905b1a9-00631bbe94-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:44.320 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:30:44.321 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:30:44.323 DEBUG:boto:Method: HEAD 2022-09-09T22:30:44.324 DEBUG:boto:Path: /jkcgbj-11/ 2022-09-09T22:30:44.325 DEBUG:boto:Data: 2022-09-09T22:30:44.326 DEBUG:boto:Headers: {} 2022-09-09T22:30:44.327 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:44.328 DEBUG:boto:Port: 8000 2022-09-09T22:30:44.329 DEBUG:boto:Params: {} 2022-09-09T22:30:44.331 DEBUG:boto:Token: None 2022-09-09T22:30:44.332 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:44 GMT /jkcgbj-11/ 2022-09-09T22:30:44.333 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:WS92voawp/IycMzl3nJMekqFYEE= 2022-09-09T22:30:44.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:WS92voawp/IycMzl3nJMekqFYEE=', 'Content-Length': '0'} 2022-09-09T22:30:44.341 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b59f4a990352bda-00631bbe94-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:44.342 DEBUG:boto:path=/jkcgbj-11/_myobj 2022-09-09T22:30:44.343 DEBUG:boto:auth_path=/jkcgbj-11/_myobj 2022-09-09T22:30:44.344 DEBUG:boto:Method: PUT 2022-09-09T22:30:44.345 DEBUG:boto:Path: /jkcgbj-11/_myobj 2022-09-09T22:30:44.345 DEBUG:boto:Data: 2022-09-09T22:30:44.346 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-09T22:30:44.347 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:44.348 DEBUG:boto:Port: 8000 2022-09-09T22:30:44.348 DEBUG:boto:Params: {} 2022-09-09T22:30:44.349 DEBUG:boto:Token: None 2022-09-09T22:30:44.350 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 09 Sep 2022 22:30:44 GMT /jkcgbj-11/_myobj 2022-09-09T22:30:44.350 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ls9aaVCN55rwG3gfhYvHxylj7vQ= 2022-09-09T22:30:44.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:30:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ls9aaVCN55rwG3gfhYvHxylj7vQ='} 2022-09-09T22:30:44.399 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a1a24c3845ed632b-00631bbe94-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:44.400 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:30:44.401 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:30:44.402 DEBUG:boto:Method: HEAD 2022-09-09T22:30:44.402 DEBUG:boto:Path: /jkcgbj-11/ 2022-09-09T22:30:44.403 DEBUG:boto:Data: 2022-09-09T22:30:44.404 DEBUG:boto:Headers: {} 2022-09-09T22:30:44.405 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:44.406 DEBUG:boto:Port: 8000 2022-09-09T22:30:44.407 DEBUG:boto:Params: {} 2022-09-09T22:30:44.409 DEBUG:boto:Token: None 2022-09-09T22:30:44.412 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:44 GMT /jkcgbj-11/ 2022-09-09T22:30:44.413 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:WS92voawp/IycMzl3nJMekqFYEE= 2022-09-09T22:30:44.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:WS92voawp/IycMzl3nJMekqFYEE=', 'Content-Length': '0'} 2022-09-09T22:30:44.426 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008a18db8aa38ab55-00631bbe94-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:44.428 DEBUG:boto:path=/jkcgbj-11/%3A 2022-09-09T22:30:44.429 DEBUG:boto:auth_path=/jkcgbj-11/%3A 2022-09-09T22:30:44.430 DEBUG:boto:Method: PUT 2022-09-09T22:30:44.431 DEBUG:boto:Path: /jkcgbj-11/%3A 2022-09-09T22:30:44.432 DEBUG:boto:Data: 2022-09-09T22:30:44.433 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-09T22:30:44.434 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:44.435 DEBUG:boto:Port: 8000 2022-09-09T22:30:44.435 DEBUG:boto:Params: {} 2022-09-09T22:30:44.442 DEBUG:boto:Token: None 2022-09-09T22:30:44.444 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 09 Sep 2022 22:30:44 GMT /jkcgbj-11/%3A 2022-09-09T22:30:44.445 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:WzqbEh8GMSv3u/D2YCHQSbP1kj4= 2022-09-09T22:30:44.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:30:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:WzqbEh8GMSv3u/D2YCHQSbP1kj4='} 2022-09-09T22:30:44.474 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eae4b50b6ec4fae6-00631bbe94-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:44.476 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:30:44.477 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:30:44.480 DEBUG:boto:Method: HEAD 2022-09-09T22:30:44.481 DEBUG:boto:Path: /jkcgbj-11/ 2022-09-09T22:30:44.482 DEBUG:boto:Data: 2022-09-09T22:30:44.483 DEBUG:boto:Headers: {} 2022-09-09T22:30:44.484 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:44.484 DEBUG:boto:Port: 8000 2022-09-09T22:30:44.485 DEBUG:boto:Params: {} 2022-09-09T22:30:44.486 DEBUG:boto:Token: None 2022-09-09T22:30:44.488 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:44 GMT /jkcgbj-11/ 2022-09-09T22:30:44.489 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:WS92voawp/IycMzl3nJMekqFYEE= 2022-09-09T22:30:44.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:WS92voawp/IycMzl3nJMekqFYEE=', 'Content-Length': '0'} 2022-09-09T22:30:44.494 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002402af6dc88c6662-00631bbe94-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:44.496 DEBUG:boto:path=/jkcgbj-11/%26 2022-09-09T22:30:44.497 DEBUG:boto:auth_path=/jkcgbj-11/%26 2022-09-09T22:30:44.498 DEBUG:boto:Method: PUT 2022-09-09T22:30:44.499 DEBUG:boto:Path: /jkcgbj-11/%26 2022-09-09T22:30:44.501 DEBUG:boto:Data: 2022-09-09T22:30:44.503 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-09T22:30:44.504 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:30:44.504 DEBUG:boto:Port: 8000 2022-09-09T22:30:44.505 DEBUG:boto:Params: {} 2022-09-09T22:30:44.507 DEBUG:boto:Token: None 2022-09-09T22:30:44.508 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 09 Sep 2022 22:30:44 GMT /jkcgbj-11/%26 2022-09-09T22:30:44.509 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CUgPujcRs/0H6/1QR+vqxbJdGjY= 2022-09-09T22:30:44.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:30:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CUgPujcRs/0H6/1QR+vqxbJdGjY='} 2022-09-09T22:30:44.559 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cece56ba5e09e607-00631bbe94-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:44.561 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:30:44.562 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:30:44.563 DEBUG:boto:Method: HEAD 2022-09-09T22:30:44.582 DEBUG:boto:Path: /jkcgbj-12/ 2022-09-09T22:30:44.583 DEBUG:boto:Data: 2022-09-09T22:30:44.585 DEBUG:boto:Headers: {} 2022-09-09T22:30:44.586 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:44.587 DEBUG:boto:Port: 8001 2022-09-09T22:30:44.589 DEBUG:boto:Params: {} 2022-09-09T22:30:44.590 DEBUG:boto:Token: None 2022-09-09T22:30:44.591 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:44 GMT /jkcgbj-12/ 2022-09-09T22:30:44.594 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UGbbMWqCPprQUDcr5VavUqhyiuA= 2022-09-09T22:30:44.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UGbbMWqCPprQUDcr5VavUqhyiuA=', 'Content-Length': '0'} 2022-09-09T22:30:44.601 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004242cfab57975ce-00631bbe94-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:44.602 DEBUG:boto:path=/jkcgbj-12/myobj 2022-09-09T22:30:44.603 DEBUG:boto:auth_path=/jkcgbj-12/myobj 2022-09-09T22:30:44.604 DEBUG:boto:Method: PUT 2022-09-09T22:30:44.605 DEBUG:boto:Path: /jkcgbj-12/myobj 2022-09-09T22:30:44.605 DEBUG:boto:Data: 2022-09-09T22:30:44.606 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-09T22:30:44.606 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:44.607 DEBUG:boto:Port: 8001 2022-09-09T22:30:44.608 DEBUG:boto:Params: {} 2022-09-09T22:30:44.609 DEBUG:boto:Token: None 2022-09-09T22:30:44.609 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 09 Sep 2022 22:30:44 GMT /jkcgbj-12/myobj 2022-09-09T22:30:44.610 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:cgnkJJYQpF/gDBAG+Jm8UeVlw4c= 2022-09-09T22:30:44.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:30:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:cgnkJJYQpF/gDBAG+Jm8UeVlw4c='} 2022-09-09T22:30:44.641 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000080a3ba4077a4e5fd-00631bbe94-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:44.643 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:30:44.644 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:30:44.644 DEBUG:boto:Method: HEAD 2022-09-09T22:30:44.645 DEBUG:boto:Path: /jkcgbj-12/ 2022-09-09T22:30:44.646 DEBUG:boto:Data: 2022-09-09T22:30:44.647 DEBUG:boto:Headers: {} 2022-09-09T22:30:44.648 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:44.649 DEBUG:boto:Port: 8001 2022-09-09T22:30:44.649 DEBUG:boto:Params: {} 2022-09-09T22:30:44.650 DEBUG:boto:Token: None 2022-09-09T22:30:44.651 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:44 GMT /jkcgbj-12/ 2022-09-09T22:30:44.653 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UGbbMWqCPprQUDcr5VavUqhyiuA= 2022-09-09T22:30:44.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UGbbMWqCPprQUDcr5VavUqhyiuA=', 'Content-Length': '0'} 2022-09-09T22:30:44.663 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ecaf5eeca8483ac6-00631bbe94-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:44.665 DEBUG:boto:path=/jkcgbj-12/_myobj 2022-09-09T22:30:44.666 DEBUG:boto:auth_path=/jkcgbj-12/_myobj 2022-09-09T22:30:44.667 DEBUG:boto:Method: PUT 2022-09-09T22:30:44.667 DEBUG:boto:Path: /jkcgbj-12/_myobj 2022-09-09T22:30:44.669 DEBUG:boto:Data: 2022-09-09T22:30:44.670 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-09T22:30:44.670 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:44.672 DEBUG:boto:Port: 8001 2022-09-09T22:30:44.673 DEBUG:boto:Params: {} 2022-09-09T22:30:44.674 DEBUG:boto:Token: None 2022-09-09T22:30:44.674 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 09 Sep 2022 22:30:44 GMT /jkcgbj-12/_myobj 2022-09-09T22:30:44.675 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:B8Lyb2pNWvpkIm511d/KfouKLFE= 2022-09-09T22:30:44.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:30:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:B8Lyb2pNWvpkIm511d/KfouKLFE='} 2022-09-09T22:30:44.722 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000085bebda1442e8844-00631bbe94-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:44.723 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:30:44.724 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:30:44.725 DEBUG:boto:Method: HEAD 2022-09-09T22:30:44.726 DEBUG:boto:Path: /jkcgbj-12/ 2022-09-09T22:30:44.727 DEBUG:boto:Data: 2022-09-09T22:30:44.728 DEBUG:boto:Headers: {} 2022-09-09T22:30:44.729 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:44.730 DEBUG:boto:Port: 8001 2022-09-09T22:30:44.731 DEBUG:boto:Params: {} 2022-09-09T22:30:44.732 DEBUG:boto:Token: None 2022-09-09T22:30:44.733 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:44 GMT /jkcgbj-12/ 2022-09-09T22:30:44.734 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UGbbMWqCPprQUDcr5VavUqhyiuA= 2022-09-09T22:30:44.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UGbbMWqCPprQUDcr5VavUqhyiuA=', 'Content-Length': '0'} 2022-09-09T22:30:44.741 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007f3e42e9a440abf-00631bbe94-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:44.743 DEBUG:boto:path=/jkcgbj-12/%3A 2022-09-09T22:30:44.744 DEBUG:boto:auth_path=/jkcgbj-12/%3A 2022-09-09T22:30:44.745 DEBUG:boto:Method: PUT 2022-09-09T22:30:44.745 DEBUG:boto:Path: /jkcgbj-12/%3A 2022-09-09T22:30:44.747 DEBUG:boto:Data: 2022-09-09T22:30:44.748 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-09T22:30:44.749 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:44.750 DEBUG:boto:Port: 8001 2022-09-09T22:30:44.752 DEBUG:boto:Params: {} 2022-09-09T22:30:44.753 DEBUG:boto:Token: None 2022-09-09T22:30:44.755 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 09 Sep 2022 22:30:44 GMT /jkcgbj-12/%3A 2022-09-09T22:30:44.756 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:/ZUwPSsyNkpZkLxBPT99nIg3ytc= 2022-09-09T22:30:44.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:30:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:/ZUwPSsyNkpZkLxBPT99nIg3ytc='} 2022-09-09T22:30:44.783 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000641061afcebaba1c-00631bbe94-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:44.784 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:30:44.785 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:30:44.787 DEBUG:boto:Method: HEAD 2022-09-09T22:30:44.790 DEBUG:boto:Path: /jkcgbj-12/ 2022-09-09T22:30:44.791 DEBUG:boto:Data: 2022-09-09T22:30:44.792 DEBUG:boto:Headers: {} 2022-09-09T22:30:44.794 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:44.795 DEBUG:boto:Port: 8001 2022-09-09T22:30:44.797 DEBUG:boto:Params: {} 2022-09-09T22:30:44.798 DEBUG:boto:Token: None 2022-09-09T22:30:44.801 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:30:44 GMT /jkcgbj-12/ 2022-09-09T22:30:44.802 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UGbbMWqCPprQUDcr5VavUqhyiuA= 2022-09-09T22:30:44.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:30:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UGbbMWqCPprQUDcr5VavUqhyiuA=', 'Content-Length': '0'} 2022-09-09T22:30:44.808 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff278f427fc24894-00631bbe94-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:44.810 DEBUG:boto:path=/jkcgbj-12/%26 2022-09-09T22:30:44.811 DEBUG:boto:auth_path=/jkcgbj-12/%26 2022-09-09T22:30:44.813 DEBUG:boto:Method: PUT 2022-09-09T22:30:44.814 DEBUG:boto:Path: /jkcgbj-12/%26 2022-09-09T22:30:44.823 DEBUG:boto:Data: 2022-09-09T22:30:44.825 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-09T22:30:44.826 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:30:44.826 DEBUG:boto:Port: 8001 2022-09-09T22:30:44.827 DEBUG:boto:Params: {} 2022-09-09T22:30:44.828 DEBUG:boto:Token: None 2022-09-09T22:30:44.829 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 09 Sep 2022 22:30:44 GMT /jkcgbj-12/%26 2022-09-09T22:30:44.832 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:E1JpuPo5l+I5mB8IyB2sAldPYsc= 2022-09-09T22:30:44.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:30:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:E1JpuPo5l+I5mB8IyB2sAldPYsc='} 2022-09-09T22:30:44.882 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d9a3949a1e5773e4-00631bbe94-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:30:44.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:44.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:44.886 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:44.934 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:44.934 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:44.985 INFO:teuthology.orchestra.run.smithi005.stderr:No --period given, using current period=503b1d61-4513-48c8-a308-17b76219610c 2022-09-09T22:30:45.014 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T22:30:45.015 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:45.016 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:45.016 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:28:06.146147Z" 2022-09-09T22:30:45.017 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:45.017 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:45.018 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:30:45.018 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:30:45.019 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:45.019 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:45.020 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:30:45.020 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:30:37.757014Z" 2022-09-09T22:30:45.022 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:45.022 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:45.023 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762643.933454_47.1", 2022-09-09T22:30:45.023 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:30:43.933454Z" 2022-09-09T22:30:45.024 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:45.025 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T22:30:45.027 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662762643.933454_47.1'} 2022-09-09T22:30:45.028 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-09T22:30:45.029 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:45.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:45.031 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:45.096 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:45.096 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:45.144 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:45.134+0000 7fcdc7d07dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:45.144 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:45.138+0000 7fcdc7d07dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:45.172 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:30:45.172 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:30:45.172 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:30:45.173 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:30:45.173 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:30:45.173 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:30:45.173 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:45.174 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:45.174 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:30:45.174 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:45.174 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:30:45.175 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:45.175 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:45.175 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:45.176 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:45.176 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:30:45.176 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:45.176 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:30:45.177 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:45.177 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:45.177 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:45.177 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:45.178 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:30:45.178 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:45.178 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:45.178 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:45.179 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:45.179 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:45.179 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:45.179 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:45.180 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:45.180 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:45.180 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:45.180 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:45.181 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:30:45.181 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:45.181 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:45.181 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:30:45.192 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:45.192 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:45.192 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:45.193 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:30:45.193 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:45.193 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:45.193 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:45.194 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:45.194 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:30:45.194 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:45.194 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:45.195 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762643.827940_36.1", 2022-09-09T22:30:45.195 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:45.195 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:45.195 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:45.196 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:43.827940Z", 2022-09-09T22:30:45.196 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:45.197 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:45.197 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:45.198 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:30:45.199 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:45.199 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:30:45.199 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:30:45.200 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:30:45.200 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:45.200 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:30:45Z" 2022-09-09T22:30:45.201 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:30:45.201 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662762643.827940_36.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:43.827940Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:30:45Z" } 2022-09-09T22:30:45.202 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662762643.827940_36.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:43.827940Z', 'realm_epoch': 2}}] 2022-09-09T22:30:45.202 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662762643.933454_47.1'} 2022-09-09T22:30:45.203 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662762643.827940_36.1'} 2022-09-09T22:30:45.203 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662762643.933454_47.1 target=1_1662762643.827940_36.1 2022-09-09T22:30:50.207 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:50.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:50.210 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:50.254 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:50.255 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:50.317 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:50.318+0000 7f5aa5ebcdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:50.319 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:50.318+0000 7f5aa5ebcdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:50.342 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:30:50.343 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:30:50.343 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:30:50.344 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:30:50.344 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:30:50.344 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:30:50.344 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:50.345 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:50.345 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:30:50.345 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:50.345 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:30:50.346 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:50.346 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:50.346 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:50.347 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:50.347 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:30:50.347 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:50.347 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:30:50.348 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:50.348 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:50.348 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:50.348 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:50.349 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:30:50.349 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:50.349 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:50.349 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:50.350 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:50.350 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:50.350 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:50.350 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:50.351 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:50.351 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:50.351 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:50.351 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:50.352 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:30:50.352 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:50.352 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:50.352 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:30:50.353 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:50.353 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:50.353 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:50.354 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:30:50.354 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:50.354 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:50.354 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:50.355 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:50.355 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:30:50.355 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:50.355 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:50.356 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762643.933454_47.1", 2022-09-09T22:30:50.356 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:50.356 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:50.357 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:50.357 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:43.933454Z", 2022-09-09T22:30:50.357 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:50.357 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:50.358 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:50.358 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:30:50.358 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:50.359 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:30:50.359 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:30:50.359 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:30:50.359 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:50.360 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:30:50Z" 2022-09-09T22:30:50.360 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:30:50.361 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662762643.933454_47.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:43.933454Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:30:50Z" } 2022-09-09T22:30:50.362 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662762643.933454_47.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:43.933454Z', 'realm_epoch': 2}}] 2022-09-09T22:30:50.362 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662762643.933454_47.1'} 2022-09-09T22:30:50.363 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662762643.933454_47.1'} 2022-09-09T22:30:50.363 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-09T22:30:50.364 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-09T22:30:50.364 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:50.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:50.366 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:50.415 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:50.416 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:50.468 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:30:50.467+0000 7f2823c40dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:50.470 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:30:50.471+0000 7f2823c40dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:50.495 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:30:50.495 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T22:30:50.496 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T22:30:50.496 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T22:30:50.496 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T22:30:50.496 INFO:teuthology.orchestra.run.smithi005.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:30:50.497 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:50.497 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:50.497 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T22:30:50.498 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:50.498 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T22:30:50.498 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:50.498 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:50.499 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:50.499 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:50.499 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 1, 2022-09-09T22:30:50.499 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:50.500 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:30:50.500 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:50.500 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:50.501 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:50.502 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:50.502 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T22:30:50.502 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:50.503 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:50.503 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:30:50.503 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:50.503 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:30:50.504 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:50.504 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:30:50.504 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:50.505 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:50.505 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:50.505 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:50.506 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T22:30:50.506 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:50.506 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:50.506 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:30:50.507 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:50.507 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:30:50.507 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:50.508 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:30:50.508 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:50.508 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:50.508 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:50.509 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:30:50.509 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T22:30:50.509 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:30:50.510 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:30:50.510 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762643.933454_47.1", 2022-09-09T22:30:50.510 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:30:50.510 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:30:50.511 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:30:50.511 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:43.933454Z", 2022-09-09T22:30:50.512 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:30:50.512 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:50.512 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:30:50.512 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:30:50.513 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:50.513 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T22:30:50.513 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 1, 2022-09-09T22:30:50.514 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 1 2022-09-09T22:30:50.514 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:30:50.514 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T22:30:50Z" 2022-09-09T22:30:50.514 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:30:50.516 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662762643.933454_47.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:43.933454Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:30:50Z" } 2022-09-09T22:30:50.517 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662762643.933454_47.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:43.933454Z', 'realm_epoch': 2}}] 2022-09-09T22:30:50.517 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662762643.933454_47.1'} 2022-09-09T22:30:50.518 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662762643.933454_47.1'} 2022-09-09T22:30:50.518 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-09T22:30:50.519 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-09T22:30:50.519 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:50.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:30:50.521 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:30:50.566 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:50.567 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:50.628 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:50.626+0000 7fd792bf4dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:50.629 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:50.630+0000 7fd792bf4dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:30:50.652 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:30:50.653 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:30:50.654 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:30:50.655 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:30:50.655 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:30:50.656 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:30:50.656 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:50.656 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:50.656 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:30:50.657 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:50.657 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:30:50.657 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:50.658 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:50.658 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:30:50.658 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:50.658 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:30:50.659 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:50.659 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:30:50.660 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:50.660 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:50.661 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:50.661 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:50.662 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:30:50.662 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:50.662 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:50.662 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:30:50.663 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:50.663 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:50.663 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:50.664 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:30:50.664 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:50.664 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:50.665 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:50.665 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:50.665 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:30:50.665 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:50.666 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:50.666 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:30:50.666 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:50.667 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:50.667 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:50.667 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:30:50.667 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:50.668 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:50.668 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:50.668 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:30:50.668 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:30:50.669 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:30:50.669 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:30:50.669 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762643.933454_47.1", 2022-09-09T22:30:50.669 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:30:50.670 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:30:50.670 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:30:50.671 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:43.933454Z", 2022-09-09T22:30:50.671 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:30:50.672 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:50.672 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:30:50.673 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:30:50.673 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:50.673 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:30:50.674 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:30:50.674 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:30:50.675 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:30:50.675 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:30:50Z" 2022-09-09T22:30:50.676 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:30:50.677 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662762643.933454_47.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:43.933454Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:30:50Z" } 2022-09-09T22:30:50.678 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662762643.933454_47.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:43.933454Z', 'realm_epoch': 2}}] 2022-09-09T22:30:50.679 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662762643.933454_47.1'} 2022-09-09T22:30:50.680 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662762643.933454_47.1'} 2022-09-09T22:30:50.680 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-09T22:30:50.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:30:50.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:30:50.682 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:30:50.736 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:30:50.737 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:30:50.803 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:50.802+0000 7f0f81e87dc0 1 waiting to reach incremental sync.. 2022-09-09T22:30:55.803 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:30:55.802+0000 7f0f81e87dc0 1 waiting to reach incremental sync.. 2022-09-09T22:31:00.810 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:31:00.806+0000 7f0f81e87dc0 1 bucket sync caught up with source: 2022-09-09T22:31:00.810 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000001.6.5, , 00000000001.4.5, , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:31:00.811 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000001.6.5, , 00000000001.4.5, , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:31:00.811 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:31:00.806+0000 7f0f81e87dc0 0 bucket checkpoint complete 2022-09-09T22:31:00.836 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-10 zones={test-zone2, test-zone1} 2022-09-09T22:31:00.837 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:00.838 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:00.839 DEBUG:boto:Method: HEAD 2022-09-09T22:31:00.840 DEBUG:boto:Path: /jkcgbj-10/ 2022-09-09T22:31:00.841 DEBUG:boto:Data: 2022-09-09T22:31:00.842 DEBUG:boto:Headers: {} 2022-09-09T22:31:00.843 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:31:00.844 DEBUG:boto:Port: 8000 2022-09-09T22:31:00.844 DEBUG:boto:Params: {} 2022-09-09T22:31:00.846 DEBUG:boto:Token: None 2022-09-09T22:31:00.848 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:00 GMT /jkcgbj-10/ 2022-09-09T22:31:00.849 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sZvdAtVqtxIWzW+QyEVQAPbBqXc= 2022-09-09T22:31:00.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sZvdAtVqtxIWzW+QyEVQAPbBqXc=', 'Content-Length': '0'} 2022-09-09T22:31:00.855 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000051b3e5c5ac95d5fd-00631bbea4-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:31:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:00.857 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:00.859 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:00.861 DEBUG:boto:Method: HEAD 2022-09-09T22:31:00.861 DEBUG:boto:Path: /jkcgbj-10/ 2022-09-09T22:31:00.862 DEBUG:boto:Data: 2022-09-09T22:31:00.863 DEBUG:boto:Headers: {} 2022-09-09T22:31:00.864 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:00.865 DEBUG:boto:Port: 8000 2022-09-09T22:31:00.867 DEBUG:boto:Params: {} 2022-09-09T22:31:00.868 DEBUG:boto:Token: None 2022-09-09T22:31:00.870 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:00 GMT /jkcgbj-10/ 2022-09-09T22:31:00.871 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sZvdAtVqtxIWzW+QyEVQAPbBqXc= 2022-09-09T22:31:00.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sZvdAtVqtxIWzW+QyEVQAPbBqXc=', 'Content-Length': '0'} 2022-09-09T22:31:00.880 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b2520b398784fff-00631bbea4-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:31:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:00.884 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:31:00.886 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:00.906 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:00.907 DEBUG:boto:path=/jkcgbj-10/?versions&max-keys=999 2022-09-09T22:31:00.908 DEBUG:boto:auth_path=/jkcgbj-10/?versions&max-keys=999 2022-09-09T22:31:00.908 DEBUG:boto:Method: GET 2022-09-09T22:31:00.909 DEBUG:boto:Path: /jkcgbj-10/?versions&max-keys=999 2022-09-09T22:31:00.909 DEBUG:boto:Data: 2022-09-09T22:31:00.910 DEBUG:boto:Headers: {} 2022-09-09T22:31:00.911 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:31:00.912 DEBUG:boto:Port: 8000 2022-09-09T22:31:00.913 DEBUG:boto:Params: {} 2022-09-09T22:31:00.914 DEBUG:boto:Token: None 2022-09-09T22:31:00.915 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:00 GMT /jkcgbj-10/?versions 2022-09-09T22:31:00.915 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9yM/sTXfSMVqVBjCgQw0CN0/SXg= 2022-09-09T22:31:00.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9yM/sTXfSMVqVBjCgQw0CN0/SXg=', 'Content-Length': '0'} 2022-09-09T22:31:00.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c806cb2fcbf367e4-00631bbea4-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:00.923 DEBUG:boto:b'jkcgbj-10999false&nulltrue2022-09-09T22:30:44.219Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-09T22:30:44.155Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-09T22:30:44.054Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-09T22:30:43.986Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:00.925 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-09T22:31:00.926 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-09T22:31:00.927 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-09T22:31:00.927 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-09T22:31:00.928 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:31:00.928 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:00.929 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:00.930 DEBUG:boto:path=/jkcgbj-10/?versions&max-keys=999 2022-09-09T22:31:00.930 DEBUG:boto:auth_path=/jkcgbj-10/?versions&max-keys=999 2022-09-09T22:31:00.931 DEBUG:boto:Method: GET 2022-09-09T22:31:00.932 DEBUG:boto:Path: /jkcgbj-10/?versions&max-keys=999 2022-09-09T22:31:00.932 DEBUG:boto:Data: 2022-09-09T22:31:00.933 DEBUG:boto:Headers: {} 2022-09-09T22:31:00.933 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:00.934 DEBUG:boto:Port: 8000 2022-09-09T22:31:00.934 DEBUG:boto:Params: {} 2022-09-09T22:31:00.935 DEBUG:boto:Token: None 2022-09-09T22:31:00.935 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:00 GMT /jkcgbj-10/?versions 2022-09-09T22:31:00.936 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9yM/sTXfSMVqVBjCgQw0CN0/SXg= 2022-09-09T22:31:00.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9yM/sTXfSMVqVBjCgQw0CN0/SXg=', 'Content-Length': '0'} 2022-09-09T22:31:00.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d7507e0a6de32f8-00631bbea4-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:00.943 DEBUG:boto:b'jkcgbj-10999false&nulltrue2022-09-09T22:30:44.219Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-09T22:30:44.155Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-09T22:30:44.054Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-09T22:30:43.986Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:00.944 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-09T22:31:00.945 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-09T22:31:00.945 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-09T22:31:00.946 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-09T22:31:00.946 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:00.947 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:00.947 DEBUG:boto:path=/jkcgbj-10/?versions&max-keys=999 2022-09-09T22:31:00.948 DEBUG:boto:auth_path=/jkcgbj-10/?versions&max-keys=999 2022-09-09T22:31:00.948 DEBUG:boto:Method: GET 2022-09-09T22:31:00.949 DEBUG:boto:Path: /jkcgbj-10/?versions&max-keys=999 2022-09-09T22:31:00.949 DEBUG:boto:Data: 2022-09-09T22:31:00.950 DEBUG:boto:Headers: {} 2022-09-09T22:31:00.950 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:31:00.951 DEBUG:boto:Port: 8000 2022-09-09T22:31:00.951 DEBUG:boto:Params: {} 2022-09-09T22:31:00.952 DEBUG:boto:Token: None 2022-09-09T22:31:00.953 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:00 GMT /jkcgbj-10/?versions 2022-09-09T22:31:00.954 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9yM/sTXfSMVqVBjCgQw0CN0/SXg= 2022-09-09T22:31:00.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9yM/sTXfSMVqVBjCgQw0CN0/SXg=', 'Content-Length': '0'} 2022-09-09T22:31:00.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2551732d77b35fa-00631bbea4-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:00.961 DEBUG:boto:b'jkcgbj-10999false&nulltrue2022-09-09T22:30:44.219Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-09T22:30:44.155Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-09T22:30:44.054Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-09T22:30:43.986Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:00.962 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:00.963 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:00.963 DEBUG:boto:path=/jkcgbj-10/?versions&max-keys=999 2022-09-09T22:31:00.966 DEBUG:boto:auth_path=/jkcgbj-10/?versions&max-keys=999 2022-09-09T22:31:00.966 DEBUG:boto:Method: GET 2022-09-09T22:31:00.967 DEBUG:boto:Path: /jkcgbj-10/?versions&max-keys=999 2022-09-09T22:31:00.968 DEBUG:boto:Data: 2022-09-09T22:31:00.969 DEBUG:boto:Headers: {} 2022-09-09T22:31:00.970 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:00.970 DEBUG:boto:Port: 8000 2022-09-09T22:31:00.972 DEBUG:boto:Params: {} 2022-09-09T22:31:00.973 DEBUG:boto:Token: None 2022-09-09T22:31:00.975 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:00 GMT /jkcgbj-10/?versions 2022-09-09T22:31:00.976 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9yM/sTXfSMVqVBjCgQw0CN0/SXg= 2022-09-09T22:31:00.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9yM/sTXfSMVqVBjCgQw0CN0/SXg=', 'Content-Length': '0'} 2022-09-09T22:31:00.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c09cfb6e820d8f80-00631bbea4-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:00.990 DEBUG:boto:b'jkcgbj-10999false&nulltrue2022-09-09T22:30:44.219Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-09T22:30:44.155Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-09T22:30:44.054Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-09T22:30:43.986Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:00.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-09T22:31:00.992 DEBUG:boto:path=/jkcgbj-10/%26 2022-09-09T22:31:00.993 DEBUG:boto:auth_path=/jkcgbj-10/%26 2022-09-09T22:31:00.994 DEBUG:boto:path=/jkcgbj-10/%26?versionId=null 2022-09-09T22:31:00.995 DEBUG:boto:auth_path=/jkcgbj-10/%26?versionId=null 2022-09-09T22:31:00.998 DEBUG:boto:Method: GET 2022-09-09T22:31:01.000 DEBUG:boto:Path: /jkcgbj-10/%26?versionId=null 2022-09-09T22:31:01.001 DEBUG:boto:Data: 2022-09-09T22:31:01.002 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.003 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:31:01.004 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.005 DEBUG:boto:Params: {} 2022-09-09T22:31:01.006 DEBUG:boto:Token: None 2022-09-09T22:31:01.007 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/%26?versionId=null 2022-09-09T22:31:01.010 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:/rAOIRVDHhp77aJDtfxUr7HSiIA= 2022-09-09T22:31:01.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:/rAOIRVDHhp77aJDtfxUr7HSiIA=', 'Content-Length': '0'} 2022-09-09T22:31:01.026 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2081d187d3ea5f9-00631bbea5-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.028 DEBUG:boto:path=/jkcgbj-10/%26 2022-09-09T22:31:01.034 DEBUG:boto:auth_path=/jkcgbj-10/%26 2022-09-09T22:31:01.035 DEBUG:boto:path=/jkcgbj-10/%26?versionId=null 2022-09-09T22:31:01.035 DEBUG:boto:auth_path=/jkcgbj-10/%26?versionId=null 2022-09-09T22:31:01.036 DEBUG:boto:Method: GET 2022-09-09T22:31:01.036 DEBUG:boto:Path: /jkcgbj-10/%26?versionId=null 2022-09-09T22:31:01.037 DEBUG:boto:Data: 2022-09-09T22:31:01.037 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.038 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:01.038 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.039 DEBUG:boto:Params: {} 2022-09-09T22:31:01.040 DEBUG:boto:Token: None 2022-09-09T22:31:01.040 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/%26?versionId=null 2022-09-09T22:31:01.042 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:/rAOIRVDHhp77aJDtfxUr7HSiIA= 2022-09-09T22:31:01.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:/rAOIRVDHhp77aJDtfxUr7HSiIA=', 'Content-Length': '0'} 2022-09-09T22:31:01.051 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013d3ff4e36ed4e2e-00631bbea5-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.053 DEBUG:boto:path=/jkcgbj-10/%26 2022-09-09T22:31:01.053 DEBUG:boto:auth_path=/jkcgbj-10/%26 2022-09-09T22:31:01.054 DEBUG:boto:path=/jkcgbj-10/%26?versionId=null 2022-09-09T22:31:01.054 DEBUG:boto:auth_path=/jkcgbj-10/%26?versionId=null 2022-09-09T22:31:01.055 DEBUG:boto:Method: HEAD 2022-09-09T22:31:01.056 DEBUG:boto:Path: /jkcgbj-10/%26?versionId=null 2022-09-09T22:31:01.056 DEBUG:boto:Data: 2022-09-09T22:31:01.057 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.057 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:31:01.058 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.058 DEBUG:boto:Params: {} 2022-09-09T22:31:01.059 DEBUG:boto:Token: None 2022-09-09T22:31:01.060 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/%26?versionId=null 2022-09-09T22:31:01.061 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:5qTa2SRH2sM7aqmGBlFc2TW6vkk= 2022-09-09T22:31:01.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:5qTa2SRH2sM7aqmGBlFc2TW6vkk=', 'Content-Length': '0'} 2022-09-09T22:31:01.066 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000154909c2871e41b-00631bbea5-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.067 DEBUG:boto:path=/jkcgbj-10/%26 2022-09-09T22:31:01.069 DEBUG:boto:auth_path=/jkcgbj-10/%26 2022-09-09T22:31:01.070 DEBUG:boto:path=/jkcgbj-10/%26?versionId=null 2022-09-09T22:31:01.071 DEBUG:boto:auth_path=/jkcgbj-10/%26?versionId=null 2022-09-09T22:31:01.071 DEBUG:boto:Method: HEAD 2022-09-09T22:31:01.072 DEBUG:boto:Path: /jkcgbj-10/%26?versionId=null 2022-09-09T22:31:01.072 DEBUG:boto:Data: 2022-09-09T22:31:01.073 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.073 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:01.074 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.076 DEBUG:boto:Params: {} 2022-09-09T22:31:01.076 DEBUG:boto:Token: None 2022-09-09T22:31:01.077 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/%26?versionId=null 2022-09-09T22:31:01.077 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:5qTa2SRH2sM7aqmGBlFc2TW6vkk= 2022-09-09T22:31:01.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:5qTa2SRH2sM7aqmGBlFc2TW6vkk=', 'Content-Length': '0'} 2022-09-09T22:31:01.088 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020acd7f5e479b246-00631bbea5-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-09T22:31:01.090 DEBUG:boto:path=/jkcgbj-10/%26 2022-09-09T22:31:01.091 DEBUG:boto:auth_path=/jkcgbj-10/%26 2022-09-09T22:31:01.091 DEBUG:boto:path=/jkcgbj-10/%26?versionId=null 2022-09-09T22:31:01.092 DEBUG:boto:auth_path=/jkcgbj-10/%26?versionId=null 2022-09-09T22:31:01.092 DEBUG:boto:Method: GET 2022-09-09T22:31:01.093 DEBUG:boto:Path: /jkcgbj-10/%26?versionId=null 2022-09-09T22:31:01.093 DEBUG:boto:Data: 2022-09-09T22:31:01.094 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.094 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:31:01.095 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.095 DEBUG:boto:Params: {} 2022-09-09T22:31:01.096 DEBUG:boto:Token: None 2022-09-09T22:31:01.098 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/%26?versionId=null 2022-09-09T22:31:01.099 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:/rAOIRVDHhp77aJDtfxUr7HSiIA= 2022-09-09T22:31:01.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:/rAOIRVDHhp77aJDtfxUr7HSiIA=', 'Content-Length': '0'} 2022-09-09T22:31:01.107 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000039eca245d57178f2-00631bbea5-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.108 DEBUG:boto:path=/jkcgbj-10/%26 2022-09-09T22:31:01.109 DEBUG:boto:auth_path=/jkcgbj-10/%26 2022-09-09T22:31:01.109 DEBUG:boto:path=/jkcgbj-10/%26?versionId=null 2022-09-09T22:31:01.110 DEBUG:boto:auth_path=/jkcgbj-10/%26?versionId=null 2022-09-09T22:31:01.111 DEBUG:boto:Method: GET 2022-09-09T22:31:01.111 DEBUG:boto:Path: /jkcgbj-10/%26?versionId=null 2022-09-09T22:31:01.112 DEBUG:boto:Data: 2022-09-09T22:31:01.112 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.113 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:01.113 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.114 DEBUG:boto:Params: {} 2022-09-09T22:31:01.114 DEBUG:boto:Token: None 2022-09-09T22:31:01.115 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/%26?versionId=null 2022-09-09T22:31:01.116 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:/rAOIRVDHhp77aJDtfxUr7HSiIA= 2022-09-09T22:31:01.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:/rAOIRVDHhp77aJDtfxUr7HSiIA=', 'Content-Length': '0'} 2022-09-09T22:31:01.120 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005384e73526bcd3e-00631bbea5-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.166 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:01.167 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:01.167 DEBUG:boto:path=/jkcgbj-10/?prefix=%26 2022-09-09T22:31:01.168 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=%26 2022-09-09T22:31:01.169 DEBUG:boto:Method: GET 2022-09-09T22:31:01.169 DEBUG:boto:Path: /jkcgbj-10/?prefix=%26 2022-09-09T22:31:01.170 DEBUG:boto:Data: 2022-09-09T22:31:01.170 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.171 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:31:01.171 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.172 DEBUG:boto:Params: {} 2022-09-09T22:31:01.172 DEBUG:boto:Token: None 2022-09-09T22:31:01.175 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/ 2022-09-09T22:31:01.176 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2rXMYFsIHzm7bYFng+9WyP9SJqQ= 2022-09-09T22:31:01.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2rXMYFsIHzm7bYFng+9WyP9SJqQ=', 'Content-Length': '0'} 2022-09-09T22:31:01.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043cda955571abdb6-00631bbea5-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.183 DEBUG:boto:b'jkcgbj-10&1000false&2022-09-09T22:30:44.219Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:01.185 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:01.186 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:01.186 DEBUG:boto:path=/jkcgbj-10/?prefix=%26 2022-09-09T22:31:01.187 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=%26 2022-09-09T22:31:01.187 DEBUG:boto:Method: GET 2022-09-09T22:31:01.188 DEBUG:boto:Path: /jkcgbj-10/?prefix=%26 2022-09-09T22:31:01.188 DEBUG:boto:Data: 2022-09-09T22:31:01.189 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.189 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:01.190 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.190 DEBUG:boto:Params: {} 2022-09-09T22:31:01.191 DEBUG:boto:Token: None 2022-09-09T22:31:01.192 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/ 2022-09-09T22:31:01.192 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2rXMYFsIHzm7bYFng+9WyP9SJqQ= 2022-09-09T22:31:01.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2rXMYFsIHzm7bYFng+9WyP9SJqQ=', 'Content-Length': '0'} 2022-09-09T22:31:01.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026e9ee147473faa3-00631bbea5-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.200 DEBUG:boto:b'jkcgbj-10&1000false&2022-09-09T22:30:44.219Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:01.201 DEBUG:boto:path=/jkcgbj-10/%26 2022-09-09T22:31:01.202 DEBUG:boto:auth_path=/jkcgbj-10/%26 2022-09-09T22:31:01.202 DEBUG:boto:Method: HEAD 2022-09-09T22:31:01.203 DEBUG:boto:Path: /jkcgbj-10/%26 2022-09-09T22:31:01.203 DEBUG:boto:Data: 2022-09-09T22:31:01.206 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.207 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:31:01.208 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.208 DEBUG:boto:Params: {} 2022-09-09T22:31:01.209 DEBUG:boto:Token: None 2022-09-09T22:31:01.210 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/%26 2022-09-09T22:31:01.211 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:I4Es1budYWzZhpJ9RtMPDcN8L+4= 2022-09-09T22:31:01.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:I4Es1budYWzZhpJ9RtMPDcN8L+4=', 'Content-Length': '0'} 2022-09-09T22:31:01.216 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000089a10fc333a1b014-00631bbea5-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.217 DEBUG:boto:path=/jkcgbj-10/%26 2022-09-09T22:31:01.218 DEBUG:boto:auth_path=/jkcgbj-10/%26 2022-09-09T22:31:01.222 DEBUG:boto:Method: HEAD 2022-09-09T22:31:01.223 DEBUG:boto:Path: /jkcgbj-10/%26 2022-09-09T22:31:01.223 DEBUG:boto:Data: 2022-09-09T22:31:01.224 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.225 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:01.226 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.226 DEBUG:boto:Params: {} 2022-09-09T22:31:01.227 DEBUG:boto:Token: None 2022-09-09T22:31:01.227 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/%26 2022-09-09T22:31:01.228 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:I4Es1budYWzZhpJ9RtMPDcN8L+4= 2022-09-09T22:31:01.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:I4Es1budYWzZhpJ9RtMPDcN8L+4=', 'Content-Length': '0'} 2022-09-09T22:31:01.232 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d19fb581d20f6516-00631bbea5-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-09T22:31:01.233 DEBUG:boto:path=/jkcgbj-10/%26 2022-09-09T22:31:01.234 DEBUG:boto:auth_path=/jkcgbj-10/%26 2022-09-09T22:31:01.236 DEBUG:boto:Method: GET 2022-09-09T22:31:01.239 DEBUG:boto:Path: /jkcgbj-10/%26 2022-09-09T22:31:01.240 DEBUG:boto:Data: 2022-09-09T22:31:01.241 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.242 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:31:01.243 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.244 DEBUG:boto:Params: {} 2022-09-09T22:31:01.245 DEBUG:boto:Token: None 2022-09-09T22:31:01.246 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/%26 2022-09-09T22:31:01.247 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jP7Izkd5QOYMfsVlUx1v7NcwOW0= 2022-09-09T22:31:01.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jP7Izkd5QOYMfsVlUx1v7NcwOW0=', 'Content-Length': '0'} 2022-09-09T22:31:01.254 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000052f8fd17657d8aff-00631bbea5-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.298 DEBUG:boto:path=/jkcgbj-10/%26 2022-09-09T22:31:01.305 DEBUG:boto:auth_path=/jkcgbj-10/%26 2022-09-09T22:31:01.306 DEBUG:boto:Method: GET 2022-09-09T22:31:01.307 DEBUG:boto:Path: /jkcgbj-10/%26 2022-09-09T22:31:01.308 DEBUG:boto:Data: 2022-09-09T22:31:01.309 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.310 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:01.311 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.312 DEBUG:boto:Params: {} 2022-09-09T22:31:01.313 DEBUG:boto:Token: None 2022-09-09T22:31:01.316 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/%26 2022-09-09T22:31:01.317 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jP7Izkd5QOYMfsVlUx1v7NcwOW0= 2022-09-09T22:31:01.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jP7Izkd5QOYMfsVlUx1v7NcwOW0=', 'Content-Length': '0'} 2022-09-09T22:31:01.324 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d64a4f66781fd36c-00631bbea5-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.325 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:01.325 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:01.326 DEBUG:boto:path=/jkcgbj-10/?prefix=%26 2022-09-09T22:31:01.326 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=%26 2022-09-09T22:31:01.327 DEBUG:boto:Method: GET 2022-09-09T22:31:01.328 DEBUG:boto:Path: /jkcgbj-10/?prefix=%26 2022-09-09T22:31:01.328 DEBUG:boto:Data: 2022-09-09T22:31:01.329 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.330 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:31:01.330 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.331 DEBUG:boto:Params: {} 2022-09-09T22:31:01.332 DEBUG:boto:Token: None 2022-09-09T22:31:01.332 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/ 2022-09-09T22:31:01.338 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2rXMYFsIHzm7bYFng+9WyP9SJqQ= 2022-09-09T22:31:01.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2rXMYFsIHzm7bYFng+9WyP9SJqQ=', 'Content-Length': '0'} 2022-09-09T22:31:01.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bd0cc7042f8b624-00631bbea5-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.350 DEBUG:boto:b'jkcgbj-10&1000false&2022-09-09T22:30:44.219Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:01.352 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:01.353 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:01.353 DEBUG:boto:path=/jkcgbj-10/?prefix=%26 2022-09-09T22:31:01.354 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=%26 2022-09-09T22:31:01.354 DEBUG:boto:Method: GET 2022-09-09T22:31:01.355 DEBUG:boto:Path: /jkcgbj-10/?prefix=%26 2022-09-09T22:31:01.355 DEBUG:boto:Data: 2022-09-09T22:31:01.356 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.357 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:01.358 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.358 DEBUG:boto:Params: {} 2022-09-09T22:31:01.360 DEBUG:boto:Token: None 2022-09-09T22:31:01.360 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/ 2022-09-09T22:31:01.362 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2rXMYFsIHzm7bYFng+9WyP9SJqQ= 2022-09-09T22:31:01.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2rXMYFsIHzm7bYFng+9WyP9SJqQ=', 'Content-Length': '0'} 2022-09-09T22:31:01.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066a667a55325163a-00631bbea5-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.410 DEBUG:boto:b'jkcgbj-10&1000false&2022-09-09T22:30:44.219Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:01.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-09T22:31:01.412 DEBUG:boto:path=/jkcgbj-10/%3A 2022-09-09T22:31:01.413 DEBUG:boto:auth_path=/jkcgbj-10/%3A 2022-09-09T22:31:01.414 DEBUG:boto:path=/jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:01.415 DEBUG:boto:auth_path=/jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:01.416 DEBUG:boto:Method: GET 2022-09-09T22:31:01.417 DEBUG:boto:Path: /jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:01.418 DEBUG:boto:Data: 2022-09-09T22:31:01.419 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.420 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:31:01.421 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.423 DEBUG:boto:Params: {} 2022-09-09T22:31:01.424 DEBUG:boto:Token: None 2022-09-09T22:31:01.425 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:01.427 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JvWLkfLNZm1fWSNk0jlwLZbf0Ws= 2022-09-09T22:31:01.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JvWLkfLNZm1fWSNk0jlwLZbf0Ws=', 'Content-Length': '0'} 2022-09-09T22:31:01.432 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000048246496f9ffa93e-00631bbea5-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.434 DEBUG:boto:path=/jkcgbj-10/%3A 2022-09-09T22:31:01.434 DEBUG:boto:auth_path=/jkcgbj-10/%3A 2022-09-09T22:31:01.435 DEBUG:boto:path=/jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:01.437 DEBUG:boto:auth_path=/jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:01.438 DEBUG:boto:Method: GET 2022-09-09T22:31:01.439 DEBUG:boto:Path: /jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:01.440 DEBUG:boto:Data: 2022-09-09T22:31:01.441 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.442 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:01.443 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.444 DEBUG:boto:Params: {} 2022-09-09T22:31:01.445 DEBUG:boto:Token: None 2022-09-09T22:31:01.446 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:01.447 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JvWLkfLNZm1fWSNk0jlwLZbf0Ws= 2022-09-09T22:31:01.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JvWLkfLNZm1fWSNk0jlwLZbf0Ws=', 'Content-Length': '0'} 2022-09-09T22:31:01.453 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d5ea2cd1aaa3d69d-00631bbea5-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.454 DEBUG:boto:path=/jkcgbj-10/%3A 2022-09-09T22:31:01.455 DEBUG:boto:auth_path=/jkcgbj-10/%3A 2022-09-09T22:31:01.456 DEBUG:boto:path=/jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:01.458 DEBUG:boto:auth_path=/jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:01.459 DEBUG:boto:Method: HEAD 2022-09-09T22:31:01.460 DEBUG:boto:Path: /jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:01.460 DEBUG:boto:Data: 2022-09-09T22:31:01.461 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.462 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:31:01.463 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.464 DEBUG:boto:Params: {} 2022-09-09T22:31:01.465 DEBUG:boto:Token: None 2022-09-09T22:31:01.466 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:01.467 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:gvVeIltJWwhjesvUOzdZIrs5UHE= 2022-09-09T22:31:01.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:gvVeIltJWwhjesvUOzdZIrs5UHE=', 'Content-Length': '0'} 2022-09-09T22:31:01.481 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006693f41cdcb0d732-00631bbea5-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.482 DEBUG:boto:path=/jkcgbj-10/%3A 2022-09-09T22:31:01.483 DEBUG:boto:auth_path=/jkcgbj-10/%3A 2022-09-09T22:31:01.484 DEBUG:boto:path=/jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:01.485 DEBUG:boto:auth_path=/jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:01.486 DEBUG:boto:Method: HEAD 2022-09-09T22:31:01.487 DEBUG:boto:Path: /jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:01.487 DEBUG:boto:Data: 2022-09-09T22:31:01.488 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.490 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:01.491 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.491 DEBUG:boto:Params: {} 2022-09-09T22:31:01.492 DEBUG:boto:Token: None 2022-09-09T22:31:01.492 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:01.493 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:gvVeIltJWwhjesvUOzdZIrs5UHE= 2022-09-09T22:31:01.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:gvVeIltJWwhjesvUOzdZIrs5UHE=', 'Content-Length': '0'} 2022-09-09T22:31:01.497 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b1d223d7ef324cd-00631bbea5-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-09T22:31:01.498 DEBUG:boto:path=/jkcgbj-10/%3A 2022-09-09T22:31:01.499 DEBUG:boto:auth_path=/jkcgbj-10/%3A 2022-09-09T22:31:01.499 DEBUG:boto:path=/jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:01.500 DEBUG:boto:auth_path=/jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:01.500 DEBUG:boto:Method: GET 2022-09-09T22:31:01.501 DEBUG:boto:Path: /jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:01.502 DEBUG:boto:Data: 2022-09-09T22:31:01.502 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.503 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:31:01.503 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.504 DEBUG:boto:Params: {} 2022-09-09T22:31:01.505 DEBUG:boto:Token: None 2022-09-09T22:31:01.505 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:01.506 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JvWLkfLNZm1fWSNk0jlwLZbf0Ws= 2022-09-09T22:31:01.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JvWLkfLNZm1fWSNk0jlwLZbf0Ws=', 'Content-Length': '0'} 2022-09-09T22:31:01.522 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029ea5f378c799ad5-00631bbea5-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.524 DEBUG:boto:path=/jkcgbj-10/%3A 2022-09-09T22:31:01.534 DEBUG:boto:auth_path=/jkcgbj-10/%3A 2022-09-09T22:31:01.543 DEBUG:boto:path=/jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:01.545 DEBUG:boto:auth_path=/jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:01.546 DEBUG:boto:Method: GET 2022-09-09T22:31:01.547 DEBUG:boto:Path: /jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:01.548 DEBUG:boto:Data: 2022-09-09T22:31:01.550 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.567 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:01.568 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.569 DEBUG:boto:Params: {} 2022-09-09T22:31:01.570 DEBUG:boto:Token: None 2022-09-09T22:31:01.571 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:01.573 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JvWLkfLNZm1fWSNk0jlwLZbf0Ws= 2022-09-09T22:31:01.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JvWLkfLNZm1fWSNk0jlwLZbf0Ws=', 'Content-Length': '0'} 2022-09-09T22:31:01.578 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004901e19390fb009e-00631bbea5-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.579 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:01.581 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:01.582 DEBUG:boto:path=/jkcgbj-10/?prefix=%3A 2022-09-09T22:31:01.583 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=%3A 2022-09-09T22:31:01.584 DEBUG:boto:Method: GET 2022-09-09T22:31:01.584 DEBUG:boto:Path: /jkcgbj-10/?prefix=%3A 2022-09-09T22:31:01.585 DEBUG:boto:Data: 2022-09-09T22:31:01.586 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.587 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:31:01.588 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.589 DEBUG:boto:Params: {} 2022-09-09T22:31:01.591 DEBUG:boto:Token: None 2022-09-09T22:31:01.592 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/ 2022-09-09T22:31:01.594 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2rXMYFsIHzm7bYFng+9WyP9SJqQ= 2022-09-09T22:31:01.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2rXMYFsIHzm7bYFng+9WyP9SJqQ=', 'Content-Length': '0'} 2022-09-09T22:31:01.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4d8fa12eeb35ce5-00631bbea5-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.604 DEBUG:boto:b'jkcgbj-10:1000false:2022-09-09T22:30:44.155Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:01.605 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:01.606 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:01.607 DEBUG:boto:path=/jkcgbj-10/?prefix=%3A 2022-09-09T22:31:01.608 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=%3A 2022-09-09T22:31:01.609 DEBUG:boto:Method: GET 2022-09-09T22:31:01.610 DEBUG:boto:Path: /jkcgbj-10/?prefix=%3A 2022-09-09T22:31:01.611 DEBUG:boto:Data: 2022-09-09T22:31:01.611 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.612 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:01.613 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.630 DEBUG:boto:Params: {} 2022-09-09T22:31:01.632 DEBUG:boto:Token: None 2022-09-09T22:31:01.633 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/ 2022-09-09T22:31:01.634 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2rXMYFsIHzm7bYFng+9WyP9SJqQ= 2022-09-09T22:31:01.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2rXMYFsIHzm7bYFng+9WyP9SJqQ=', 'Content-Length': '0'} 2022-09-09T22:31:01.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004838972ed5b9cf9d-00631bbea5-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.648 DEBUG:boto:b'jkcgbj-10:1000false:2022-09-09T22:30:44.155Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:01.650 DEBUG:boto:path=/jkcgbj-10/%3A 2022-09-09T22:31:01.650 DEBUG:boto:auth_path=/jkcgbj-10/%3A 2022-09-09T22:31:01.651 DEBUG:boto:Method: HEAD 2022-09-09T22:31:01.651 DEBUG:boto:Path: /jkcgbj-10/%3A 2022-09-09T22:31:01.652 DEBUG:boto:Data: 2022-09-09T22:31:01.652 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.653 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:31:01.653 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.654 DEBUG:boto:Params: {} 2022-09-09T22:31:01.655 DEBUG:boto:Token: None 2022-09-09T22:31:01.655 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/%3A 2022-09-09T22:31:01.656 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:gxtIzfFx7eqAmf8SyNEx6CEpc1c= 2022-09-09T22:31:01.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:gxtIzfFx7eqAmf8SyNEx6CEpc1c=', 'Content-Length': '0'} 2022-09-09T22:31:01.660 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000059877a00b49929b8-00631bbea5-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.661 DEBUG:boto:path=/jkcgbj-10/%3A 2022-09-09T22:31:01.662 DEBUG:boto:auth_path=/jkcgbj-10/%3A 2022-09-09T22:31:01.662 DEBUG:boto:Method: HEAD 2022-09-09T22:31:01.663 DEBUG:boto:Path: /jkcgbj-10/%3A 2022-09-09T22:31:01.663 DEBUG:boto:Data: 2022-09-09T22:31:01.664 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.664 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:01.665 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.665 DEBUG:boto:Params: {} 2022-09-09T22:31:01.667 DEBUG:boto:Token: None 2022-09-09T22:31:01.668 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/%3A 2022-09-09T22:31:01.669 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:gxtIzfFx7eqAmf8SyNEx6CEpc1c= 2022-09-09T22:31:01.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:gxtIzfFx7eqAmf8SyNEx6CEpc1c=', 'Content-Length': '0'} 2022-09-09T22:31:01.683 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c4d3211cef05ff1-00631bbea5-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-09T22:31:01.686 DEBUG:boto:path=/jkcgbj-10/%3A 2022-09-09T22:31:01.687 DEBUG:boto:auth_path=/jkcgbj-10/%3A 2022-09-09T22:31:01.688 DEBUG:boto:Method: GET 2022-09-09T22:31:01.689 DEBUG:boto:Path: /jkcgbj-10/%3A 2022-09-09T22:31:01.690 DEBUG:boto:Data: 2022-09-09T22:31:01.691 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.692 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:31:01.693 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.694 DEBUG:boto:Params: {} 2022-09-09T22:31:01.695 DEBUG:boto:Token: None 2022-09-09T22:31:01.696 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/%3A 2022-09-09T22:31:01.698 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:N72JxwLtsKbv8UpA0cJqnC1WVxo= 2022-09-09T22:31:01.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:N72JxwLtsKbv8UpA0cJqnC1WVxo=', 'Content-Length': '0'} 2022-09-09T22:31:01.709 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0ed5c429ad67bfc-00631bbea5-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.710 DEBUG:boto:path=/jkcgbj-10/%3A 2022-09-09T22:31:01.711 DEBUG:boto:auth_path=/jkcgbj-10/%3A 2022-09-09T22:31:01.711 DEBUG:boto:Method: GET 2022-09-09T22:31:01.712 DEBUG:boto:Path: /jkcgbj-10/%3A 2022-09-09T22:31:01.712 DEBUG:boto:Data: 2022-09-09T22:31:01.713 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.713 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:01.714 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.714 DEBUG:boto:Params: {} 2022-09-09T22:31:01.715 DEBUG:boto:Token: None 2022-09-09T22:31:01.716 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/%3A 2022-09-09T22:31:01.716 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:N72JxwLtsKbv8UpA0cJqnC1WVxo= 2022-09-09T22:31:01.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:N72JxwLtsKbv8UpA0cJqnC1WVxo=', 'Content-Length': '0'} 2022-09-09T22:31:01.735 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b79eb5ab4aeabac-00631bbea5-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.737 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:01.738 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:01.739 DEBUG:boto:path=/jkcgbj-10/?prefix=%3A 2022-09-09T22:31:01.740 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=%3A 2022-09-09T22:31:01.741 DEBUG:boto:Method: GET 2022-09-09T22:31:01.742 DEBUG:boto:Path: /jkcgbj-10/?prefix=%3A 2022-09-09T22:31:01.743 DEBUG:boto:Data: 2022-09-09T22:31:01.744 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.745 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:31:01.747 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.748 DEBUG:boto:Params: {} 2022-09-09T22:31:01.750 DEBUG:boto:Token: None 2022-09-09T22:31:01.751 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/ 2022-09-09T22:31:01.752 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2rXMYFsIHzm7bYFng+9WyP9SJqQ= 2022-09-09T22:31:01.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2rXMYFsIHzm7bYFng+9WyP9SJqQ=', 'Content-Length': '0'} 2022-09-09T22:31:01.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057afa761bb6ee8de-00631bbea5-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.767 DEBUG:boto:b'jkcgbj-10:1000false:2022-09-09T22:30:44.155Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:01.769 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:01.770 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:01.771 DEBUG:boto:path=/jkcgbj-10/?prefix=%3A 2022-09-09T22:31:01.772 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=%3A 2022-09-09T22:31:01.774 DEBUG:boto:Method: GET 2022-09-09T22:31:01.775 DEBUG:boto:Path: /jkcgbj-10/?prefix=%3A 2022-09-09T22:31:01.779 DEBUG:boto:Data: 2022-09-09T22:31:01.780 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.780 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:01.781 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.781 DEBUG:boto:Params: {} 2022-09-09T22:31:01.783 DEBUG:boto:Token: None 2022-09-09T22:31:01.785 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/ 2022-09-09T22:31:01.786 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2rXMYFsIHzm7bYFng+9WyP9SJqQ= 2022-09-09T22:31:01.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2rXMYFsIHzm7bYFng+9WyP9SJqQ=', 'Content-Length': '0'} 2022-09-09T22:31:01.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1b881f889db22f0-00631bbea5-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.794 DEBUG:boto:b'jkcgbj-10:1000false:2022-09-09T22:30:44.155Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:01.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-09T22:31:01.813 DEBUG:boto:path=/jkcgbj-10/_myobj 2022-09-09T22:31:01.814 DEBUG:boto:auth_path=/jkcgbj-10/_myobj 2022-09-09T22:31:01.815 DEBUG:boto:path=/jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:01.816 DEBUG:boto:auth_path=/jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:01.817 DEBUG:boto:Method: GET 2022-09-09T22:31:01.818 DEBUG:boto:Path: /jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:01.819 DEBUG:boto:Data: 2022-09-09T22:31:01.820 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.821 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:31:01.822 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.823 DEBUG:boto:Params: {} 2022-09-09T22:31:01.824 DEBUG:boto:Token: None 2022-09-09T22:31:01.825 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:01.827 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:29qBhaaM/uti1RQfZcOFn9FfA/4= 2022-09-09T22:31:01.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:29qBhaaM/uti1RQfZcOFn9FfA/4=', 'Content-Length': '0'} 2022-09-09T22:31:01.832 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d199c523c657cfe4-00631bbea5-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.833 DEBUG:boto:path=/jkcgbj-10/_myobj 2022-09-09T22:31:01.835 DEBUG:boto:auth_path=/jkcgbj-10/_myobj 2022-09-09T22:31:01.836 DEBUG:boto:path=/jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:01.837 DEBUG:boto:auth_path=/jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:01.838 DEBUG:boto:Method: GET 2022-09-09T22:31:01.839 DEBUG:boto:Path: /jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:01.840 DEBUG:boto:Data: 2022-09-09T22:31:01.841 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.842 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:01.843 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.844 DEBUG:boto:Params: {} 2022-09-09T22:31:01.845 DEBUG:boto:Token: None 2022-09-09T22:31:01.846 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:01.848 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:29qBhaaM/uti1RQfZcOFn9FfA/4= 2022-09-09T22:31:01.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:29qBhaaM/uti1RQfZcOFn9FfA/4=', 'Content-Length': '0'} 2022-09-09T22:31:01.853 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dfe4e305d85927e1-00631bbea5-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.855 DEBUG:boto:path=/jkcgbj-10/_myobj 2022-09-09T22:31:01.856 DEBUG:boto:auth_path=/jkcgbj-10/_myobj 2022-09-09T22:31:01.858 DEBUG:boto:path=/jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:01.859 DEBUG:boto:auth_path=/jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:01.860 DEBUG:boto:Method: HEAD 2022-09-09T22:31:01.860 DEBUG:boto:Path: /jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:01.862 DEBUG:boto:Data: 2022-09-09T22:31:01.863 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.864 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:31:01.865 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.866 DEBUG:boto:Params: {} 2022-09-09T22:31:01.867 DEBUG:boto:Token: None 2022-09-09T22:31:01.868 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:01.869 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vOe2geOUePS+aUn5foA4W/ljr2I= 2022-09-09T22:31:01.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vOe2geOUePS+aUn5foA4W/ljr2I=', 'Content-Length': '0'} 2022-09-09T22:31:01.876 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000042233ca6a63a0610-00631bbea5-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.877 DEBUG:boto:path=/jkcgbj-10/_myobj 2022-09-09T22:31:01.878 DEBUG:boto:auth_path=/jkcgbj-10/_myobj 2022-09-09T22:31:01.879 DEBUG:boto:path=/jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:01.880 DEBUG:boto:auth_path=/jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:01.881 DEBUG:boto:Method: HEAD 2022-09-09T22:31:01.882 DEBUG:boto:Path: /jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:01.882 DEBUG:boto:Data: 2022-09-09T22:31:01.884 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.890 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:01.892 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.893 DEBUG:boto:Params: {} 2022-09-09T22:31:01.894 DEBUG:boto:Token: None 2022-09-09T22:31:01.895 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:01.897 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vOe2geOUePS+aUn5foA4W/ljr2I= 2022-09-09T22:31:01.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vOe2geOUePS+aUn5foA4W/ljr2I=', 'Content-Length': '0'} 2022-09-09T22:31:01.903 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d2b2dba3ee1bd204-00631bbea5-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-09T22:31:01.904 DEBUG:boto:path=/jkcgbj-10/_myobj 2022-09-09T22:31:01.905 DEBUG:boto:auth_path=/jkcgbj-10/_myobj 2022-09-09T22:31:01.905 DEBUG:boto:path=/jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:01.907 DEBUG:boto:auth_path=/jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:01.908 DEBUG:boto:Method: GET 2022-09-09T22:31:01.908 DEBUG:boto:Path: /jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:01.909 DEBUG:boto:Data: 2022-09-09T22:31:01.910 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.953 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:31:01.955 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.956 DEBUG:boto:Params: {} 2022-09-09T22:31:01.957 DEBUG:boto:Token: None 2022-09-09T22:31:01.958 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:01.960 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:29qBhaaM/uti1RQfZcOFn9FfA/4= 2022-09-09T22:31:01.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:29qBhaaM/uti1RQfZcOFn9FfA/4=', 'Content-Length': '0'} 2022-09-09T22:31:01.964 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015951b8d0a33758b-00631bbea5-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.966 DEBUG:boto:path=/jkcgbj-10/_myobj 2022-09-09T22:31:01.966 DEBUG:boto:auth_path=/jkcgbj-10/_myobj 2022-09-09T22:31:01.967 DEBUG:boto:path=/jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:01.967 DEBUG:boto:auth_path=/jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:01.968 DEBUG:boto:Method: GET 2022-09-09T22:31:01.968 DEBUG:boto:Path: /jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:01.969 DEBUG:boto:Data: 2022-09-09T22:31:01.970 DEBUG:boto:Headers: {} 2022-09-09T22:31:01.970 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:01.971 DEBUG:boto:Port: 8000 2022-09-09T22:31:01.971 DEBUG:boto:Params: {} 2022-09-09T22:31:01.972 DEBUG:boto:Token: None 2022-09-09T22:31:01.973 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:01 GMT /jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:01.973 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:29qBhaaM/uti1RQfZcOFn9FfA/4= 2022-09-09T22:31:01.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:29qBhaaM/uti1RQfZcOFn9FfA/4=', 'Content-Length': '0'} 2022-09-09T22:31:01.977 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b7c1fa756163789-00631bbea5-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:01.979 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:01.979 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:02.017 DEBUG:boto:path=/jkcgbj-10/?prefix=_myobj 2022-09-09T22:31:02.018 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=_myobj 2022-09-09T22:31:02.018 DEBUG:boto:Method: GET 2022-09-09T22:31:02.019 DEBUG:boto:Path: /jkcgbj-10/?prefix=_myobj 2022-09-09T22:31:02.019 DEBUG:boto:Data: 2022-09-09T22:31:02.020 DEBUG:boto:Headers: {} 2022-09-09T22:31:02.020 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:31:02.021 DEBUG:boto:Port: 8000 2022-09-09T22:31:02.021 DEBUG:boto:Params: {} 2022-09-09T22:31:02.023 DEBUG:boto:Token: None 2022-09-09T22:31:02.024 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:02 GMT /jkcgbj-10/ 2022-09-09T22:31:02.025 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LOGplgfpS50JI5CPYwu0MjDHnxg= 2022-09-09T22:31:02.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LOGplgfpS50JI5CPYwu0MjDHnxg=', 'Content-Length': '0'} 2022-09-09T22:31:02.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065b3247c1175b067-00631bbea6-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:02.039 DEBUG:boto:b'jkcgbj-10_myobj1000false_myobj2022-09-09T22:30:44.054Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:02.050 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:02.051 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:02.052 DEBUG:boto:path=/jkcgbj-10/?prefix=_myobj 2022-09-09T22:31:02.053 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=_myobj 2022-09-09T22:31:02.058 DEBUG:boto:Method: GET 2022-09-09T22:31:02.059 DEBUG:boto:Path: /jkcgbj-10/?prefix=_myobj 2022-09-09T22:31:02.060 DEBUG:boto:Data: 2022-09-09T22:31:02.061 DEBUG:boto:Headers: {} 2022-09-09T22:31:02.062 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:02.063 DEBUG:boto:Port: 8000 2022-09-09T22:31:02.065 DEBUG:boto:Params: {} 2022-09-09T22:31:02.069 DEBUG:boto:Token: None 2022-09-09T22:31:02.071 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:02 GMT /jkcgbj-10/ 2022-09-09T22:31:02.071 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LOGplgfpS50JI5CPYwu0MjDHnxg= 2022-09-09T22:31:02.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LOGplgfpS50JI5CPYwu0MjDHnxg=', 'Content-Length': '0'} 2022-09-09T22:31:02.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c50489e5c5e3a76a-00631bbea6-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:02.079 DEBUG:boto:b'jkcgbj-10_myobj1000false_myobj2022-09-09T22:30:44.054Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:02.080 DEBUG:boto:path=/jkcgbj-10/_myobj 2022-09-09T22:31:02.080 DEBUG:boto:auth_path=/jkcgbj-10/_myobj 2022-09-09T22:31:02.081 DEBUG:boto:Method: HEAD 2022-09-09T22:31:02.083 DEBUG:boto:Path: /jkcgbj-10/_myobj 2022-09-09T22:31:02.084 DEBUG:boto:Data: 2022-09-09T22:31:02.085 DEBUG:boto:Headers: {} 2022-09-09T22:31:02.085 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:31:02.086 DEBUG:boto:Port: 8000 2022-09-09T22:31:02.086 DEBUG:boto:Params: {} 2022-09-09T22:31:02.087 DEBUG:boto:Token: None 2022-09-09T22:31:02.098 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:02 GMT /jkcgbj-10/_myobj 2022-09-09T22:31:02.099 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:XSQd8CbyWA28Jsv/ByDe4ljrwKY= 2022-09-09T22:31:02.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:XSQd8CbyWA28Jsv/ByDe4ljrwKY=', 'Content-Length': '0'} 2022-09-09T22:31:02.106 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eabfb0e072efc552-00631bbea6-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:02.107 DEBUG:boto:path=/jkcgbj-10/_myobj 2022-09-09T22:31:02.110 DEBUG:boto:auth_path=/jkcgbj-10/_myobj 2022-09-09T22:31:02.110 DEBUG:boto:Method: HEAD 2022-09-09T22:31:02.111 DEBUG:boto:Path: /jkcgbj-10/_myobj 2022-09-09T22:31:02.113 DEBUG:boto:Data: 2022-09-09T22:31:02.114 DEBUG:boto:Headers: {} 2022-09-09T22:31:02.114 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:02.115 DEBUG:boto:Port: 8000 2022-09-09T22:31:02.115 DEBUG:boto:Params: {} 2022-09-09T22:31:02.120 DEBUG:boto:Token: None 2022-09-09T22:31:02.120 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:02 GMT /jkcgbj-10/_myobj 2022-09-09T22:31:02.132 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:XSQd8CbyWA28Jsv/ByDe4ljrwKY= 2022-09-09T22:31:02.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:XSQd8CbyWA28Jsv/ByDe4ljrwKY=', 'Content-Length': '0'} 2022-09-09T22:31:02.156 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a86ec2115d570fc-00631bbea6-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:02.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-09T22:31:02.161 DEBUG:boto:path=/jkcgbj-10/_myobj 2022-09-09T22:31:02.161 DEBUG:boto:auth_path=/jkcgbj-10/_myobj 2022-09-09T22:31:02.162 DEBUG:boto:Method: GET 2022-09-09T22:31:02.162 DEBUG:boto:Path: /jkcgbj-10/_myobj 2022-09-09T22:31:02.163 DEBUG:boto:Data: 2022-09-09T22:31:02.164 DEBUG:boto:Headers: {} 2022-09-09T22:31:02.164 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:31:02.165 DEBUG:boto:Port: 8000 2022-09-09T22:31:02.166 DEBUG:boto:Params: {} 2022-09-09T22:31:02.167 DEBUG:boto:Token: None 2022-09-09T22:31:02.168 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:02 GMT /jkcgbj-10/_myobj 2022-09-09T22:31:02.169 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Oc0lHb6sCpSkZ0m3Ct8Hf7NeOr0= 2022-09-09T22:31:02.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Oc0lHb6sCpSkZ0m3Ct8Hf7NeOr0=', 'Content-Length': '0'} 2022-09-09T22:31:02.174 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b284bc7a86d3ef36-00631bbea6-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:02.175 DEBUG:boto:path=/jkcgbj-10/_myobj 2022-09-09T22:31:02.176 DEBUG:boto:auth_path=/jkcgbj-10/_myobj 2022-09-09T22:31:02.176 DEBUG:boto:Method: GET 2022-09-09T22:31:02.177 DEBUG:boto:Path: /jkcgbj-10/_myobj 2022-09-09T22:31:02.177 DEBUG:boto:Data: 2022-09-09T22:31:02.179 DEBUG:boto:Headers: {} 2022-09-09T22:31:02.182 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:02.182 DEBUG:boto:Port: 8000 2022-09-09T22:31:02.183 DEBUG:boto:Params: {} 2022-09-09T22:31:02.183 DEBUG:boto:Token: None 2022-09-09T22:31:02.184 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:02 GMT /jkcgbj-10/_myobj 2022-09-09T22:31:02.185 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Oc0lHb6sCpSkZ0m3Ct8Hf7NeOr0= 2022-09-09T22:31:02.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Oc0lHb6sCpSkZ0m3Ct8Hf7NeOr0=', 'Content-Length': '0'} 2022-09-09T22:31:02.196 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c3160ca6aceff32-00631bbea6-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:02.198 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:02.199 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:02.200 DEBUG:boto:path=/jkcgbj-10/?prefix=_myobj 2022-09-09T22:31:02.201 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=_myobj 2022-09-09T22:31:02.202 DEBUG:boto:Method: GET 2022-09-09T22:31:02.203 DEBUG:boto:Path: /jkcgbj-10/?prefix=_myobj 2022-09-09T22:31:02.204 DEBUG:boto:Data: 2022-09-09T22:31:02.218 DEBUG:boto:Headers: {} 2022-09-09T22:31:02.219 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:31:02.220 DEBUG:boto:Port: 8000 2022-09-09T22:31:02.221 DEBUG:boto:Params: {} 2022-09-09T22:31:02.222 DEBUG:boto:Token: None 2022-09-09T22:31:02.223 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:02 GMT /jkcgbj-10/ 2022-09-09T22:31:02.224 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LOGplgfpS50JI5CPYwu0MjDHnxg= 2022-09-09T22:31:02.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LOGplgfpS50JI5CPYwu0MjDHnxg=', 'Content-Length': '0'} 2022-09-09T22:31:02.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f549cd67ab7cd59-00631bbea6-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:02.232 DEBUG:boto:b'jkcgbj-10_myobj1000false_myobj2022-09-09T22:30:44.054Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:02.234 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:02.235 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:02.236 DEBUG:boto:path=/jkcgbj-10/?prefix=_myobj 2022-09-09T22:31:02.237 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=_myobj 2022-09-09T22:31:02.238 DEBUG:boto:Method: GET 2022-09-09T22:31:02.240 DEBUG:boto:Path: /jkcgbj-10/?prefix=_myobj 2022-09-09T22:31:02.241 DEBUG:boto:Data: 2022-09-09T22:31:02.242 DEBUG:boto:Headers: {} 2022-09-09T22:31:02.244 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:02.245 DEBUG:boto:Port: 8000 2022-09-09T22:31:02.257 DEBUG:boto:Params: {} 2022-09-09T22:31:02.258 DEBUG:boto:Token: None 2022-09-09T22:31:02.259 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:02 GMT /jkcgbj-10/ 2022-09-09T22:31:02.260 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LOGplgfpS50JI5CPYwu0MjDHnxg= 2022-09-09T22:31:02.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LOGplgfpS50JI5CPYwu0MjDHnxg=', 'Content-Length': '0'} 2022-09-09T22:31:02.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9c33329759ec869-00631bbea6-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:02.276 DEBUG:boto:b'jkcgbj-10_myobj1000false_myobj2022-09-09T22:30:44.054Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:02.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:31:02.278 DEBUG:boto:path=/jkcgbj-10/myobj 2022-09-09T22:31:02.279 DEBUG:boto:auth_path=/jkcgbj-10/myobj 2022-09-09T22:31:02.280 DEBUG:boto:path=/jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:02.281 DEBUG:boto:auth_path=/jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:02.282 DEBUG:boto:Method: GET 2022-09-09T22:31:02.282 DEBUG:boto:Path: /jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:02.283 DEBUG:boto:Data: 2022-09-09T22:31:02.284 DEBUG:boto:Headers: {} 2022-09-09T22:31:02.286 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:31:02.287 DEBUG:boto:Port: 8000 2022-09-09T22:31:02.288 DEBUG:boto:Params: {} 2022-09-09T22:31:02.289 DEBUG:boto:Token: None 2022-09-09T22:31:02.291 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:02 GMT /jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:02.292 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TN3o8iWol6GY7rOH8gu/SJwZWnQ= 2022-09-09T22:31:02.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TN3o8iWol6GY7rOH8gu/SJwZWnQ=', 'Content-Length': '0'} 2022-09-09T22:31:02.297 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c225c9a5f51f51f8-00631bbea6-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:02.299 DEBUG:boto:path=/jkcgbj-10/myobj 2022-09-09T22:31:02.300 DEBUG:boto:auth_path=/jkcgbj-10/myobj 2022-09-09T22:31:02.302 DEBUG:boto:path=/jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:02.303 DEBUG:boto:auth_path=/jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:02.304 DEBUG:boto:Method: GET 2022-09-09T22:31:02.305 DEBUG:boto:Path: /jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:02.306 DEBUG:boto:Data: 2022-09-09T22:31:02.308 DEBUG:boto:Headers: {} 2022-09-09T22:31:02.309 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:02.310 DEBUG:boto:Port: 8000 2022-09-09T22:31:02.311 DEBUG:boto:Params: {} 2022-09-09T22:31:02.312 DEBUG:boto:Token: None 2022-09-09T22:31:02.313 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:02 GMT /jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:02.315 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TN3o8iWol6GY7rOH8gu/SJwZWnQ= 2022-09-09T22:31:02.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TN3o8iWol6GY7rOH8gu/SJwZWnQ=', 'Content-Length': '0'} 2022-09-09T22:31:02.319 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000092b733b1286ae632-00631bbea6-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:02.321 DEBUG:boto:path=/jkcgbj-10/myobj 2022-09-09T22:31:02.322 DEBUG:boto:auth_path=/jkcgbj-10/myobj 2022-09-09T22:31:02.324 DEBUG:boto:path=/jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:02.325 DEBUG:boto:auth_path=/jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:02.326 DEBUG:boto:Method: HEAD 2022-09-09T22:31:02.328 DEBUG:boto:Path: /jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:02.329 DEBUG:boto:Data: 2022-09-09T22:31:02.330 DEBUG:boto:Headers: {} 2022-09-09T22:31:02.330 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:31:02.331 DEBUG:boto:Port: 8000 2022-09-09T22:31:02.331 DEBUG:boto:Params: {} 2022-09-09T22:31:02.332 DEBUG:boto:Token: None 2022-09-09T22:31:02.332 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:02 GMT /jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:02.333 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9V4HW+wqRzziq2D5OEtLxg0NJUk= 2022-09-09T22:31:02.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9V4HW+wqRzziq2D5OEtLxg0NJUk=', 'Content-Length': '0'} 2022-09-09T22:31:02.337 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c1d4119f8c77ac7-00631bbea6-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:02.338 DEBUG:boto:path=/jkcgbj-10/myobj 2022-09-09T22:31:02.338 DEBUG:boto:auth_path=/jkcgbj-10/myobj 2022-09-09T22:31:02.339 DEBUG:boto:path=/jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:02.339 DEBUG:boto:auth_path=/jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:02.340 DEBUG:boto:Method: HEAD 2022-09-09T22:31:02.340 DEBUG:boto:Path: /jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:02.341 DEBUG:boto:Data: 2022-09-09T22:31:02.341 DEBUG:boto:Headers: {} 2022-09-09T22:31:02.342 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:02.343 DEBUG:boto:Port: 8000 2022-09-09T22:31:02.344 DEBUG:boto:Params: {} 2022-09-09T22:31:02.344 DEBUG:boto:Token: None 2022-09-09T22:31:02.345 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:02 GMT /jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:02.345 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9V4HW+wqRzziq2D5OEtLxg0NJUk= 2022-09-09T22:31:02.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9V4HW+wqRzziq2D5OEtLxg0NJUk=', 'Content-Length': '0'} 2022-09-09T22:31:02.350 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eda6b2d4ffb67679-00631bbea6-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:02.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:31:02.352 DEBUG:boto:path=/jkcgbj-10/myobj 2022-09-09T22:31:02.352 DEBUG:boto:auth_path=/jkcgbj-10/myobj 2022-09-09T22:31:02.353 DEBUG:boto:path=/jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:02.353 DEBUG:boto:auth_path=/jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:02.354 DEBUG:boto:Method: GET 2022-09-09T22:31:02.355 DEBUG:boto:Path: /jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:02.355 DEBUG:boto:Data: 2022-09-09T22:31:02.356 DEBUG:boto:Headers: {} 2022-09-09T22:31:02.356 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:31:02.358 DEBUG:boto:Port: 8000 2022-09-09T22:31:02.358 DEBUG:boto:Params: {} 2022-09-09T22:31:02.359 DEBUG:boto:Token: None 2022-09-09T22:31:02.359 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:02 GMT /jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:02.360 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TN3o8iWol6GY7rOH8gu/SJwZWnQ= 2022-09-09T22:31:02.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TN3o8iWol6GY7rOH8gu/SJwZWnQ=', 'Content-Length': '0'} 2022-09-09T22:31:02.365 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000347604a60e7f0317-00631bbea6-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:02.366 DEBUG:boto:path=/jkcgbj-10/myobj 2022-09-09T22:31:02.367 DEBUG:boto:auth_path=/jkcgbj-10/myobj 2022-09-09T22:31:02.368 DEBUG:boto:path=/jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:02.369 DEBUG:boto:auth_path=/jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:02.369 DEBUG:boto:Method: GET 2022-09-09T22:31:02.370 DEBUG:boto:Path: /jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:02.370 DEBUG:boto:Data: 2022-09-09T22:31:02.371 DEBUG:boto:Headers: {} 2022-09-09T22:31:02.371 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:02.372 DEBUG:boto:Port: 8000 2022-09-09T22:31:02.373 DEBUG:boto:Params: {} 2022-09-09T22:31:02.374 DEBUG:boto:Token: None 2022-09-09T22:31:02.375 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:02 GMT /jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:02.377 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TN3o8iWol6GY7rOH8gu/SJwZWnQ= 2022-09-09T22:31:02.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TN3o8iWol6GY7rOH8gu/SJwZWnQ=', 'Content-Length': '0'} 2022-09-09T22:31:02.381 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b28c1827ccde37e5-00631bbea6-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:02.426 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:02.427 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:02.428 DEBUG:boto:path=/jkcgbj-10/?prefix=myobj 2022-09-09T22:31:02.428 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=myobj 2022-09-09T22:31:02.429 DEBUG:boto:Method: GET 2022-09-09T22:31:02.430 DEBUG:boto:Path: /jkcgbj-10/?prefix=myobj 2022-09-09T22:31:02.430 DEBUG:boto:Data: 2022-09-09T22:31:02.431 DEBUG:boto:Headers: {} 2022-09-09T22:31:02.431 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:31:02.432 DEBUG:boto:Port: 8000 2022-09-09T22:31:02.433 DEBUG:boto:Params: {} 2022-09-09T22:31:02.433 DEBUG:boto:Token: None 2022-09-09T22:31:02.434 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:02 GMT /jkcgbj-10/ 2022-09-09T22:31:02.435 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LOGplgfpS50JI5CPYwu0MjDHnxg= 2022-09-09T22:31:02.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LOGplgfpS50JI5CPYwu0MjDHnxg=', 'Content-Length': '0'} 2022-09-09T22:31:02.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001377313f9ea81078-00631bbea6-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:02.443 DEBUG:boto:b'jkcgbj-10myobj1000falsemyobj2022-09-09T22:30:43.986Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:02.446 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:02.447 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:02.447 DEBUG:boto:path=/jkcgbj-10/?prefix=myobj 2022-09-09T22:31:02.448 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=myobj 2022-09-09T22:31:02.448 DEBUG:boto:Method: GET 2022-09-09T22:31:02.449 DEBUG:boto:Path: /jkcgbj-10/?prefix=myobj 2022-09-09T22:31:02.450 DEBUG:boto:Data: 2022-09-09T22:31:02.451 DEBUG:boto:Headers: {} 2022-09-09T22:31:02.452 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:02.453 DEBUG:boto:Port: 8000 2022-09-09T22:31:02.453 DEBUG:boto:Params: {} 2022-09-09T22:31:02.454 DEBUG:boto:Token: None 2022-09-09T22:31:02.455 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:02 GMT /jkcgbj-10/ 2022-09-09T22:31:02.455 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LOGplgfpS50JI5CPYwu0MjDHnxg= 2022-09-09T22:31:02.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LOGplgfpS50JI5CPYwu0MjDHnxg=', 'Content-Length': '0'} 2022-09-09T22:31:02.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d19d09bf7365375a-00631bbea6-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:02.489 DEBUG:boto:b'jkcgbj-10myobj1000falsemyobj2022-09-09T22:30:43.986Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:02.523 DEBUG:boto:path=/jkcgbj-10/myobj 2022-09-09T22:31:02.551 DEBUG:boto:auth_path=/jkcgbj-10/myobj 2022-09-09T22:31:02.552 DEBUG:boto:Method: HEAD 2022-09-09T22:31:02.553 DEBUG:boto:Path: /jkcgbj-10/myobj 2022-09-09T22:31:02.554 DEBUG:boto:Data: 2022-09-09T22:31:02.555 DEBUG:boto:Headers: {} 2022-09-09T22:31:02.556 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:31:02.563 DEBUG:boto:Port: 8000 2022-09-09T22:31:02.563 DEBUG:boto:Params: {} 2022-09-09T22:31:02.565 DEBUG:boto:Token: None 2022-09-09T22:31:02.575 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:02 GMT /jkcgbj-10/myobj 2022-09-09T22:31:02.576 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pe/nN4qnATXcI/3h5JwXRawRXbY= 2022-09-09T22:31:02.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pe/nN4qnATXcI/3h5JwXRawRXbY=', 'Content-Length': '0'} 2022-09-09T22:31:02.581 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b95a02cea3778826-00631bbea6-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:02.582 DEBUG:boto:path=/jkcgbj-10/myobj 2022-09-09T22:31:02.583 DEBUG:boto:auth_path=/jkcgbj-10/myobj 2022-09-09T22:31:02.583 DEBUG:boto:Method: HEAD 2022-09-09T22:31:02.584 DEBUG:boto:Path: /jkcgbj-10/myobj 2022-09-09T22:31:02.584 DEBUG:boto:Data: 2022-09-09T22:31:02.585 DEBUG:boto:Headers: {} 2022-09-09T22:31:02.585 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:02.587 DEBUG:boto:Port: 8000 2022-09-09T22:31:02.588 DEBUG:boto:Params: {} 2022-09-09T22:31:02.589 DEBUG:boto:Token: None 2022-09-09T22:31:02.590 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:02 GMT /jkcgbj-10/myobj 2022-09-09T22:31:02.590 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pe/nN4qnATXcI/3h5JwXRawRXbY= 2022-09-09T22:31:02.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pe/nN4qnATXcI/3h5JwXRawRXbY=', 'Content-Length': '0'} 2022-09-09T22:31:02.594 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf43c7e718e662dc-00631bbea6-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:02.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:31:02.595 DEBUG:boto:path=/jkcgbj-10/myobj 2022-09-09T22:31:02.596 DEBUG:boto:auth_path=/jkcgbj-10/myobj 2022-09-09T22:31:02.597 DEBUG:boto:Method: GET 2022-09-09T22:31:02.597 DEBUG:boto:Path: /jkcgbj-10/myobj 2022-09-09T22:31:02.598 DEBUG:boto:Data: 2022-09-09T22:31:02.607 DEBUG:boto:Headers: {} 2022-09-09T22:31:02.607 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:31:02.608 DEBUG:boto:Port: 8000 2022-09-09T22:31:02.608 DEBUG:boto:Params: {} 2022-09-09T22:31:02.618 DEBUG:boto:Token: None 2022-09-09T22:31:02.620 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:02 GMT /jkcgbj-10/myobj 2022-09-09T22:31:02.621 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:S1ZrcbpzCWhXs7xxVp8L5K5Z7Sk= 2022-09-09T22:31:02.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:S1ZrcbpzCWhXs7xxVp8L5K5Z7Sk=', 'Content-Length': '0'} 2022-09-09T22:31:02.629 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b1f33be2e7f7bfa-00631bbea6-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:02.631 DEBUG:boto:path=/jkcgbj-10/myobj 2022-09-09T22:31:02.632 DEBUG:boto:auth_path=/jkcgbj-10/myobj 2022-09-09T22:31:02.633 DEBUG:boto:Method: GET 2022-09-09T22:31:02.634 DEBUG:boto:Path: /jkcgbj-10/myobj 2022-09-09T22:31:02.635 DEBUG:boto:Data: 2022-09-09T22:31:02.637 DEBUG:boto:Headers: {} 2022-09-09T22:31:02.638 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:02.639 DEBUG:boto:Port: 8000 2022-09-09T22:31:02.640 DEBUG:boto:Params: {} 2022-09-09T22:31:02.641 DEBUG:boto:Token: None 2022-09-09T22:31:02.643 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:02 GMT /jkcgbj-10/myobj 2022-09-09T22:31:02.644 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:S1ZrcbpzCWhXs7xxVp8L5K5Z7Sk= 2022-09-09T22:31:02.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:S1ZrcbpzCWhXs7xxVp8L5K5Z7Sk=', 'Content-Length': '0'} 2022-09-09T22:31:02.649 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000bd52fdd031e3aab-00631bbea6-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:02.651 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:02.651 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:02.652 DEBUG:boto:path=/jkcgbj-10/?prefix=myobj 2022-09-09T22:31:02.653 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=myobj 2022-09-09T22:31:02.653 DEBUG:boto:Method: GET 2022-09-09T22:31:02.654 DEBUG:boto:Path: /jkcgbj-10/?prefix=myobj 2022-09-09T22:31:02.655 DEBUG:boto:Data: 2022-09-09T22:31:02.656 DEBUG:boto:Headers: {} 2022-09-09T22:31:02.657 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:31:02.662 DEBUG:boto:Port: 8000 2022-09-09T22:31:02.663 DEBUG:boto:Params: {} 2022-09-09T22:31:02.664 DEBUG:boto:Token: None 2022-09-09T22:31:02.664 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:02 GMT /jkcgbj-10/ 2022-09-09T22:31:02.665 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LOGplgfpS50JI5CPYwu0MjDHnxg= 2022-09-09T22:31:02.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LOGplgfpS50JI5CPYwu0MjDHnxg=', 'Content-Length': '0'} 2022-09-09T22:31:02.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eefa71a83c9199d3-00631bbea6-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:02.680 DEBUG:boto:b'jkcgbj-10myobj1000falsemyobj2022-09-09T22:30:43.986Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:02.682 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:02.684 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:02.685 DEBUG:boto:path=/jkcgbj-10/?prefix=myobj 2022-09-09T22:31:02.686 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=myobj 2022-09-09T22:31:02.687 DEBUG:boto:Method: GET 2022-09-09T22:31:02.688 DEBUG:boto:Path: /jkcgbj-10/?prefix=myobj 2022-09-09T22:31:02.689 DEBUG:boto:Data: 2022-09-09T22:31:02.690 DEBUG:boto:Headers: {} 2022-09-09T22:31:02.691 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:02.692 DEBUG:boto:Port: 8000 2022-09-09T22:31:02.693 DEBUG:boto:Params: {} 2022-09-09T22:31:02.694 DEBUG:boto:Token: None 2022-09-09T22:31:02.695 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:02 GMT /jkcgbj-10/ 2022-09-09T22:31:02.696 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LOGplgfpS50JI5CPYwu0MjDHnxg= 2022-09-09T22:31:02.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LOGplgfpS50JI5CPYwu0MjDHnxg=', 'Content-Length': '0'} 2022-09-09T22:31:02.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092e820f1bff7be2f-00631bbea6-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:02.706 DEBUG:boto:b'jkcgbj-10myobj1000falsemyobj2022-09-09T22:30:43.986Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:02.708 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-10 zones={test-zone2, test-zone1} 2022-09-09T22:31:02.709 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:31:02.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:31:02.711 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:31:02.764 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:31:02.765 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:31:02.830 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:31:02.831+0000 7f2165fa8dc0 1 bucket sync caught up with source: 2022-09-09T22:31:02.830 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000001.6.5, , 00000000001.4.5, , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:31:02.830 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000001.6.5, , 00000000001.4.5, , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:31:02.831 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:31:02.831+0000 7f2165fa8dc0 0 bucket checkpoint complete 2022-09-09T22:31:02.854 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-10 zones={test-zone3, test-zone1} 2022-09-09T22:31:02.855 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:02.855 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:02.856 DEBUG:boto:Method: HEAD 2022-09-09T22:31:02.856 DEBUG:boto:Path: /jkcgbj-10/ 2022-09-09T22:31:02.857 DEBUG:boto:Data: 2022-09-09T22:31:02.868 DEBUG:boto:Headers: {} 2022-09-09T22:31:02.868 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:31:02.869 DEBUG:boto:Port: 8001 2022-09-09T22:31:02.870 DEBUG:boto:Params: {} 2022-09-09T22:31:02.872 DEBUG:boto:Token: None 2022-09-09T22:31:02.873 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:02 GMT /jkcgbj-10/ 2022-09-09T22:31:02.875 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:SV44BjKvsyhPkv6ZwRojU7ej0Bw= 2022-09-09T22:31:02.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:SV44BjKvsyhPkv6ZwRojU7ej0Bw=', 'Content-Length': '0'} 2022-09-09T22:31:02.882 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e7abf4eff6210ee-00631bbea6-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:31:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:02.884 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:02.884 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:02.885 DEBUG:boto:Method: HEAD 2022-09-09T22:31:02.887 DEBUG:boto:Path: /jkcgbj-10/ 2022-09-09T22:31:02.888 DEBUG:boto:Data: 2022-09-09T22:31:02.889 DEBUG:boto:Headers: {} 2022-09-09T22:31:02.890 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:02.891 DEBUG:boto:Port: 8000 2022-09-09T22:31:02.892 DEBUG:boto:Params: {} 2022-09-09T22:31:02.893 DEBUG:boto:Token: None 2022-09-09T22:31:02.894 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:02 GMT /jkcgbj-10/ 2022-09-09T22:31:02.895 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:SV44BjKvsyhPkv6ZwRojU7ej0Bw= 2022-09-09T22:31:02.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:SV44BjKvsyhPkv6ZwRojU7ej0Bw=', 'Content-Length': '0'} 2022-09-09T22:31:02.901 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fe611cf9b9ea2554-00631bbea6-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:31:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:02.902 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:31:02.903 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:02.903 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:02.904 DEBUG:boto:path=/jkcgbj-10/?versions&max-keys=999 2022-09-09T22:31:02.904 DEBUG:boto:auth_path=/jkcgbj-10/?versions&max-keys=999 2022-09-09T22:31:02.905 DEBUG:boto:Method: GET 2022-09-09T22:31:02.905 DEBUG:boto:Path: /jkcgbj-10/?versions&max-keys=999 2022-09-09T22:31:02.906 DEBUG:boto:Data: 2022-09-09T22:31:02.906 DEBUG:boto:Headers: {} 2022-09-09T22:31:02.907 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:31:02.908 DEBUG:boto:Port: 8001 2022-09-09T22:31:02.909 DEBUG:boto:Params: {} 2022-09-09T22:31:02.909 DEBUG:boto:Token: None 2022-09-09T22:31:02.910 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:02 GMT /jkcgbj-10/?versions 2022-09-09T22:31:02.912 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:BNtmYNH4u+iZzk+x+gHpqtvjygY= 2022-09-09T22:31:02.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:BNtmYNH4u+iZzk+x+gHpqtvjygY=', 'Content-Length': '0'} 2022-09-09T22:31:02.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e410899a634c234-00631bbea6-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:02.925 DEBUG:boto:b'jkcgbj-10999false&nulltrue2022-09-09T22:30:44.219Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-09T22:30:44.155Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-09T22:30:44.054Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-09T22:30:43.986Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:02.929 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-09T22:31:02.930 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-09T22:31:02.931 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-09T22:31:02.932 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-09T22:31:02.934 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:31:02.935 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:02.936 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:02.937 DEBUG:boto:path=/jkcgbj-10/?versions&max-keys=999 2022-09-09T22:31:02.940 DEBUG:boto:auth_path=/jkcgbj-10/?versions&max-keys=999 2022-09-09T22:31:02.941 DEBUG:boto:Method: GET 2022-09-09T22:31:02.942 DEBUG:boto:Path: /jkcgbj-10/?versions&max-keys=999 2022-09-09T22:31:02.943 DEBUG:boto:Data: 2022-09-09T22:31:02.944 DEBUG:boto:Headers: {} 2022-09-09T22:31:02.945 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:02.946 DEBUG:boto:Port: 8000 2022-09-09T22:31:02.947 DEBUG:boto:Params: {} 2022-09-09T22:31:02.948 DEBUG:boto:Token: None 2022-09-09T22:31:02.949 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:02 GMT /jkcgbj-10/?versions 2022-09-09T22:31:02.951 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:BNtmYNH4u+iZzk+x+gHpqtvjygY= 2022-09-09T22:31:02.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:BNtmYNH4u+iZzk+x+gHpqtvjygY=', 'Content-Length': '0'} 2022-09-09T22:31:02.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018d027f6b09f4d80-00631bbea6-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:02.961 DEBUG:boto:b'jkcgbj-10999false&nulltrue2022-09-09T22:30:44.219Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-09T22:30:44.155Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-09T22:30:44.054Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-09T22:30:43.986Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:02.962 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-09T22:31:02.962 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-09T22:31:02.963 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-09T22:31:02.963 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-09T22:31:02.964 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:02.965 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:02.972 DEBUG:boto:path=/jkcgbj-10/?versions&max-keys=999 2022-09-09T22:31:02.973 DEBUG:boto:auth_path=/jkcgbj-10/?versions&max-keys=999 2022-09-09T22:31:02.974 DEBUG:boto:Method: GET 2022-09-09T22:31:02.974 DEBUG:boto:Path: /jkcgbj-10/?versions&max-keys=999 2022-09-09T22:31:02.975 DEBUG:boto:Data: 2022-09-09T22:31:02.975 DEBUG:boto:Headers: {} 2022-09-09T22:31:02.976 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:31:02.977 DEBUG:boto:Port: 8001 2022-09-09T22:31:02.977 DEBUG:boto:Params: {} 2022-09-09T22:31:02.978 DEBUG:boto:Token: None 2022-09-09T22:31:02.978 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:02 GMT /jkcgbj-10/?versions 2022-09-09T22:31:02.979 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:BNtmYNH4u+iZzk+x+gHpqtvjygY= 2022-09-09T22:31:02.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:BNtmYNH4u+iZzk+x+gHpqtvjygY=', 'Content-Length': '0'} 2022-09-09T22:31:02.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033a37db19694e212-00631bbea6-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:02.990 DEBUG:boto:b'jkcgbj-10999false&nulltrue2022-09-09T22:30:44.219Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-09T22:30:44.155Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-09T22:30:44.054Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-09T22:30:43.986Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:02.991 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:02.992 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:02.993 DEBUG:boto:path=/jkcgbj-10/?versions&max-keys=999 2022-09-09T22:31:02.993 DEBUG:boto:auth_path=/jkcgbj-10/?versions&max-keys=999 2022-09-09T22:31:02.994 DEBUG:boto:Method: GET 2022-09-09T22:31:02.994 DEBUG:boto:Path: /jkcgbj-10/?versions&max-keys=999 2022-09-09T22:31:02.995 DEBUG:boto:Data: 2022-09-09T22:31:02.995 DEBUG:boto:Headers: {} 2022-09-09T22:31:02.996 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:02.996 DEBUG:boto:Port: 8000 2022-09-09T22:31:02.997 DEBUG:boto:Params: {} 2022-09-09T22:31:02.998 DEBUG:boto:Token: None 2022-09-09T22:31:02.998 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:02 GMT /jkcgbj-10/?versions 2022-09-09T22:31:02.999 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:BNtmYNH4u+iZzk+x+gHpqtvjygY= 2022-09-09T22:31:03.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:BNtmYNH4u+iZzk+x+gHpqtvjygY=', 'Content-Length': '0'} 2022-09-09T22:31:03.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ab8d3bbedc71776-00631bbea7-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:03.007 DEBUG:boto:b'jkcgbj-10999false&nulltrue2022-09-09T22:30:44.219Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-09T22:30:44.155Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-09T22:30:44.054Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-09T22:30:43.986Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:03.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-09T22:31:03.008 DEBUG:boto:path=/jkcgbj-10/%26 2022-09-09T22:31:03.009 DEBUG:boto:auth_path=/jkcgbj-10/%26 2022-09-09T22:31:03.009 DEBUG:boto:path=/jkcgbj-10/%26?versionId=null 2022-09-09T22:31:03.010 DEBUG:boto:auth_path=/jkcgbj-10/%26?versionId=null 2022-09-09T22:31:03.011 DEBUG:boto:Method: GET 2022-09-09T22:31:03.011 DEBUG:boto:Path: /jkcgbj-10/%26?versionId=null 2022-09-09T22:31:03.012 DEBUG:boto:Data: 2022-09-09T22:31:03.012 DEBUG:boto:Headers: {} 2022-09-09T22:31:03.013 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:31:03.013 DEBUG:boto:Port: 8001 2022-09-09T22:31:03.014 DEBUG:boto:Params: {} 2022-09-09T22:31:03.015 DEBUG:boto:Token: None 2022-09-09T22:31:03.015 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:03 GMT /jkcgbj-10/%26?versionId=null 2022-09-09T22:31:03.016 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8bVnKB1+nDJA5LPwY6ffOQzyN2I= 2022-09-09T22:31:03.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8bVnKB1+nDJA5LPwY6ffOQzyN2I=', 'Content-Length': '0'} 2022-09-09T22:31:03.020 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da4c2d6995458d66-00631bbea7-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:03.021 DEBUG:boto:path=/jkcgbj-10/%26 2022-09-09T22:31:03.022 DEBUG:boto:auth_path=/jkcgbj-10/%26 2022-09-09T22:31:03.022 DEBUG:boto:path=/jkcgbj-10/%26?versionId=null 2022-09-09T22:31:03.023 DEBUG:boto:auth_path=/jkcgbj-10/%26?versionId=null 2022-09-09T22:31:03.024 DEBUG:boto:Method: GET 2022-09-09T22:31:03.024 DEBUG:boto:Path: /jkcgbj-10/%26?versionId=null 2022-09-09T22:31:03.025 DEBUG:boto:Data: 2022-09-09T22:31:03.025 DEBUG:boto:Headers: {} 2022-09-09T22:31:03.026 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:03.026 DEBUG:boto:Port: 8000 2022-09-09T22:31:03.027 DEBUG:boto:Params: {} 2022-09-09T22:31:03.028 DEBUG:boto:Token: None 2022-09-09T22:31:03.029 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:03 GMT /jkcgbj-10/%26?versionId=null 2022-09-09T22:31:03.029 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8bVnKB1+nDJA5LPwY6ffOQzyN2I= 2022-09-09T22:31:03.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8bVnKB1+nDJA5LPwY6ffOQzyN2I=', 'Content-Length': '0'} 2022-09-09T22:31:03.034 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c1031564ded9f304-00631bbea7-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:03.035 DEBUG:boto:path=/jkcgbj-10/%26 2022-09-09T22:31:03.036 DEBUG:boto:auth_path=/jkcgbj-10/%26 2022-09-09T22:31:03.036 DEBUG:boto:path=/jkcgbj-10/%26?versionId=null 2022-09-09T22:31:03.037 DEBUG:boto:auth_path=/jkcgbj-10/%26?versionId=null 2022-09-09T22:31:03.037 DEBUG:boto:Method: HEAD 2022-09-09T22:31:03.038 DEBUG:boto:Path: /jkcgbj-10/%26?versionId=null 2022-09-09T22:31:03.038 DEBUG:boto:Data: 2022-09-09T22:31:03.039 DEBUG:boto:Headers: {} 2022-09-09T22:31:03.039 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:31:03.040 DEBUG:boto:Port: 8001 2022-09-09T22:31:03.040 DEBUG:boto:Params: {} 2022-09-09T22:31:03.041 DEBUG:boto:Token: None 2022-09-09T22:31:03.041 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:03 GMT /jkcgbj-10/%26?versionId=null 2022-09-09T22:31:03.042 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9kPod8Q1hzFrKvy7msF2+QaVUxg= 2022-09-09T22:31:03.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9kPod8Q1hzFrKvy7msF2+QaVUxg=', 'Content-Length': '0'} 2022-09-09T22:31:03.045 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000850860bc3dee7e57-00631bbea7-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:03.046 DEBUG:boto:path=/jkcgbj-10/%26 2022-09-09T22:31:03.047 DEBUG:boto:auth_path=/jkcgbj-10/%26 2022-09-09T22:31:03.047 DEBUG:boto:path=/jkcgbj-10/%26?versionId=null 2022-09-09T22:31:03.048 DEBUG:boto:auth_path=/jkcgbj-10/%26?versionId=null 2022-09-09T22:31:03.048 DEBUG:boto:Method: HEAD 2022-09-09T22:31:03.049 DEBUG:boto:Path: /jkcgbj-10/%26?versionId=null 2022-09-09T22:31:03.049 DEBUG:boto:Data: 2022-09-09T22:31:03.050 DEBUG:boto:Headers: {} 2022-09-09T22:31:03.050 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:03.052 DEBUG:boto:Port: 8000 2022-09-09T22:31:03.053 DEBUG:boto:Params: {} 2022-09-09T22:31:03.054 DEBUG:boto:Token: None 2022-09-09T22:31:03.055 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:03 GMT /jkcgbj-10/%26?versionId=null 2022-09-09T22:31:03.056 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9kPod8Q1hzFrKvy7msF2+QaVUxg= 2022-09-09T22:31:03.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9kPod8Q1hzFrKvy7msF2+QaVUxg=', 'Content-Length': '0'} 2022-09-09T22:31:03.061 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fdcf30b3e3d40682-00631bbea7-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:03.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-09T22:31:03.065 DEBUG:boto:path=/jkcgbj-10/%26 2022-09-09T22:31:03.067 DEBUG:boto:auth_path=/jkcgbj-10/%26 2022-09-09T22:31:03.069 DEBUG:boto:path=/jkcgbj-10/%26?versionId=null 2022-09-09T22:31:03.070 DEBUG:boto:auth_path=/jkcgbj-10/%26?versionId=null 2022-09-09T22:31:03.072 DEBUG:boto:Method: GET 2022-09-09T22:31:03.074 DEBUG:boto:Path: /jkcgbj-10/%26?versionId=null 2022-09-09T22:31:03.076 DEBUG:boto:Data: 2022-09-09T22:31:03.077 DEBUG:boto:Headers: {} 2022-09-09T22:31:03.078 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:31:03.079 DEBUG:boto:Port: 8001 2022-09-09T22:31:03.080 DEBUG:boto:Params: {} 2022-09-09T22:31:03.081 DEBUG:boto:Token: None 2022-09-09T22:31:03.083 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:03 GMT /jkcgbj-10/%26?versionId=null 2022-09-09T22:31:03.084 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8bVnKB1+nDJA5LPwY6ffOQzyN2I= 2022-09-09T22:31:03.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8bVnKB1+nDJA5LPwY6ffOQzyN2I=', 'Content-Length': '0'} 2022-09-09T22:31:03.091 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df11d8631ce6ec90-00631bbea7-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:03.094 DEBUG:boto:path=/jkcgbj-10/%26 2022-09-09T22:31:03.096 DEBUG:boto:auth_path=/jkcgbj-10/%26 2022-09-09T22:31:03.096 DEBUG:boto:path=/jkcgbj-10/%26?versionId=null 2022-09-09T22:31:03.097 DEBUG:boto:auth_path=/jkcgbj-10/%26?versionId=null 2022-09-09T22:31:03.099 DEBUG:boto:Method: GET 2022-09-09T22:31:03.100 DEBUG:boto:Path: /jkcgbj-10/%26?versionId=null 2022-09-09T22:31:03.102 DEBUG:boto:Data: 2022-09-09T22:31:03.103 DEBUG:boto:Headers: {} 2022-09-09T22:31:03.103 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:03.104 DEBUG:boto:Port: 8000 2022-09-09T22:31:03.105 DEBUG:boto:Params: {} 2022-09-09T22:31:03.107 DEBUG:boto:Token: None 2022-09-09T22:31:03.108 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:03 GMT /jkcgbj-10/%26?versionId=null 2022-09-09T22:31:03.109 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8bVnKB1+nDJA5LPwY6ffOQzyN2I= 2022-09-09T22:31:03.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8bVnKB1+nDJA5LPwY6ffOQzyN2I=', 'Content-Length': '0'} 2022-09-09T22:31:03.114 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a4ff521458dbf4b-00631bbea7-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:03.115 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:03.116 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:03.116 DEBUG:boto:path=/jkcgbj-10/?prefix=%26 2022-09-09T22:31:03.117 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=%26 2022-09-09T22:31:03.118 DEBUG:boto:Method: GET 2022-09-09T22:31:03.118 DEBUG:boto:Path: /jkcgbj-10/?prefix=%26 2022-09-09T22:31:03.119 DEBUG:boto:Data: 2022-09-09T22:31:03.120 DEBUG:boto:Headers: {} 2022-09-09T22:31:03.122 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:31:03.123 DEBUG:boto:Port: 8001 2022-09-09T22:31:03.124 DEBUG:boto:Params: {} 2022-09-09T22:31:03.126 DEBUG:boto:Token: None 2022-09-09T22:31:03.126 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:03 GMT /jkcgbj-10/ 2022-09-09T22:31:03.127 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JGo66aQrZxpgQvXWSGdtaSolpp8= 2022-09-09T22:31:03.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JGo66aQrZxpgQvXWSGdtaSolpp8=', 'Content-Length': '0'} 2022-09-09T22:31:03.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092a14ac26be385ab-00631bbea7-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:03.138 DEBUG:boto:b'jkcgbj-10&1000false&2022-09-09T22:30:44.219Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:03.139 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:03.140 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:03.141 DEBUG:boto:path=/jkcgbj-10/?prefix=%26 2022-09-09T22:31:03.142 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=%26 2022-09-09T22:31:03.143 DEBUG:boto:Method: GET 2022-09-09T22:31:03.144 DEBUG:boto:Path: /jkcgbj-10/?prefix=%26 2022-09-09T22:31:03.145 DEBUG:boto:Data: 2022-09-09T22:31:03.146 DEBUG:boto:Headers: {} 2022-09-09T22:31:03.147 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:03.148 DEBUG:boto:Port: 8000 2022-09-09T22:31:03.149 DEBUG:boto:Params: {} 2022-09-09T22:31:03.150 DEBUG:boto:Token: None 2022-09-09T22:31:03.151 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:03 GMT /jkcgbj-10/ 2022-09-09T22:31:03.152 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JGo66aQrZxpgQvXWSGdtaSolpp8= 2022-09-09T22:31:03.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JGo66aQrZxpgQvXWSGdtaSolpp8=', 'Content-Length': '0'} 2022-09-09T22:31:03.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025dba00bd462cff3-00631bbea7-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:03.160 DEBUG:boto:b'jkcgbj-10&1000false&2022-09-09T22:30:44.219Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:03.162 DEBUG:boto:path=/jkcgbj-10/%26 2022-09-09T22:31:03.163 DEBUG:boto:auth_path=/jkcgbj-10/%26 2022-09-09T22:31:03.164 DEBUG:boto:Method: HEAD 2022-09-09T22:31:03.165 DEBUG:boto:Path: /jkcgbj-10/%26 2022-09-09T22:31:03.166 DEBUG:boto:Data: 2022-09-09T22:31:03.166 DEBUG:boto:Headers: {} 2022-09-09T22:31:03.167 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:31:03.168 DEBUG:boto:Port: 8001 2022-09-09T22:31:03.169 DEBUG:boto:Params: {} 2022-09-09T22:31:03.170 DEBUG:boto:Token: None 2022-09-09T22:31:03.171 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:03 GMT /jkcgbj-10/%26 2022-09-09T22:31:03.172 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:89Lb6w1SJkPzEl5FWCtVE5H1v1s= 2022-09-09T22:31:03.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:89Lb6w1SJkPzEl5FWCtVE5H1v1s=', 'Content-Length': '0'} 2022-09-09T22:31:03.177 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd520ea6573ad2b9-00631bbea7-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:03.178 DEBUG:boto:path=/jkcgbj-10/%26 2022-09-09T22:31:03.179 DEBUG:boto:auth_path=/jkcgbj-10/%26 2022-09-09T22:31:03.179 DEBUG:boto:Method: HEAD 2022-09-09T22:31:03.180 DEBUG:boto:Path: /jkcgbj-10/%26 2022-09-09T22:31:03.180 DEBUG:boto:Data: 2022-09-09T22:31:03.181 DEBUG:boto:Headers: {} 2022-09-09T22:31:03.181 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:03.182 DEBUG:boto:Port: 8000 2022-09-09T22:31:03.183 DEBUG:boto:Params: {} 2022-09-09T22:31:03.183 DEBUG:boto:Token: None 2022-09-09T22:31:03.184 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:03 GMT /jkcgbj-10/%26 2022-09-09T22:31:03.184 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:89Lb6w1SJkPzEl5FWCtVE5H1v1s= 2022-09-09T22:31:03.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:89Lb6w1SJkPzEl5FWCtVE5H1v1s=', 'Content-Length': '0'} 2022-09-09T22:31:03.188 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000066148ec106ffb7bd-00631bbea7-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:03.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-09T22:31:03.190 DEBUG:boto:path=/jkcgbj-10/%26 2022-09-09T22:31:03.190 DEBUG:boto:auth_path=/jkcgbj-10/%26 2022-09-09T22:31:03.191 DEBUG:boto:Method: GET 2022-09-09T22:31:03.192 DEBUG:boto:Path: /jkcgbj-10/%26 2022-09-09T22:31:03.192 DEBUG:boto:Data: 2022-09-09T22:31:03.193 DEBUG:boto:Headers: {} 2022-09-09T22:31:03.193 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:31:03.194 DEBUG:boto:Port: 8001 2022-09-09T22:31:03.194 DEBUG:boto:Params: {} 2022-09-09T22:31:03.195 DEBUG:boto:Token: None 2022-09-09T22:31:03.196 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:03 GMT /jkcgbj-10/%26 2022-09-09T22:31:03.196 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:m9R1wBo/b5r6WgXpnviiDdnNRI8= 2022-09-09T22:31:03.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:m9R1wBo/b5r6WgXpnviiDdnNRI8=', 'Content-Length': '0'} 2022-09-09T22:31:03.200 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000012ad1d70fc1baa78-00631bbea7-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:03.201 DEBUG:boto:path=/jkcgbj-10/%26 2022-09-09T22:31:03.202 DEBUG:boto:auth_path=/jkcgbj-10/%26 2022-09-09T22:31:03.202 DEBUG:boto:Method: GET 2022-09-09T22:31:03.203 DEBUG:boto:Path: /jkcgbj-10/%26 2022-09-09T22:31:03.203 DEBUG:boto:Data: 2022-09-09T22:31:03.204 DEBUG:boto:Headers: {} 2022-09-09T22:31:03.204 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:03.205 DEBUG:boto:Port: 8000 2022-09-09T22:31:03.205 DEBUG:boto:Params: {} 2022-09-09T22:31:03.206 DEBUG:boto:Token: None 2022-09-09T22:31:03.207 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:03 GMT /jkcgbj-10/%26 2022-09-09T22:31:03.207 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:m9R1wBo/b5r6WgXpnviiDdnNRI8= 2022-09-09T22:31:03.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:m9R1wBo/b5r6WgXpnviiDdnNRI8=', 'Content-Length': '0'} 2022-09-09T22:31:03.211 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000003d5e704110c18bd-00631bbea7-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:03.258 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:03.259 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:03.260 DEBUG:boto:path=/jkcgbj-10/?prefix=%26 2022-09-09T22:31:03.261 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=%26 2022-09-09T22:31:03.262 DEBUG:boto:Method: GET 2022-09-09T22:31:03.264 DEBUG:boto:Path: /jkcgbj-10/?prefix=%26 2022-09-09T22:31:03.265 DEBUG:boto:Data: 2022-09-09T22:31:03.266 DEBUG:boto:Headers: {} 2022-09-09T22:31:03.267 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:31:03.268 DEBUG:boto:Port: 8001 2022-09-09T22:31:03.269 DEBUG:boto:Params: {} 2022-09-09T22:31:03.270 DEBUG:boto:Token: None 2022-09-09T22:31:03.271 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:03 GMT /jkcgbj-10/ 2022-09-09T22:31:03.272 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JGo66aQrZxpgQvXWSGdtaSolpp8= 2022-09-09T22:31:03.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JGo66aQrZxpgQvXWSGdtaSolpp8=', 'Content-Length': '0'} 2022-09-09T22:31:03.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000723fec55bc6c104b-00631bbea7-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:03.280 DEBUG:boto:b'jkcgbj-10&1000false&2022-09-09T22:30:44.219Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:03.281 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:03.282 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:03.282 DEBUG:boto:path=/jkcgbj-10/?prefix=%26 2022-09-09T22:31:03.283 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=%26 2022-09-09T22:31:03.283 DEBUG:boto:Method: GET 2022-09-09T22:31:03.284 DEBUG:boto:Path: /jkcgbj-10/?prefix=%26 2022-09-09T22:31:03.284 DEBUG:boto:Data: 2022-09-09T22:31:03.286 DEBUG:boto:Headers: {} 2022-09-09T22:31:03.286 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:03.287 DEBUG:boto:Port: 8000 2022-09-09T22:31:03.287 DEBUG:boto:Params: {} 2022-09-09T22:31:03.288 DEBUG:boto:Token: None 2022-09-09T22:31:03.288 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:03 GMT /jkcgbj-10/ 2022-09-09T22:31:03.289 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JGo66aQrZxpgQvXWSGdtaSolpp8= 2022-09-09T22:31:03.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JGo66aQrZxpgQvXWSGdtaSolpp8=', 'Content-Length': '0'} 2022-09-09T22:31:03.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e56278b7b2483a6-00631bbea7-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:03.298 DEBUG:boto:b'jkcgbj-10&1000false&2022-09-09T22:30:44.219Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:03.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-09T22:31:03.299 DEBUG:boto:path=/jkcgbj-10/%3A 2022-09-09T22:31:03.300 DEBUG:boto:auth_path=/jkcgbj-10/%3A 2022-09-09T22:31:03.301 DEBUG:boto:path=/jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:03.301 DEBUG:boto:auth_path=/jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:03.302 DEBUG:boto:Method: GET 2022-09-09T22:31:03.303 DEBUG:boto:Path: /jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:03.303 DEBUG:boto:Data: 2022-09-09T22:31:03.304 DEBUG:boto:Headers: {} 2022-09-09T22:31:03.305 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:31:03.305 DEBUG:boto:Port: 8001 2022-09-09T22:31:03.306 DEBUG:boto:Params: {} 2022-09-09T22:31:03.307 DEBUG:boto:Token: None 2022-09-09T22:31:03.307 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:03 GMT /jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:03.308 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1dyNLGulc8PU8sO8ivoqNv88xU4= 2022-09-09T22:31:03.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1dyNLGulc8PU8sO8ivoqNv88xU4=', 'Content-Length': '0'} 2022-09-09T22:31:03.313 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050a6bf00a3f1d0f7-00631bbea7-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:03.314 DEBUG:boto:path=/jkcgbj-10/%3A 2022-09-09T22:31:03.315 DEBUG:boto:auth_path=/jkcgbj-10/%3A 2022-09-09T22:31:03.315 DEBUG:boto:path=/jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:03.316 DEBUG:boto:auth_path=/jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:03.317 DEBUG:boto:Method: GET 2022-09-09T22:31:03.318 DEBUG:boto:Path: /jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:03.318 DEBUG:boto:Data: 2022-09-09T22:31:03.319 DEBUG:boto:Headers: {} 2022-09-09T22:31:03.319 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:03.320 DEBUG:boto:Port: 8000 2022-09-09T22:31:03.321 DEBUG:boto:Params: {} 2022-09-09T22:31:03.321 DEBUG:boto:Token: None 2022-09-09T22:31:03.322 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:03 GMT /jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:03.323 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1dyNLGulc8PU8sO8ivoqNv88xU4= 2022-09-09T22:31:03.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1dyNLGulc8PU8sO8ivoqNv88xU4=', 'Content-Length': '0'} 2022-09-09T22:31:03.330 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d17408db6c164948-00631bbea7-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:03.332 DEBUG:boto:path=/jkcgbj-10/%3A 2022-09-09T22:31:03.333 DEBUG:boto:auth_path=/jkcgbj-10/%3A 2022-09-09T22:31:03.334 DEBUG:boto:path=/jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:03.336 DEBUG:boto:auth_path=/jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:03.336 DEBUG:boto:Method: HEAD 2022-09-09T22:31:03.337 DEBUG:boto:Path: /jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:03.338 DEBUG:boto:Data: 2022-09-09T22:31:03.339 DEBUG:boto:Headers: {} 2022-09-09T22:31:03.340 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:31:03.340 DEBUG:boto:Port: 8001 2022-09-09T22:31:03.341 DEBUG:boto:Params: {} 2022-09-09T22:31:03.341 DEBUG:boto:Token: None 2022-09-09T22:31:03.342 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:03 GMT /jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:03.343 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7Vm+hsFB263SDHBf5LfY7tvtSMI= 2022-09-09T22:31:03.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7Vm+hsFB263SDHBf5LfY7tvtSMI=', 'Content-Length': '0'} 2022-09-09T22:31:03.346 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068f0adee7e950653-00631bbea7-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:03.348 DEBUG:boto:path=/jkcgbj-10/%3A 2022-09-09T22:31:03.349 DEBUG:boto:auth_path=/jkcgbj-10/%3A 2022-09-09T22:31:03.350 DEBUG:boto:path=/jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:03.350 DEBUG:boto:auth_path=/jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:03.352 DEBUG:boto:Method: HEAD 2022-09-09T22:31:03.353 DEBUG:boto:Path: /jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:03.353 DEBUG:boto:Data: 2022-09-09T22:31:03.354 DEBUG:boto:Headers: {} 2022-09-09T22:31:03.354 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:03.355 DEBUG:boto:Port: 8000 2022-09-09T22:31:03.356 DEBUG:boto:Params: {} 2022-09-09T22:31:03.357 DEBUG:boto:Token: None 2022-09-09T22:31:03.358 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:03 GMT /jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:03.359 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7Vm+hsFB263SDHBf5LfY7tvtSMI= 2022-09-09T22:31:03.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7Vm+hsFB263SDHBf5LfY7tvtSMI=', 'Content-Length': '0'} 2022-09-09T22:31:03.363 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5284020d676e815-00631bbea7-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:03.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-09T22:31:03.364 DEBUG:boto:path=/jkcgbj-10/%3A 2022-09-09T22:31:03.365 DEBUG:boto:auth_path=/jkcgbj-10/%3A 2022-09-09T22:31:03.366 DEBUG:boto:path=/jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:03.366 DEBUG:boto:auth_path=/jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:03.367 DEBUG:boto:Method: GET 2022-09-09T22:31:03.368 DEBUG:boto:Path: /jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:03.369 DEBUG:boto:Data: 2022-09-09T22:31:03.370 DEBUG:boto:Headers: {} 2022-09-09T22:31:03.370 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:31:03.371 DEBUG:boto:Port: 8001 2022-09-09T22:31:03.372 DEBUG:boto:Params: {} 2022-09-09T22:31:03.374 DEBUG:boto:Token: None 2022-09-09T22:31:03.375 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:03 GMT /jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:03.375 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1dyNLGulc8PU8sO8ivoqNv88xU4= 2022-09-09T22:31:03.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1dyNLGulc8PU8sO8ivoqNv88xU4=', 'Content-Length': '0'} 2022-09-09T22:31:03.379 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bf57a448286967ea-00631bbea7-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:03.381 DEBUG:boto:path=/jkcgbj-10/%3A 2022-09-09T22:31:03.382 DEBUG:boto:auth_path=/jkcgbj-10/%3A 2022-09-09T22:31:03.382 DEBUG:boto:path=/jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:03.383 DEBUG:boto:auth_path=/jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:03.383 DEBUG:boto:Method: GET 2022-09-09T22:31:03.384 DEBUG:boto:Path: /jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:03.384 DEBUG:boto:Data: 2022-09-09T22:31:03.385 DEBUG:boto:Headers: {} 2022-09-09T22:31:03.386 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:03.386 DEBUG:boto:Port: 8000 2022-09-09T22:31:03.387 DEBUG:boto:Params: {} 2022-09-09T22:31:03.388 DEBUG:boto:Token: None 2022-09-09T22:31:03.389 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:03 GMT /jkcgbj-10/%3A?versionId=null 2022-09-09T22:31:03.390 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1dyNLGulc8PU8sO8ivoqNv88xU4= 2022-09-09T22:31:03.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1dyNLGulc8PU8sO8ivoqNv88xU4=', 'Content-Length': '0'} 2022-09-09T22:31:03.394 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077d2fc1399fc9b49-00631bbea7-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:03.440 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:03.442 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:03.443 DEBUG:boto:path=/jkcgbj-10/?prefix=%3A 2022-09-09T22:31:03.444 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=%3A 2022-09-09T22:31:03.683 DEBUG:boto:Method: GET 2022-09-09T22:31:03.685 DEBUG:boto:Path: /jkcgbj-10/?prefix=%3A 2022-09-09T22:31:03.690 DEBUG:boto:Data: 2022-09-09T22:31:03.691 DEBUG:boto:Headers: {} 2022-09-09T22:31:03.692 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:31:03.693 DEBUG:boto:Port: 8001 2022-09-09T22:31:03.696 DEBUG:boto:Params: {} 2022-09-09T22:31:03.696 DEBUG:boto:Token: None 2022-09-09T22:31:03.697 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:03 GMT /jkcgbj-10/ 2022-09-09T22:31:03.698 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JGo66aQrZxpgQvXWSGdtaSolpp8= 2022-09-09T22:31:03.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JGo66aQrZxpgQvXWSGdtaSolpp8=', 'Content-Length': '0'} 2022-09-09T22:31:03.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6bc966fb382498a-00631bbea7-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:03.718 DEBUG:boto:b'jkcgbj-10:1000false:2022-09-09T22:30:44.155Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:03.719 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:03.720 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:03.720 DEBUG:boto:path=/jkcgbj-10/?prefix=%3A 2022-09-09T22:31:03.721 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=%3A 2022-09-09T22:31:03.721 DEBUG:boto:Method: GET 2022-09-09T22:31:03.722 DEBUG:boto:Path: /jkcgbj-10/?prefix=%3A 2022-09-09T22:31:03.722 DEBUG:boto:Data: 2022-09-09T22:31:03.723 DEBUG:boto:Headers: {} 2022-09-09T22:31:03.723 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:03.724 DEBUG:boto:Port: 8000 2022-09-09T22:31:03.724 DEBUG:boto:Params: {} 2022-09-09T22:31:03.725 DEBUG:boto:Token: None 2022-09-09T22:31:03.726 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:03 GMT /jkcgbj-10/ 2022-09-09T22:31:03.727 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JGo66aQrZxpgQvXWSGdtaSolpp8= 2022-09-09T22:31:03.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JGo66aQrZxpgQvXWSGdtaSolpp8=', 'Content-Length': '0'} 2022-09-09T22:31:03.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008836d97371234d26-00631bbea7-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:03.742 DEBUG:boto:b'jkcgbj-10:1000false:2022-09-09T22:30:44.155Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:03.743 DEBUG:boto:path=/jkcgbj-10/%3A 2022-09-09T22:31:03.744 DEBUG:boto:auth_path=/jkcgbj-10/%3A 2022-09-09T22:31:03.744 DEBUG:boto:Method: HEAD 2022-09-09T22:31:03.745 DEBUG:boto:Path: /jkcgbj-10/%3A 2022-09-09T22:31:03.745 DEBUG:boto:Data: 2022-09-09T22:31:03.746 DEBUG:boto:Headers: {} 2022-09-09T22:31:03.747 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:31:03.747 DEBUG:boto:Port: 8001 2022-09-09T22:31:03.748 DEBUG:boto:Params: {} 2022-09-09T22:31:03.748 DEBUG:boto:Token: None 2022-09-09T22:31:03.749 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:03 GMT /jkcgbj-10/%3A 2022-09-09T22:31:03.749 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:fMOBvw/5R40YfAvMnba3k9OgncE= 2022-09-09T22:31:03.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:fMOBvw/5R40YfAvMnba3k9OgncE=', 'Content-Length': '0'} 2022-09-09T22:31:03.768 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c53c373ad3b4aa6-00631bbea7-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:03.769 DEBUG:boto:path=/jkcgbj-10/%3A 2022-09-09T22:31:03.769 DEBUG:boto:auth_path=/jkcgbj-10/%3A 2022-09-09T22:31:03.770 DEBUG:boto:Method: HEAD 2022-09-09T22:31:03.770 DEBUG:boto:Path: /jkcgbj-10/%3A 2022-09-09T22:31:03.771 DEBUG:boto:Data: 2022-09-09T22:31:03.772 DEBUG:boto:Headers: {} 2022-09-09T22:31:03.772 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:03.773 DEBUG:boto:Port: 8000 2022-09-09T22:31:03.773 DEBUG:boto:Params: {} 2022-09-09T22:31:03.774 DEBUG:boto:Token: None 2022-09-09T22:31:03.777 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:03 GMT /jkcgbj-10/%3A 2022-09-09T22:31:03.777 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:fMOBvw/5R40YfAvMnba3k9OgncE= 2022-09-09T22:31:03.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:fMOBvw/5R40YfAvMnba3k9OgncE=', 'Content-Length': '0'} 2022-09-09T22:31:03.785 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f57aba4442ceb33c-00631bbea7-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:03.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-09T22:31:03.786 DEBUG:boto:path=/jkcgbj-10/%3A 2022-09-09T22:31:03.787 DEBUG:boto:auth_path=/jkcgbj-10/%3A 2022-09-09T22:31:03.788 DEBUG:boto:Method: GET 2022-09-09T22:31:03.788 DEBUG:boto:Path: /jkcgbj-10/%3A 2022-09-09T22:31:03.789 DEBUG:boto:Data: 2022-09-09T22:31:03.790 DEBUG:boto:Headers: {} 2022-09-09T22:31:03.790 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:31:03.791 DEBUG:boto:Port: 8001 2022-09-09T22:31:03.792 DEBUG:boto:Params: {} 2022-09-09T22:31:03.792 DEBUG:boto:Token: None 2022-09-09T22:31:03.793 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:03 GMT /jkcgbj-10/%3A 2022-09-09T22:31:03.793 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JKgKKLJkltnGM9/OU9v3YN9wfhk= 2022-09-09T22:31:03.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JKgKKLJkltnGM9/OU9v3YN9wfhk=', 'Content-Length': '0'} 2022-09-09T22:31:03.846 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e84ed3677d12edc-00631bbea7-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:03.886 DEBUG:boto:path=/jkcgbj-10/%3A 2022-09-09T22:31:03.887 DEBUG:boto:auth_path=/jkcgbj-10/%3A 2022-09-09T22:31:03.887 DEBUG:boto:Method: GET 2022-09-09T22:31:03.888 DEBUG:boto:Path: /jkcgbj-10/%3A 2022-09-09T22:31:03.889 DEBUG:boto:Data: 2022-09-09T22:31:03.890 DEBUG:boto:Headers: {} 2022-09-09T22:31:03.890 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:03.891 DEBUG:boto:Port: 8000 2022-09-09T22:31:03.891 DEBUG:boto:Params: {} 2022-09-09T22:31:03.893 DEBUG:boto:Token: None 2022-09-09T22:31:03.894 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:03 GMT /jkcgbj-10/%3A 2022-09-09T22:31:03.895 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JKgKKLJkltnGM9/OU9v3YN9wfhk= 2022-09-09T22:31:03.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JKgKKLJkltnGM9/OU9v3YN9wfhk=', 'Content-Length': '0'} 2022-09-09T22:31:03.906 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7d95571bb8e80aa-00631bbea7-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:03.912 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:03.913 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:03.920 DEBUG:boto:path=/jkcgbj-10/?prefix=%3A 2022-09-09T22:31:03.921 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=%3A 2022-09-09T22:31:03.922 DEBUG:boto:Method: GET 2022-09-09T22:31:03.924 DEBUG:boto:Path: /jkcgbj-10/?prefix=%3A 2022-09-09T22:31:03.925 DEBUG:boto:Data: 2022-09-09T22:31:03.925 DEBUG:boto:Headers: {} 2022-09-09T22:31:03.926 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:31:03.927 DEBUG:boto:Port: 8001 2022-09-09T22:31:03.927 DEBUG:boto:Params: {} 2022-09-09T22:31:03.928 DEBUG:boto:Token: None 2022-09-09T22:31:03.928 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:03 GMT /jkcgbj-10/ 2022-09-09T22:31:03.929 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JGo66aQrZxpgQvXWSGdtaSolpp8= 2022-09-09T22:31:03.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JGo66aQrZxpgQvXWSGdtaSolpp8=', 'Content-Length': '0'} 2022-09-09T22:31:03.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006dee7ed451d91fee-00631bbea7-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:03.946 DEBUG:boto:b'jkcgbj-10:1000false:2022-09-09T22:30:44.155Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:03.948 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:03.948 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:03.949 DEBUG:boto:path=/jkcgbj-10/?prefix=%3A 2022-09-09T22:31:03.949 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=%3A 2022-09-09T22:31:03.950 DEBUG:boto:Method: GET 2022-09-09T22:31:03.951 DEBUG:boto:Path: /jkcgbj-10/?prefix=%3A 2022-09-09T22:31:03.951 DEBUG:boto:Data: 2022-09-09T22:31:03.952 DEBUG:boto:Headers: {} 2022-09-09T22:31:03.952 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:03.953 DEBUG:boto:Port: 8000 2022-09-09T22:31:03.954 DEBUG:boto:Params: {} 2022-09-09T22:31:03.955 DEBUG:boto:Token: None 2022-09-09T22:31:03.955 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:03 GMT /jkcgbj-10/ 2022-09-09T22:31:03.956 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JGo66aQrZxpgQvXWSGdtaSolpp8= 2022-09-09T22:31:03.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JGo66aQrZxpgQvXWSGdtaSolpp8=', 'Content-Length': '0'} 2022-09-09T22:31:03.968 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a6290af1089c97c-00631bbea7-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:03.971 DEBUG:boto:b'jkcgbj-10:1000false:2022-09-09T22:30:44.155Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:03.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-09T22:31:03.973 DEBUG:boto:path=/jkcgbj-10/_myobj 2022-09-09T22:31:03.973 DEBUG:boto:auth_path=/jkcgbj-10/_myobj 2022-09-09T22:31:03.974 DEBUG:boto:path=/jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:03.974 DEBUG:boto:auth_path=/jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:03.975 DEBUG:boto:Method: GET 2022-09-09T22:31:03.976 DEBUG:boto:Path: /jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:03.976 DEBUG:boto:Data: 2022-09-09T22:31:03.977 DEBUG:boto:Headers: {} 2022-09-09T22:31:03.977 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:31:03.978 DEBUG:boto:Port: 8001 2022-09-09T22:31:03.978 DEBUG:boto:Params: {} 2022-09-09T22:31:03.979 DEBUG:boto:Token: None 2022-09-09T22:31:03.980 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:03 GMT /jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:03.981 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Y3+DQ9RLyawIBQDBnTX1Ylo89tE= 2022-09-09T22:31:03.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Y3+DQ9RLyawIBQDBnTX1Ylo89tE=', 'Content-Length': '0'} 2022-09-09T22:31:03.985 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000058ecb643d5b2dd77-00631bbea7-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:03.989 DEBUG:boto:path=/jkcgbj-10/_myobj 2022-09-09T22:31:03.990 DEBUG:boto:auth_path=/jkcgbj-10/_myobj 2022-09-09T22:31:03.990 DEBUG:boto:path=/jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:03.992 DEBUG:boto:auth_path=/jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:03.993 DEBUG:boto:Method: GET 2022-09-09T22:31:03.994 DEBUG:boto:Path: /jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:03.994 DEBUG:boto:Data: 2022-09-09T22:31:03.995 DEBUG:boto:Headers: {} 2022-09-09T22:31:03.995 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:03.996 DEBUG:boto:Port: 8000 2022-09-09T22:31:03.997 DEBUG:boto:Params: {} 2022-09-09T22:31:03.997 DEBUG:boto:Token: None 2022-09-09T22:31:03.998 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:03 GMT /jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:03.998 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Y3+DQ9RLyawIBQDBnTX1Ylo89tE= 2022-09-09T22:31:03.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Y3+DQ9RLyawIBQDBnTX1Ylo89tE=', 'Content-Length': '0'} 2022-09-09T22:31:04.009 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc25f5830f9eb4d1-00631bbea8-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:04.012 DEBUG:boto:path=/jkcgbj-10/_myobj 2022-09-09T22:31:04.013 DEBUG:boto:auth_path=/jkcgbj-10/_myobj 2022-09-09T22:31:04.013 DEBUG:boto:path=/jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:04.014 DEBUG:boto:auth_path=/jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:04.014 DEBUG:boto:Method: HEAD 2022-09-09T22:31:04.015 DEBUG:boto:Path: /jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:04.015 DEBUG:boto:Data: 2022-09-09T22:31:04.016 DEBUG:boto:Headers: {} 2022-09-09T22:31:04.017 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:31:04.017 DEBUG:boto:Port: 8001 2022-09-09T22:31:04.018 DEBUG:boto:Params: {} 2022-09-09T22:31:04.019 DEBUG:boto:Token: None 2022-09-09T22:31:04.019 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:04 GMT /jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:04.020 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R/IOhwpkfbto5IpRlwSuL/nLmp8= 2022-09-09T22:31:04.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R/IOhwpkfbto5IpRlwSuL/nLmp8=', 'Content-Length': '0'} 2022-09-09T22:31:04.039 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004cbdb56542aa7381-00631bbea8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:04.040 DEBUG:boto:path=/jkcgbj-10/_myobj 2022-09-09T22:31:04.041 DEBUG:boto:auth_path=/jkcgbj-10/_myobj 2022-09-09T22:31:04.041 DEBUG:boto:path=/jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:04.042 DEBUG:boto:auth_path=/jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:04.042 DEBUG:boto:Method: HEAD 2022-09-09T22:31:04.043 DEBUG:boto:Path: /jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:04.044 DEBUG:boto:Data: 2022-09-09T22:31:04.044 DEBUG:boto:Headers: {} 2022-09-09T22:31:04.046 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:04.046 DEBUG:boto:Port: 8000 2022-09-09T22:31:04.047 DEBUG:boto:Params: {} 2022-09-09T22:31:04.047 DEBUG:boto:Token: None 2022-09-09T22:31:04.048 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:04 GMT /jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:04.048 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R/IOhwpkfbto5IpRlwSuL/nLmp8= 2022-09-09T22:31:04.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R/IOhwpkfbto5IpRlwSuL/nLmp8=', 'Content-Length': '0'} 2022-09-09T22:31:04.062 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007070bc8231108ab-00631bbea8-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:04.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-09T22:31:04.064 DEBUG:boto:path=/jkcgbj-10/_myobj 2022-09-09T22:31:04.064 DEBUG:boto:auth_path=/jkcgbj-10/_myobj 2022-09-09T22:31:04.065 DEBUG:boto:path=/jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:04.066 DEBUG:boto:auth_path=/jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:04.066 DEBUG:boto:Method: GET 2022-09-09T22:31:04.067 DEBUG:boto:Path: /jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:04.068 DEBUG:boto:Data: 2022-09-09T22:31:04.068 DEBUG:boto:Headers: {} 2022-09-09T22:31:04.069 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:31:04.071 DEBUG:boto:Port: 8001 2022-09-09T22:31:04.071 DEBUG:boto:Params: {} 2022-09-09T22:31:04.072 DEBUG:boto:Token: None 2022-09-09T22:31:04.073 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:04 GMT /jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:04.074 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:whs/o+NLHFPNcnXzD+mh98pV6TE= 2022-09-09T22:31:04.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:whs/o+NLHFPNcnXzD+mh98pV6TE=', 'Content-Length': '0'} 2022-09-09T22:31:04.084 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000059bfbda10f779631-00631bbea8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:04.129 DEBUG:boto:path=/jkcgbj-10/_myobj 2022-09-09T22:31:04.130 DEBUG:boto:auth_path=/jkcgbj-10/_myobj 2022-09-09T22:31:04.131 DEBUG:boto:path=/jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:04.132 DEBUG:boto:auth_path=/jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:04.134 DEBUG:boto:Method: GET 2022-09-09T22:31:04.135 DEBUG:boto:Path: /jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:04.136 DEBUG:boto:Data: 2022-09-09T22:31:04.137 DEBUG:boto:Headers: {} 2022-09-09T22:31:04.138 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:04.139 DEBUG:boto:Port: 8000 2022-09-09T22:31:04.140 DEBUG:boto:Params: {} 2022-09-09T22:31:04.141 DEBUG:boto:Token: None 2022-09-09T22:31:04.142 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:04 GMT /jkcgbj-10/_myobj?versionId=null 2022-09-09T22:31:04.143 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:whs/o+NLHFPNcnXzD+mh98pV6TE= 2022-09-09T22:31:04.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:whs/o+NLHFPNcnXzD+mh98pV6TE=', 'Content-Length': '0'} 2022-09-09T22:31:04.155 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f03837b834c70bb9-00631bbea8-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:04.159 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:04.160 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:04.161 DEBUG:boto:path=/jkcgbj-10/?prefix=_myobj 2022-09-09T22:31:04.162 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=_myobj 2022-09-09T22:31:04.163 DEBUG:boto:Method: GET 2022-09-09T22:31:04.164 DEBUG:boto:Path: /jkcgbj-10/?prefix=_myobj 2022-09-09T22:31:04.164 DEBUG:boto:Data: 2022-09-09T22:31:04.165 DEBUG:boto:Headers: {} 2022-09-09T22:31:04.166 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:31:04.167 DEBUG:boto:Port: 8001 2022-09-09T22:31:04.168 DEBUG:boto:Params: {} 2022-09-09T22:31:04.169 DEBUG:boto:Token: None 2022-09-09T22:31:04.170 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:04 GMT /jkcgbj-10/ 2022-09-09T22:31:04.171 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qZIXlLJxiYDO48Yt+bdhmCuS9S4= 2022-09-09T22:31:04.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qZIXlLJxiYDO48Yt+bdhmCuS9S4=', 'Content-Length': '0'} 2022-09-09T22:31:04.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000091282d1184b0e5e-00631bbea8-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:04.188 DEBUG:boto:b'jkcgbj-10_myobj1000false_myobj2022-09-09T22:30:44.054Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:04.191 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:04.192 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:04.193 DEBUG:boto:path=/jkcgbj-10/?prefix=_myobj 2022-09-09T22:31:04.194 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=_myobj 2022-09-09T22:31:04.195 DEBUG:boto:Method: GET 2022-09-09T22:31:04.196 DEBUG:boto:Path: /jkcgbj-10/?prefix=_myobj 2022-09-09T22:31:04.197 DEBUG:boto:Data: 2022-09-09T22:31:04.198 DEBUG:boto:Headers: {} 2022-09-09T22:31:04.199 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:04.200 DEBUG:boto:Port: 8000 2022-09-09T22:31:04.201 DEBUG:boto:Params: {} 2022-09-09T22:31:04.202 DEBUG:boto:Token: None 2022-09-09T22:31:04.203 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:04 GMT /jkcgbj-10/ 2022-09-09T22:31:04.203 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qZIXlLJxiYDO48Yt+bdhmCuS9S4= 2022-09-09T22:31:04.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qZIXlLJxiYDO48Yt+bdhmCuS9S4=', 'Content-Length': '0'} 2022-09-09T22:31:04.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009adb0d36db176148-00631bbea8-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:04.218 DEBUG:boto:b'jkcgbj-10_myobj1000false_myobj2022-09-09T22:30:44.054Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:04.219 DEBUG:boto:path=/jkcgbj-10/_myobj 2022-09-09T22:31:04.219 DEBUG:boto:auth_path=/jkcgbj-10/_myobj 2022-09-09T22:31:04.220 DEBUG:boto:Method: HEAD 2022-09-09T22:31:04.220 DEBUG:boto:Path: /jkcgbj-10/_myobj 2022-09-09T22:31:04.221 DEBUG:boto:Data: 2022-09-09T22:31:04.221 DEBUG:boto:Headers: {} 2022-09-09T22:31:04.223 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:31:04.224 DEBUG:boto:Port: 8001 2022-09-09T22:31:04.225 DEBUG:boto:Params: {} 2022-09-09T22:31:04.226 DEBUG:boto:Token: None 2022-09-09T22:31:04.227 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:04 GMT /jkcgbj-10/_myobj 2022-09-09T22:31:04.228 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FoU80mXFuUB1hFfr7aRv6pjOTVY= 2022-09-09T22:31:04.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FoU80mXFuUB1hFfr7aRv6pjOTVY=', 'Content-Length': '0'} 2022-09-09T22:31:04.235 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae7175c6fc13a0a5-00631bbea8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:04.237 DEBUG:boto:path=/jkcgbj-10/_myobj 2022-09-09T22:31:04.238 DEBUG:boto:auth_path=/jkcgbj-10/_myobj 2022-09-09T22:31:04.239 DEBUG:boto:Method: HEAD 2022-09-09T22:31:04.240 DEBUG:boto:Path: /jkcgbj-10/_myobj 2022-09-09T22:31:04.241 DEBUG:boto:Data: 2022-09-09T22:31:04.242 DEBUG:boto:Headers: {} 2022-09-09T22:31:04.243 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:04.244 DEBUG:boto:Port: 8000 2022-09-09T22:31:04.245 DEBUG:boto:Params: {} 2022-09-09T22:31:04.246 DEBUG:boto:Token: None 2022-09-09T22:31:04.247 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:04 GMT /jkcgbj-10/_myobj 2022-09-09T22:31:04.248 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FoU80mXFuUB1hFfr7aRv6pjOTVY= 2022-09-09T22:31:04.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FoU80mXFuUB1hFfr7aRv6pjOTVY=', 'Content-Length': '0'} 2022-09-09T22:31:04.254 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001393d2a84d8773f0-00631bbea8-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:04.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-09T22:31:04.257 DEBUG:boto:path=/jkcgbj-10/_myobj 2022-09-09T22:31:04.258 DEBUG:boto:auth_path=/jkcgbj-10/_myobj 2022-09-09T22:31:04.260 DEBUG:boto:Method: GET 2022-09-09T22:31:04.261 DEBUG:boto:Path: /jkcgbj-10/_myobj 2022-09-09T22:31:04.263 DEBUG:boto:Data: 2022-09-09T22:31:04.264 DEBUG:boto:Headers: {} 2022-09-09T22:31:04.265 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:31:04.267 DEBUG:boto:Port: 8001 2022-09-09T22:31:04.268 DEBUG:boto:Params: {} 2022-09-09T22:31:04.270 DEBUG:boto:Token: None 2022-09-09T22:31:04.271 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:04 GMT /jkcgbj-10/_myobj 2022-09-09T22:31:04.273 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:WTilV5s95Hu663xKug+vtn6b1e8= 2022-09-09T22:31:04.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:WTilV5s95Hu663xKug+vtn6b1e8=', 'Content-Length': '0'} 2022-09-09T22:31:04.286 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ea52e3ae267ef23-00631bbea8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:04.289 DEBUG:boto:path=/jkcgbj-10/_myobj 2022-09-09T22:31:04.291 DEBUG:boto:auth_path=/jkcgbj-10/_myobj 2022-09-09T22:31:04.293 DEBUG:boto:Method: GET 2022-09-09T22:31:04.294 DEBUG:boto:Path: /jkcgbj-10/_myobj 2022-09-09T22:31:04.295 DEBUG:boto:Data: 2022-09-09T22:31:04.296 DEBUG:boto:Headers: {} 2022-09-09T22:31:04.297 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:04.299 DEBUG:boto:Port: 8000 2022-09-09T22:31:04.300 DEBUG:boto:Params: {} 2022-09-09T22:31:04.301 DEBUG:boto:Token: None 2022-09-09T22:31:04.302 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:04 GMT /jkcgbj-10/_myobj 2022-09-09T22:31:04.303 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:WTilV5s95Hu663xKug+vtn6b1e8= 2022-09-09T22:31:04.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:WTilV5s95Hu663xKug+vtn6b1e8=', 'Content-Length': '0'} 2022-09-09T22:31:04.313 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000145437f278f99a19-00631bbea8-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:04.318 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:04.319 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:04.321 DEBUG:boto:path=/jkcgbj-10/?prefix=_myobj 2022-09-09T22:31:04.323 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=_myobj 2022-09-09T22:31:04.329 DEBUG:boto:Method: GET 2022-09-09T22:31:04.330 DEBUG:boto:Path: /jkcgbj-10/?prefix=_myobj 2022-09-09T22:31:04.331 DEBUG:boto:Data: 2022-09-09T22:31:04.331 DEBUG:boto:Headers: {} 2022-09-09T22:31:04.333 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:31:04.334 DEBUG:boto:Port: 8001 2022-09-09T22:31:04.335 DEBUG:boto:Params: {} 2022-09-09T22:31:04.336 DEBUG:boto:Token: None 2022-09-09T22:31:04.337 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:04 GMT /jkcgbj-10/ 2022-09-09T22:31:04.339 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qZIXlLJxiYDO48Yt+bdhmCuS9S4= 2022-09-09T22:31:04.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qZIXlLJxiYDO48Yt+bdhmCuS9S4=', 'Content-Length': '0'} 2022-09-09T22:31:04.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d79a05cda3818530-00631bbea8-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:04.353 DEBUG:boto:b'jkcgbj-10_myobj1000false_myobj2022-09-09T22:30:44.054Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:04.356 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:04.358 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:04.359 DEBUG:boto:path=/jkcgbj-10/?prefix=_myobj 2022-09-09T22:31:04.362 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=_myobj 2022-09-09T22:31:04.363 DEBUG:boto:Method: GET 2022-09-09T22:31:04.365 DEBUG:boto:Path: /jkcgbj-10/?prefix=_myobj 2022-09-09T22:31:04.366 DEBUG:boto:Data: 2022-09-09T22:31:04.368 DEBUG:boto:Headers: {} 2022-09-09T22:31:04.369 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:04.370 DEBUG:boto:Port: 8000 2022-09-09T22:31:04.371 DEBUG:boto:Params: {} 2022-09-09T22:31:04.373 DEBUG:boto:Token: None 2022-09-09T22:31:04.374 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:04 GMT /jkcgbj-10/ 2022-09-09T22:31:04.376 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qZIXlLJxiYDO48Yt+bdhmCuS9S4= 2022-09-09T22:31:04.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qZIXlLJxiYDO48Yt+bdhmCuS9S4=', 'Content-Length': '0'} 2022-09-09T22:31:04.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a84d453e30e2232-00631bbea8-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:04.388 DEBUG:boto:b'jkcgbj-10_myobj1000false_myobj2022-09-09T22:30:44.054Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:04.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:31:04.389 DEBUG:boto:path=/jkcgbj-10/myobj 2022-09-09T22:31:04.390 DEBUG:boto:auth_path=/jkcgbj-10/myobj 2022-09-09T22:31:04.391 DEBUG:boto:path=/jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:04.391 DEBUG:boto:auth_path=/jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:04.392 DEBUG:boto:Method: GET 2022-09-09T22:31:04.392 DEBUG:boto:Path: /jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:04.393 DEBUG:boto:Data: 2022-09-09T22:31:04.394 DEBUG:boto:Headers: {} 2022-09-09T22:31:04.394 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:31:04.395 DEBUG:boto:Port: 8001 2022-09-09T22:31:04.395 DEBUG:boto:Params: {} 2022-09-09T22:31:04.396 DEBUG:boto:Token: None 2022-09-09T22:31:04.396 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:04 GMT /jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:04.397 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:VtQ/V/N/PRXBLwqbViX5M2BbuQk= 2022-09-09T22:31:04.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:VtQ/V/N/PRXBLwqbViX5M2BbuQk=', 'Content-Length': '0'} 2022-09-09T22:31:04.406 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd040f3db6fbf0fe-00631bbea8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:04.408 DEBUG:boto:path=/jkcgbj-10/myobj 2022-09-09T22:31:04.412 DEBUG:boto:auth_path=/jkcgbj-10/myobj 2022-09-09T22:31:04.413 DEBUG:boto:path=/jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:04.414 DEBUG:boto:auth_path=/jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:04.415 DEBUG:boto:Method: GET 2022-09-09T22:31:04.416 DEBUG:boto:Path: /jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:04.417 DEBUG:boto:Data: 2022-09-09T22:31:04.417 DEBUG:boto:Headers: {} 2022-09-09T22:31:04.418 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:04.418 DEBUG:boto:Port: 8000 2022-09-09T22:31:04.421 DEBUG:boto:Params: {} 2022-09-09T22:31:04.423 DEBUG:boto:Token: None 2022-09-09T22:31:04.426 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:04 GMT /jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:04.427 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:VtQ/V/N/PRXBLwqbViX5M2BbuQk= 2022-09-09T22:31:04.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:VtQ/V/N/PRXBLwqbViX5M2BbuQk=', 'Content-Length': '0'} 2022-09-09T22:31:04.434 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003db5fd2df72a7be4-00631bbea8-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:04.435 DEBUG:boto:path=/jkcgbj-10/myobj 2022-09-09T22:31:04.436 DEBUG:boto:auth_path=/jkcgbj-10/myobj 2022-09-09T22:31:04.437 DEBUG:boto:path=/jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:04.437 DEBUG:boto:auth_path=/jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:04.438 DEBUG:boto:Method: HEAD 2022-09-09T22:31:04.438 DEBUG:boto:Path: /jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:04.439 DEBUG:boto:Data: 2022-09-09T22:31:04.440 DEBUG:boto:Headers: {} 2022-09-09T22:31:04.440 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:31:04.441 DEBUG:boto:Port: 8001 2022-09-09T22:31:04.442 DEBUG:boto:Params: {} 2022-09-09T22:31:04.442 DEBUG:boto:Token: None 2022-09-09T22:31:04.443 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:04 GMT /jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:04.444 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rnXTiPFVW+p6aFjm3NxuE9M//vc= 2022-09-09T22:31:04.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rnXTiPFVW+p6aFjm3NxuE9M//vc=', 'Content-Length': '0'} 2022-09-09T22:31:04.452 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9216f265f8e43d6-00631bbea8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:04.453 DEBUG:boto:path=/jkcgbj-10/myobj 2022-09-09T22:31:04.454 DEBUG:boto:auth_path=/jkcgbj-10/myobj 2022-09-09T22:31:04.454 DEBUG:boto:path=/jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:04.455 DEBUG:boto:auth_path=/jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:04.456 DEBUG:boto:Method: HEAD 2022-09-09T22:31:04.456 DEBUG:boto:Path: /jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:04.458 DEBUG:boto:Data: 2022-09-09T22:31:04.458 DEBUG:boto:Headers: {} 2022-09-09T22:31:04.459 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:04.459 DEBUG:boto:Port: 8000 2022-09-09T22:31:04.461 DEBUG:boto:Params: {} 2022-09-09T22:31:04.462 DEBUG:boto:Token: None 2022-09-09T22:31:04.464 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:04 GMT /jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:04.466 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rnXTiPFVW+p6aFjm3NxuE9M//vc= 2022-09-09T22:31:04.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rnXTiPFVW+p6aFjm3NxuE9M//vc=', 'Content-Length': '0'} 2022-09-09T22:31:04.476 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062597a99379b6859-00631bbea8-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:04.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:31:04.478 DEBUG:boto:path=/jkcgbj-10/myobj 2022-09-09T22:31:04.478 DEBUG:boto:auth_path=/jkcgbj-10/myobj 2022-09-09T22:31:04.479 DEBUG:boto:path=/jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:04.480 DEBUG:boto:auth_path=/jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:04.489 DEBUG:boto:Method: GET 2022-09-09T22:31:04.489 DEBUG:boto:Path: /jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:04.490 DEBUG:boto:Data: 2022-09-09T22:31:04.491 DEBUG:boto:Headers: {} 2022-09-09T22:31:04.491 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:31:04.492 DEBUG:boto:Port: 8001 2022-09-09T22:31:04.492 DEBUG:boto:Params: {} 2022-09-09T22:31:04.493 DEBUG:boto:Token: None 2022-09-09T22:31:04.494 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:04 GMT /jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:04.494 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:VtQ/V/N/PRXBLwqbViX5M2BbuQk= 2022-09-09T22:31:04.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:VtQ/V/N/PRXBLwqbViX5M2BbuQk=', 'Content-Length': '0'} 2022-09-09T22:31:04.505 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003cfd46573c35dda6-00631bbea8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:04.507 DEBUG:boto:path=/jkcgbj-10/myobj 2022-09-09T22:31:04.507 DEBUG:boto:auth_path=/jkcgbj-10/myobj 2022-09-09T22:31:04.508 DEBUG:boto:path=/jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:04.509 DEBUG:boto:auth_path=/jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:04.509 DEBUG:boto:Method: GET 2022-09-09T22:31:04.510 DEBUG:boto:Path: /jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:04.511 DEBUG:boto:Data: 2022-09-09T22:31:04.513 DEBUG:boto:Headers: {} 2022-09-09T22:31:04.514 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:04.515 DEBUG:boto:Port: 8000 2022-09-09T22:31:04.516 DEBUG:boto:Params: {} 2022-09-09T22:31:04.517 DEBUG:boto:Token: None 2022-09-09T22:31:04.518 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:04 GMT /jkcgbj-10/myobj?versionId=null 2022-09-09T22:31:04.519 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:VtQ/V/N/PRXBLwqbViX5M2BbuQk= 2022-09-09T22:31:04.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:VtQ/V/N/PRXBLwqbViX5M2BbuQk=', 'Content-Length': '0'} 2022-09-09T22:31:04.530 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa16f44ee9a9c77e-00631bbea8-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:04.534 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:04.535 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:04.536 DEBUG:boto:path=/jkcgbj-10/?prefix=myobj 2022-09-09T22:31:04.537 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=myobj 2022-09-09T22:31:04.539 DEBUG:boto:Method: GET 2022-09-09T22:31:04.541 DEBUG:boto:Path: /jkcgbj-10/?prefix=myobj 2022-09-09T22:31:04.542 DEBUG:boto:Data: 2022-09-09T22:31:04.543 DEBUG:boto:Headers: {} 2022-09-09T22:31:04.544 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:31:04.545 DEBUG:boto:Port: 8001 2022-09-09T22:31:04.546 DEBUG:boto:Params: {} 2022-09-09T22:31:04.560 DEBUG:boto:Token: None 2022-09-09T22:31:04.563 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:04 GMT /jkcgbj-10/ 2022-09-09T22:31:04.565 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qZIXlLJxiYDO48Yt+bdhmCuS9S4= 2022-09-09T22:31:04.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qZIXlLJxiYDO48Yt+bdhmCuS9S4=', 'Content-Length': '0'} 2022-09-09T22:31:04.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000566bb24e1760d3c9-00631bbea8-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:04.581 DEBUG:boto:b'jkcgbj-10myobj1000falsemyobj2022-09-09T22:30:43.986Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:04.583 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:04.584 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:04.585 DEBUG:boto:path=/jkcgbj-10/?prefix=myobj 2022-09-09T22:31:04.586 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=myobj 2022-09-09T22:31:04.589 DEBUG:boto:Method: GET 2022-09-09T22:31:04.602 DEBUG:boto:Path: /jkcgbj-10/?prefix=myobj 2022-09-09T22:31:04.603 DEBUG:boto:Data: 2022-09-09T22:31:04.604 DEBUG:boto:Headers: {} 2022-09-09T22:31:04.605 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:04.619 DEBUG:boto:Port: 8000 2022-09-09T22:31:04.620 DEBUG:boto:Params: {} 2022-09-09T22:31:04.622 DEBUG:boto:Token: None 2022-09-09T22:31:04.623 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:04 GMT /jkcgbj-10/ 2022-09-09T22:31:04.624 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qZIXlLJxiYDO48Yt+bdhmCuS9S4= 2022-09-09T22:31:04.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qZIXlLJxiYDO48Yt+bdhmCuS9S4=', 'Content-Length': '0'} 2022-09-09T22:31:04.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000d7c22ba6baa596-00631bbea8-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:04.635 DEBUG:boto:b'jkcgbj-10myobj1000falsemyobj2022-09-09T22:30:43.986Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:04.638 DEBUG:boto:path=/jkcgbj-10/myobj 2022-09-09T22:31:04.639 DEBUG:boto:auth_path=/jkcgbj-10/myobj 2022-09-09T22:31:04.640 DEBUG:boto:Method: HEAD 2022-09-09T22:31:04.641 DEBUG:boto:Path: /jkcgbj-10/myobj 2022-09-09T22:31:04.643 DEBUG:boto:Data: 2022-09-09T22:31:04.645 DEBUG:boto:Headers: {} 2022-09-09T22:31:04.647 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:31:04.648 DEBUG:boto:Port: 8001 2022-09-09T22:31:04.649 DEBUG:boto:Params: {} 2022-09-09T22:31:04.651 DEBUG:boto:Token: None 2022-09-09T22:31:04.652 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:04 GMT /jkcgbj-10/myobj 2022-09-09T22:31:04.654 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KZtfKn4G2dpsdRDFcGM7S4WtI/8= 2022-09-09T22:31:04.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KZtfKn4G2dpsdRDFcGM7S4WtI/8=', 'Content-Length': '0'} 2022-09-09T22:31:04.662 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e739e172f51cc38e-00631bbea8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:04.664 DEBUG:boto:path=/jkcgbj-10/myobj 2022-09-09T22:31:04.665 DEBUG:boto:auth_path=/jkcgbj-10/myobj 2022-09-09T22:31:04.666 DEBUG:boto:Method: HEAD 2022-09-09T22:31:04.667 DEBUG:boto:Path: /jkcgbj-10/myobj 2022-09-09T22:31:04.668 DEBUG:boto:Data: 2022-09-09T22:31:04.678 DEBUG:boto:Headers: {} 2022-09-09T22:31:04.686 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:04.687 DEBUG:boto:Port: 8000 2022-09-09T22:31:04.688 DEBUG:boto:Params: {} 2022-09-09T22:31:04.689 DEBUG:boto:Token: None 2022-09-09T22:31:04.690 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:31:04 GMT /jkcgbj-10/myobj 2022-09-09T22:31:04.691 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KZtfKn4G2dpsdRDFcGM7S4WtI/8= 2022-09-09T22:31:04.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KZtfKn4G2dpsdRDFcGM7S4WtI/8=', 'Content-Length': '0'} 2022-09-09T22:31:04.699 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000655a4033faf2db3c-00631bbea8-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:04.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:31:04.701 DEBUG:boto:path=/jkcgbj-10/myobj 2022-09-09T22:31:04.702 DEBUG:boto:auth_path=/jkcgbj-10/myobj 2022-09-09T22:31:04.703 DEBUG:boto:Method: GET 2022-09-09T22:31:04.703 DEBUG:boto:Path: /jkcgbj-10/myobj 2022-09-09T22:31:04.704 DEBUG:boto:Data: 2022-09-09T22:31:04.704 DEBUG:boto:Headers: {} 2022-09-09T22:31:04.705 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:31:04.706 DEBUG:boto:Port: 8001 2022-09-09T22:31:04.707 DEBUG:boto:Params: {} 2022-09-09T22:31:04.708 DEBUG:boto:Token: None 2022-09-09T22:31:04.709 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:04 GMT /jkcgbj-10/myobj 2022-09-09T22:31:04.711 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0xMQ8lPd0oyCj2OlzmUhnr0io2A= 2022-09-09T22:31:04.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0xMQ8lPd0oyCj2OlzmUhnr0io2A=', 'Content-Length': '0'} 2022-09-09T22:31:04.726 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a52edd7abae46b88-00631bbea8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:04.728 DEBUG:boto:path=/jkcgbj-10/myobj 2022-09-09T22:31:04.729 DEBUG:boto:auth_path=/jkcgbj-10/myobj 2022-09-09T22:31:04.730 DEBUG:boto:Method: GET 2022-09-09T22:31:04.731 DEBUG:boto:Path: /jkcgbj-10/myobj 2022-09-09T22:31:04.732 DEBUG:boto:Data: 2022-09-09T22:31:04.733 DEBUG:boto:Headers: {} 2022-09-09T22:31:04.735 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:04.736 DEBUG:boto:Port: 8000 2022-09-09T22:31:04.737 DEBUG:boto:Params: {} 2022-09-09T22:31:04.738 DEBUG:boto:Token: None 2022-09-09T22:31:04.739 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:04 GMT /jkcgbj-10/myobj 2022-09-09T22:31:04.740 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0xMQ8lPd0oyCj2OlzmUhnr0io2A= 2022-09-09T22:31:04.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0xMQ8lPd0oyCj2OlzmUhnr0io2A=', 'Content-Length': '0'} 2022-09-09T22:31:04.746 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000781bf3c705d50ff7-00631bbea8-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:31:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:04.751 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:04.753 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:04.754 DEBUG:boto:path=/jkcgbj-10/?prefix=myobj 2022-09-09T22:31:04.755 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=myobj 2022-09-09T22:31:04.757 DEBUG:boto:Method: GET 2022-09-09T22:31:04.758 DEBUG:boto:Path: /jkcgbj-10/?prefix=myobj 2022-09-09T22:31:04.759 DEBUG:boto:Data: 2022-09-09T22:31:04.760 DEBUG:boto:Headers: {} 2022-09-09T22:31:04.761 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:31:04.762 DEBUG:boto:Port: 8001 2022-09-09T22:31:04.763 DEBUG:boto:Params: {} 2022-09-09T22:31:04.765 DEBUG:boto:Token: None 2022-09-09T22:31:04.766 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:04 GMT /jkcgbj-10/ 2022-09-09T22:31:04.767 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qZIXlLJxiYDO48Yt+bdhmCuS9S4= 2022-09-09T22:31:04.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qZIXlLJxiYDO48Yt+bdhmCuS9S4=', 'Content-Length': '0'} 2022-09-09T22:31:04.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de7b51976a6990fd-00631bbea8-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:04.777 DEBUG:boto:b'jkcgbj-10myobj1000falsemyobj2022-09-09T22:30:43.986Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:04.778 DEBUG:boto:path=/jkcgbj-10/ 2022-09-09T22:31:04.779 DEBUG:boto:auth_path=/jkcgbj-10/ 2022-09-09T22:31:04.779 DEBUG:boto:path=/jkcgbj-10/?prefix=myobj 2022-09-09T22:31:04.780 DEBUG:boto:auth_path=/jkcgbj-10/?prefix=myobj 2022-09-09T22:31:04.781 DEBUG:boto:Method: GET 2022-09-09T22:31:04.783 DEBUG:boto:Path: /jkcgbj-10/?prefix=myobj 2022-09-09T22:31:04.788 DEBUG:boto:Data: 2022-09-09T22:31:04.790 DEBUG:boto:Headers: {} 2022-09-09T22:31:04.791 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:31:04.792 DEBUG:boto:Port: 8000 2022-09-09T22:31:04.793 DEBUG:boto:Params: {} 2022-09-09T22:31:04.795 DEBUG:boto:Token: None 2022-09-09T22:31:04.800 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:31:04 GMT /jkcgbj-10/ 2022-09-09T22:31:04.802 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qZIXlLJxiYDO48Yt+bdhmCuS9S4= 2022-09-09T22:31:04.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:31:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qZIXlLJxiYDO48Yt+bdhmCuS9S4=', 'Content-Length': '0'} 2022-09-09T22:31:04.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000901e1f4912b9c578-00631bbea8-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:31:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:31:04.814 DEBUG:boto:b'jkcgbj-10myobj1000falsemyobj2022-09-09T22:30:43.986Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:31:04.815 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-10 zones={test-zone3, test-zone1} 2022-09-09T22:31:04.815 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:31:04.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:31:04.816 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:31:04.891 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:31:04.892 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:31:04.967 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:31:04.966+0000 7fbe9fcf6dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:31:04.968 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000001.6.5, , ] 2022-09-09T22:31:04.968 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000001.6.5, , 00000000001.4.5, , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:31:09.975 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:31:09.966+0000 7fbe9fcf6dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:31:09.976 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000001.6.5, , ] 2022-09-09T22:31:09.976 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000001.6.5, , 00000000001.4.5, , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:31:14.974 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:31:14.970+0000 7fbe9fcf6dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:31:14.975 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000001.6.5, , ] 2022-09-09T22:31:14.975 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000001.6.5, , 00000000001.4.5, , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:31:19.978 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:31:19.974+0000 7fbe9fcf6dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:31:19.979 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000001.6.5, , ] 2022-09-09T22:31:19.979 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000001.6.5, , 00000000001.4.5, , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:31:24.975 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:31:24.974+0000 7fbe9fcf6dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:31:24.976 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000001.6.5, , ] 2022-09-09T22:31:24.980 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000001.6.5, , 00000000001.4.5, , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:31:29.978 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:31:29.978+0000 7fbe9fcf6dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:31:29.979 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000001.6.5, , ] 2022-09-09T22:31:29.980 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000001.6.5, , 00000000001.4.5, , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:31:34.981 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:31:34.982+0000 7fbe9fcf6dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:31:34.982 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000001.6.5, , ] 2022-09-09T22:31:34.982 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000001.6.5, , 00000000001.4.5, , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:31:39.984 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:31:39.986+0000 7fbe9fcf6dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:31:39.985 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000001.6.5, , ] 2022-09-09T22:31:39.985 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000001.6.5, , 00000000001.4.5, , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:31:44.987 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:31:44.986+0000 7fbe9fcf6dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:31:44.987 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000001.6.5, , ] 2022-09-09T22:31:44.988 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000001.6.5, , 00000000001.4.5, , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:31:49.989 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:31:49.990+0000 7fbe9fcf6dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:31:49.990 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000001.6.5, , , , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:31:49.990 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000001.6.5, , 00000000001.4.5, , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:31:54.992 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:31:54.994+0000 7fbe9fcf6dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:31:54.993 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000001.6.5, , , , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:31:54.993 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000001.6.5, , 00000000001.4.5, , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:31:59.995 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:31:59.998+0000 7fbe9fcf6dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:31:59.995 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000001.6.5, , , , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:31:59.996 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000001.6.5, , 00000000001.4.5, , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:32:04.998 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:32:04.998+0000 7fbe9fcf6dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:32:04.998 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000001.6.5, , , , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:32:04.999 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000001.6.5, , 00000000001.4.5, , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:32:10.003 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:32:10.002+0000 7fbe9fcf6dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:32:10.004 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000001.6.5, , , , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:32:10.004 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000001.6.5, , 00000000001.4.5, , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:32:15.004 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:32:15.005+0000 7fbe9fcf6dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:32:15.005 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000001.6.5, , , , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:32:15.005 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000001.6.5, , 00000000001.4.5, , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:32:20.008 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:32:20.004+0000 7fbe9fcf6dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:32:20.008 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000001.6.5, , , , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:32:20.009 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000001.6.5, , 00000000001.4.5, , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:32:25.011 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:32:25.007+0000 7fbe9fcf6dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:32:25.012 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000001.6.5, , , , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:32:25.013 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000001.6.5, , 00000000001.4.5, , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:32:30.014 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:32:30.015+0000 7fbe9fcf6dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:32:30.014 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000001.6.5, , , , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:32:30.015 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000001.6.5, , 00000000001.4.5, , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:32:35.018 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:32:35.014+0000 7fbe9fcf6dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:32:35.019 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000001.6.5, , , , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:32:35.019 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000001.6.5, , 00000000001.4.5, , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:32:40.020 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:32:40.018+0000 7fbe9fcf6dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:32:40.021 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000001.6.5, , , , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:32:40.021 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000001.6.5, , 00000000001.4.5, , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:32:45.024 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:32:45.021+0000 7fbe9fcf6dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:32:45.025 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000001.6.5, , , , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:32:45.025 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000001.6.5, , 00000000001.4.5, , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:32:50.027 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:32:50.021+0000 7fbe9fcf6dc0 1 bucket sync caught up with source: 2022-09-09T22:32:50.028 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000001.6.5, , 00000000001.4.5, , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:32:50.029 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000001.6.5, , 00000000001.4.5, , , , 00000000001.5.5, 00000000001.6.5, , ] 2022-09-09T22:32:50.029 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:32:50.021+0000 7fbe9fcf6dc0 0 bucket checkpoint complete 2022-09-09T22:32:50.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:32:50.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:32:50.057 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:32:50.109 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:32:50.110 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:32:50.183 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:32:50.169+0000 7f7ce344cdc0 1 waiting to reach incremental sync.. 2022-09-09T22:32:55.173 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:32:55.168+0000 7f7ce344cdc0 1 waiting to reach incremental sync.. 2022-09-09T22:33:00.176 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:33:00.172+0000 7f7ce344cdc0 1 bucket sync caught up with source: 2022-09-09T22:33:00.176 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000001.17.5, , 00000000001.12.5, , , , 00000000001.25.5, 00000000001.16.5, , ] 2022-09-09T22:33:00.177 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000001.17.5, , 00000000001.12.5, , , , 00000000001.25.5, 00000000001.16.5, , ] 2022-09-09T22:33:00.177 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:33:00.172+0000 7f7ce344cdc0 0 bucket checkpoint complete 2022-09-09T22:33:00.205 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-11 zones={test-zone1, test-zone2} 2022-09-09T22:33:00.207 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:00.208 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:00.208 DEBUG:boto:Method: HEAD 2022-09-09T22:33:00.209 DEBUG:boto:Path: /jkcgbj-11/ 2022-09-09T22:33:00.210 DEBUG:boto:Data: 2022-09-09T22:33:00.210 DEBUG:boto:Headers: {} 2022-09-09T22:33:00.212 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:00.213 DEBUG:boto:Port: 8000 2022-09-09T22:33:00.214 DEBUG:boto:Params: {} 2022-09-09T22:33:00.216 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T22:33:00.217 DEBUG:boto:Token: None 2022-09-09T22:33:00.218 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:00 GMT /jkcgbj-11/ 2022-09-09T22:33:00.219 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:95GMT9po+7CRyNlaOQsf51IYu5A= 2022-09-09T22:33:00.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:95GMT9po+7CRyNlaOQsf51IYu5A=', 'Content-Length': '0'} 2022-09-09T22:33:00.227 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc1832ace0f08ed8-00631bbf1c-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:33:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:00.229 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:00.230 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:00.232 DEBUG:boto:Method: HEAD 2022-09-09T22:33:00.234 DEBUG:boto:Path: /jkcgbj-11/ 2022-09-09T22:33:00.235 DEBUG:boto:Data: 2022-09-09T22:33:00.236 DEBUG:boto:Headers: {} 2022-09-09T22:33:00.237 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:00.238 DEBUG:boto:Port: 8000 2022-09-09T22:33:00.240 DEBUG:boto:Params: {} 2022-09-09T22:33:00.241 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T22:33:00.241 DEBUG:boto:Token: None 2022-09-09T22:33:00.242 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:00 GMT /jkcgbj-11/ 2022-09-09T22:33:00.243 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:95GMT9po+7CRyNlaOQsf51IYu5A= 2022-09-09T22:33:00.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:95GMT9po+7CRyNlaOQsf51IYu5A=', 'Content-Length': '0'} 2022-09-09T22:33:00.252 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aeff15bd9ddf01f7-00631bbf1c-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:33:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:00.252 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:33:00.253 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:00.254 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:00.254 DEBUG:boto:path=/jkcgbj-11/?versions&max-keys=999 2022-09-09T22:33:00.255 DEBUG:boto:auth_path=/jkcgbj-11/?versions&max-keys=999 2022-09-09T22:33:00.257 DEBUG:boto:Method: GET 2022-09-09T22:33:00.257 DEBUG:boto:Path: /jkcgbj-11/?versions&max-keys=999 2022-09-09T22:33:00.258 DEBUG:boto:Data: 2022-09-09T22:33:00.258 DEBUG:boto:Headers: {} 2022-09-09T22:33:00.260 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:00.260 DEBUG:boto:Port: 8000 2022-09-09T22:33:00.261 DEBUG:boto:Params: {} 2022-09-09T22:33:00.262 DEBUG:boto:Token: None 2022-09-09T22:33:00.263 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:00 GMT /jkcgbj-11/?versions 2022-09-09T22:33:00.263 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R0FxsRVMPN8Rn7WXz2lM063xzqw= 2022-09-09T22:33:00.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R0FxsRVMPN8Rn7WXz2lM063xzqw=', 'Content-Length': '0'} 2022-09-09T22:33:00.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad32b23181d995a7-00631bbf1c-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:00.272 DEBUG:boto:b'jkcgbj-11999false&nulltrue2022-09-09T22:30:44.517Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-09T22:30:44.453Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-09T22:30:44.357Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-09T22:30:44.302Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:00.274 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-09T22:33:00.275 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-09T22:33:00.275 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-09T22:33:00.276 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-09T22:33:00.276 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:33:00.277 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:00.278 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:00.278 DEBUG:boto:path=/jkcgbj-11/?versions&max-keys=999 2022-09-09T22:33:00.279 DEBUG:boto:auth_path=/jkcgbj-11/?versions&max-keys=999 2022-09-09T22:33:00.279 DEBUG:boto:Method: GET 2022-09-09T22:33:00.280 DEBUG:boto:Path: /jkcgbj-11/?versions&max-keys=999 2022-09-09T22:33:00.281 DEBUG:boto:Data: 2022-09-09T22:33:00.281 DEBUG:boto:Headers: {} 2022-09-09T22:33:00.282 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:00.282 DEBUG:boto:Port: 8000 2022-09-09T22:33:00.283 DEBUG:boto:Params: {} 2022-09-09T22:33:00.283 DEBUG:boto:Token: None 2022-09-09T22:33:00.284 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:00 GMT /jkcgbj-11/?versions 2022-09-09T22:33:00.285 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R0FxsRVMPN8Rn7WXz2lM063xzqw= 2022-09-09T22:33:00.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R0FxsRVMPN8Rn7WXz2lM063xzqw=', 'Content-Length': '0'} 2022-09-09T22:33:00.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd0510263abe344c-00631bbf1c-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:00.292 DEBUG:boto:b'jkcgbj-11999false&nulltrue2022-09-09T22:30:44.517Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-09T22:30:44.453Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-09T22:30:44.357Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-09T22:30:44.302Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:00.293 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-09T22:33:00.294 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-09T22:33:00.294 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-09T22:33:00.295 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-09T22:33:00.296 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:00.297 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:00.298 DEBUG:boto:path=/jkcgbj-11/?versions&max-keys=999 2022-09-09T22:33:00.298 DEBUG:boto:auth_path=/jkcgbj-11/?versions&max-keys=999 2022-09-09T22:33:00.299 DEBUG:boto:Method: GET 2022-09-09T22:33:00.299 DEBUG:boto:Path: /jkcgbj-11/?versions&max-keys=999 2022-09-09T22:33:00.300 DEBUG:boto:Data: 2022-09-09T22:33:00.301 DEBUG:boto:Headers: {} 2022-09-09T22:33:00.302 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:00.303 DEBUG:boto:Port: 8000 2022-09-09T22:33:00.303 DEBUG:boto:Params: {} 2022-09-09T22:33:00.305 DEBUG:boto:Token: None 2022-09-09T22:33:00.305 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:00 GMT /jkcgbj-11/?versions 2022-09-09T22:33:00.306 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R0FxsRVMPN8Rn7WXz2lM063xzqw= 2022-09-09T22:33:00.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R0FxsRVMPN8Rn7WXz2lM063xzqw=', 'Content-Length': '0'} 2022-09-09T22:33:00.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5da710bb9004722-00631bbf1c-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:00.317 DEBUG:boto:b'jkcgbj-11999false&nulltrue2022-09-09T22:30:44.517Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-09T22:30:44.453Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-09T22:30:44.357Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-09T22:30:44.302Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:00.319 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:00.319 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:00.320 DEBUG:boto:path=/jkcgbj-11/?versions&max-keys=999 2022-09-09T22:33:00.320 DEBUG:boto:auth_path=/jkcgbj-11/?versions&max-keys=999 2022-09-09T22:33:00.321 DEBUG:boto:Method: GET 2022-09-09T22:33:00.321 DEBUG:boto:Path: /jkcgbj-11/?versions&max-keys=999 2022-09-09T22:33:00.322 DEBUG:boto:Data: 2022-09-09T22:33:00.322 DEBUG:boto:Headers: {} 2022-09-09T22:33:00.323 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:00.323 DEBUG:boto:Port: 8000 2022-09-09T22:33:00.324 DEBUG:boto:Params: {} 2022-09-09T22:33:00.324 DEBUG:boto:Token: None 2022-09-09T22:33:00.325 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:00 GMT /jkcgbj-11/?versions 2022-09-09T22:33:00.326 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R0FxsRVMPN8Rn7WXz2lM063xzqw= 2022-09-09T22:33:00.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R0FxsRVMPN8Rn7WXz2lM063xzqw=', 'Content-Length': '0'} 2022-09-09T22:33:00.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4c9737d385815f1-00631bbf1c-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:00.333 DEBUG:boto:b'jkcgbj-11999false&nulltrue2022-09-09T22:30:44.517Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-09T22:30:44.453Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-09T22:30:44.357Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-09T22:30:44.302Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:00.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-09T22:33:00.336 DEBUG:boto:path=/jkcgbj-11/%26 2022-09-09T22:33:00.337 DEBUG:boto:auth_path=/jkcgbj-11/%26 2022-09-09T22:33:00.337 DEBUG:boto:path=/jkcgbj-11/%26?versionId=null 2022-09-09T22:33:00.338 DEBUG:boto:auth_path=/jkcgbj-11/%26?versionId=null 2022-09-09T22:33:00.339 DEBUG:boto:Method: GET 2022-09-09T22:33:00.340 DEBUG:boto:Path: /jkcgbj-11/%26?versionId=null 2022-09-09T22:33:00.340 DEBUG:boto:Data: 2022-09-09T22:33:00.341 DEBUG:boto:Headers: {} 2022-09-09T22:33:00.342 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:00.343 DEBUG:boto:Port: 8000 2022-09-09T22:33:00.344 DEBUG:boto:Params: {} 2022-09-09T22:33:00.345 DEBUG:boto:Token: None 2022-09-09T22:33:00.346 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:00 GMT /jkcgbj-11/%26?versionId=null 2022-09-09T22:33:00.347 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:xLE3+LMODC5dFsS+/mdJLDWEREA= 2022-09-09T22:33:00.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:xLE3+LMODC5dFsS+/mdJLDWEREA=', 'Content-Length': '0'} 2022-09-09T22:33:00.355 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e448b1ff3035099-00631bbf1c-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:00.394 DEBUG:boto:path=/jkcgbj-11/%26 2022-09-09T22:33:00.395 DEBUG:boto:auth_path=/jkcgbj-11/%26 2022-09-09T22:33:00.396 DEBUG:boto:path=/jkcgbj-11/%26?versionId=null 2022-09-09T22:33:00.397 DEBUG:boto:auth_path=/jkcgbj-11/%26?versionId=null 2022-09-09T22:33:00.397 DEBUG:boto:Method: GET 2022-09-09T22:33:00.398 DEBUG:boto:Path: /jkcgbj-11/%26?versionId=null 2022-09-09T22:33:00.398 DEBUG:boto:Data: 2022-09-09T22:33:00.399 DEBUG:boto:Headers: {} 2022-09-09T22:33:00.400 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:00.400 DEBUG:boto:Port: 8000 2022-09-09T22:33:00.401 DEBUG:boto:Params: {} 2022-09-09T22:33:00.402 DEBUG:boto:Token: None 2022-09-09T22:33:00.402 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:00 GMT /jkcgbj-11/%26?versionId=null 2022-09-09T22:33:00.403 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:xLE3+LMODC5dFsS+/mdJLDWEREA= 2022-09-09T22:33:00.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:xLE3+LMODC5dFsS+/mdJLDWEREA=', 'Content-Length': '0'} 2022-09-09T22:33:00.407 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5b76d5043f8404b-00631bbf1c-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:00.408 DEBUG:boto:path=/jkcgbj-11/%26 2022-09-09T22:33:00.409 DEBUG:boto:auth_path=/jkcgbj-11/%26 2022-09-09T22:33:00.409 DEBUG:boto:path=/jkcgbj-11/%26?versionId=null 2022-09-09T22:33:00.410 DEBUG:boto:auth_path=/jkcgbj-11/%26?versionId=null 2022-09-09T22:33:00.410 DEBUG:boto:Method: HEAD 2022-09-09T22:33:00.411 DEBUG:boto:Path: /jkcgbj-11/%26?versionId=null 2022-09-09T22:33:00.412 DEBUG:boto:Data: 2022-09-09T22:33:00.412 DEBUG:boto:Headers: {} 2022-09-09T22:33:00.413 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:00.414 DEBUG:boto:Port: 8000 2022-09-09T22:33:00.415 DEBUG:boto:Params: {} 2022-09-09T22:33:00.416 DEBUG:boto:Token: None 2022-09-09T22:33:00.418 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:00 GMT /jkcgbj-11/%26?versionId=null 2022-09-09T22:33:00.419 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:alOjvGdSj9T8S0VEv/qMJE4OPTw= 2022-09-09T22:33:00.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:alOjvGdSj9T8S0VEv/qMJE4OPTw=', 'Content-Length': '0'} 2022-09-09T22:33:00.424 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000069828c1fcd725412-00631bbf1c-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:00.426 DEBUG:boto:path=/jkcgbj-11/%26 2022-09-09T22:33:00.427 DEBUG:boto:auth_path=/jkcgbj-11/%26 2022-09-09T22:33:00.427 DEBUG:boto:path=/jkcgbj-11/%26?versionId=null 2022-09-09T22:33:00.428 DEBUG:boto:auth_path=/jkcgbj-11/%26?versionId=null 2022-09-09T22:33:00.429 DEBUG:boto:Method: HEAD 2022-09-09T22:33:00.430 DEBUG:boto:Path: /jkcgbj-11/%26?versionId=null 2022-09-09T22:33:00.431 DEBUG:boto:Data: 2022-09-09T22:33:00.433 DEBUG:boto:Headers: {} 2022-09-09T22:33:00.434 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:00.436 DEBUG:boto:Port: 8000 2022-09-09T22:33:00.437 DEBUG:boto:Params: {} 2022-09-09T22:33:00.438 DEBUG:boto:Token: None 2022-09-09T22:33:00.439 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:00 GMT /jkcgbj-11/%26?versionId=null 2022-09-09T22:33:00.440 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:alOjvGdSj9T8S0VEv/qMJE4OPTw= 2022-09-09T22:33:00.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:alOjvGdSj9T8S0VEv/qMJE4OPTw=', 'Content-Length': '0'} 2022-09-09T22:33:00.445 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc7cf2baec56a995-00631bbf1c-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:00.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-09T22:33:00.448 DEBUG:boto:path=/jkcgbj-11/%26 2022-09-09T22:33:00.449 DEBUG:boto:auth_path=/jkcgbj-11/%26 2022-09-09T22:33:00.450 DEBUG:boto:path=/jkcgbj-11/%26?versionId=null 2022-09-09T22:33:00.451 DEBUG:boto:auth_path=/jkcgbj-11/%26?versionId=null 2022-09-09T22:33:00.452 DEBUG:boto:Method: GET 2022-09-09T22:33:00.453 DEBUG:boto:Path: /jkcgbj-11/%26?versionId=null 2022-09-09T22:33:00.455 DEBUG:boto:Data: 2022-09-09T22:33:00.456 DEBUG:boto:Headers: {} 2022-09-09T22:33:00.457 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:00.458 DEBUG:boto:Port: 8000 2022-09-09T22:33:00.459 DEBUG:boto:Params: {} 2022-09-09T22:33:00.460 DEBUG:boto:Token: None 2022-09-09T22:33:00.461 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:00 GMT /jkcgbj-11/%26?versionId=null 2022-09-09T22:33:00.462 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:xLE3+LMODC5dFsS+/mdJLDWEREA= 2022-09-09T22:33:00.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:xLE3+LMODC5dFsS+/mdJLDWEREA=', 'Content-Length': '0'} 2022-09-09T22:33:00.467 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d33faf1e270861d8-00631bbf1c-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:00.469 DEBUG:boto:path=/jkcgbj-11/%26 2022-09-09T22:33:00.470 DEBUG:boto:auth_path=/jkcgbj-11/%26 2022-09-09T22:33:00.471 DEBUG:boto:path=/jkcgbj-11/%26?versionId=null 2022-09-09T22:33:00.472 DEBUG:boto:auth_path=/jkcgbj-11/%26?versionId=null 2022-09-09T22:33:00.473 DEBUG:boto:Method: GET 2022-09-09T22:33:00.474 DEBUG:boto:Path: /jkcgbj-11/%26?versionId=null 2022-09-09T22:33:00.475 DEBUG:boto:Data: 2022-09-09T22:33:00.476 DEBUG:boto:Headers: {} 2022-09-09T22:33:00.477 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:00.478 DEBUG:boto:Port: 8000 2022-09-09T22:33:00.479 DEBUG:boto:Params: {} 2022-09-09T22:33:00.480 DEBUG:boto:Token: None 2022-09-09T22:33:00.481 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:00 GMT /jkcgbj-11/%26?versionId=null 2022-09-09T22:33:00.482 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:xLE3+LMODC5dFsS+/mdJLDWEREA= 2022-09-09T22:33:00.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:xLE3+LMODC5dFsS+/mdJLDWEREA=', 'Content-Length': '0'} 2022-09-09T22:33:00.488 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005fec7972bc825d3b-00631bbf1c-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:00.530 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:00.532 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:00.534 DEBUG:boto:path=/jkcgbj-11/?prefix=%26 2022-09-09T22:33:00.551 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=%26 2022-09-09T22:33:00.553 DEBUG:boto:Method: GET 2022-09-09T22:33:00.554 DEBUG:boto:Path: /jkcgbj-11/?prefix=%26 2022-09-09T22:33:00.556 DEBUG:boto:Data: 2022-09-09T22:33:00.557 DEBUG:boto:Headers: {} 2022-09-09T22:33:00.558 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:00.560 DEBUG:boto:Port: 8000 2022-09-09T22:33:00.560 DEBUG:boto:Params: {} 2022-09-09T22:33:00.562 DEBUG:boto:Token: None 2022-09-09T22:33:00.563 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:00 GMT /jkcgbj-11/ 2022-09-09T22:33:00.564 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8FCpWH1bQKG4EVZ5P6fmsuDznB4= 2022-09-09T22:33:00.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8FCpWH1bQKG4EVZ5P6fmsuDznB4=', 'Content-Length': '0'} 2022-09-09T22:33:00.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb2be31e0feacdd9-00631bbf1c-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:00.573 DEBUG:boto:b'jkcgbj-11&1000false&2022-09-09T22:30:44.517Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:00.574 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:00.577 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:00.578 DEBUG:boto:path=/jkcgbj-11/?prefix=%26 2022-09-09T22:33:00.579 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=%26 2022-09-09T22:33:00.580 DEBUG:boto:Method: GET 2022-09-09T22:33:00.580 DEBUG:boto:Path: /jkcgbj-11/?prefix=%26 2022-09-09T22:33:00.581 DEBUG:boto:Data: 2022-09-09T22:33:00.581 DEBUG:boto:Headers: {} 2022-09-09T22:33:00.583 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:00.583 DEBUG:boto:Port: 8000 2022-09-09T22:33:00.584 DEBUG:boto:Params: {} 2022-09-09T22:33:00.585 DEBUG:boto:Token: None 2022-09-09T22:33:00.585 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:00 GMT /jkcgbj-11/ 2022-09-09T22:33:00.586 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8FCpWH1bQKG4EVZ5P6fmsuDznB4= 2022-09-09T22:33:00.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8FCpWH1bQKG4EVZ5P6fmsuDznB4=', 'Content-Length': '0'} 2022-09-09T22:33:00.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fbb188ba9a58123-00631bbf1c-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:00.596 DEBUG:boto:b'jkcgbj-11&1000false&2022-09-09T22:30:44.517Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:00.598 DEBUG:boto:path=/jkcgbj-11/%26 2022-09-09T22:33:00.599 DEBUG:boto:auth_path=/jkcgbj-11/%26 2022-09-09T22:33:00.600 DEBUG:boto:Method: HEAD 2022-09-09T22:33:00.601 DEBUG:boto:Path: /jkcgbj-11/%26 2022-09-09T22:33:00.602 DEBUG:boto:Data: 2022-09-09T22:33:00.603 DEBUG:boto:Headers: {} 2022-09-09T22:33:00.604 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:00.604 DEBUG:boto:Port: 8000 2022-09-09T22:33:00.605 DEBUG:boto:Params: {} 2022-09-09T22:33:00.606 DEBUG:boto:Token: None 2022-09-09T22:33:00.608 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:00 GMT /jkcgbj-11/%26 2022-09-09T22:33:00.609 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4X+7jXKvhxnLpfjqajcWmzTN4/M= 2022-09-09T22:33:00.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4X+7jXKvhxnLpfjqajcWmzTN4/M=', 'Content-Length': '0'} 2022-09-09T22:33:00.613 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096623505b987cb88-00631bbf1c-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:00.614 DEBUG:boto:path=/jkcgbj-11/%26 2022-09-09T22:33:00.614 DEBUG:boto:auth_path=/jkcgbj-11/%26 2022-09-09T22:33:00.615 DEBUG:boto:Method: HEAD 2022-09-09T22:33:00.616 DEBUG:boto:Path: /jkcgbj-11/%26 2022-09-09T22:33:00.616 DEBUG:boto:Data: 2022-09-09T22:33:00.617 DEBUG:boto:Headers: {} 2022-09-09T22:33:00.617 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:00.618 DEBUG:boto:Port: 8000 2022-09-09T22:33:00.619 DEBUG:boto:Params: {} 2022-09-09T22:33:00.619 DEBUG:boto:Token: None 2022-09-09T22:33:00.620 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:00 GMT /jkcgbj-11/%26 2022-09-09T22:33:00.620 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4X+7jXKvhxnLpfjqajcWmzTN4/M= 2022-09-09T22:33:00.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4X+7jXKvhxnLpfjqajcWmzTN4/M=', 'Content-Length': '0'} 2022-09-09T22:33:00.624 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a51e292829edfae-00631bbf1c-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:00.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-09T22:33:00.625 DEBUG:boto:path=/jkcgbj-11/%26 2022-09-09T22:33:00.626 DEBUG:boto:auth_path=/jkcgbj-11/%26 2022-09-09T22:33:00.626 DEBUG:boto:Method: GET 2022-09-09T22:33:00.627 DEBUG:boto:Path: /jkcgbj-11/%26 2022-09-09T22:33:00.627 DEBUG:boto:Data: 2022-09-09T22:33:00.628 DEBUG:boto:Headers: {} 2022-09-09T22:33:00.629 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:00.629 DEBUG:boto:Port: 8000 2022-09-09T22:33:00.630 DEBUG:boto:Params: {} 2022-09-09T22:33:00.630 DEBUG:boto:Token: None 2022-09-09T22:33:00.631 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:00 GMT /jkcgbj-11/%26 2022-09-09T22:33:00.632 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sRls4a9HCMCIHvq+8Pe44VcAmTQ= 2022-09-09T22:33:00.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sRls4a9HCMCIHvq+8Pe44VcAmTQ=', 'Content-Length': '0'} 2022-09-09T22:33:00.635 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d4c3cd9bb2de730-00631bbf1c-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:00.636 DEBUG:boto:path=/jkcgbj-11/%26 2022-09-09T22:33:00.637 DEBUG:boto:auth_path=/jkcgbj-11/%26 2022-09-09T22:33:00.637 DEBUG:boto:Method: GET 2022-09-09T22:33:00.638 DEBUG:boto:Path: /jkcgbj-11/%26 2022-09-09T22:33:00.639 DEBUG:boto:Data: 2022-09-09T22:33:00.639 DEBUG:boto:Headers: {} 2022-09-09T22:33:00.640 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:00.640 DEBUG:boto:Port: 8000 2022-09-09T22:33:00.641 DEBUG:boto:Params: {} 2022-09-09T22:33:00.641 DEBUG:boto:Token: None 2022-09-09T22:33:00.642 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:00 GMT /jkcgbj-11/%26 2022-09-09T22:33:00.643 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sRls4a9HCMCIHvq+8Pe44VcAmTQ= 2022-09-09T22:33:00.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sRls4a9HCMCIHvq+8Pe44VcAmTQ=', 'Content-Length': '0'} 2022-09-09T22:33:00.647 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ded8c8f7d98474c2-00631bbf1c-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:00.648 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:00.648 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:00.649 DEBUG:boto:path=/jkcgbj-11/?prefix=%26 2022-09-09T22:33:00.650 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=%26 2022-09-09T22:33:00.650 DEBUG:boto:Method: GET 2022-09-09T22:33:00.651 DEBUG:boto:Path: /jkcgbj-11/?prefix=%26 2022-09-09T22:33:00.651 DEBUG:boto:Data: 2022-09-09T22:33:00.652 DEBUG:boto:Headers: {} 2022-09-09T22:33:00.652 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:00.653 DEBUG:boto:Port: 8000 2022-09-09T22:33:00.653 DEBUG:boto:Params: {} 2022-09-09T22:33:00.654 DEBUG:boto:Token: None 2022-09-09T22:33:00.655 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:00 GMT /jkcgbj-11/ 2022-09-09T22:33:00.656 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8FCpWH1bQKG4EVZ5P6fmsuDznB4= 2022-09-09T22:33:00.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8FCpWH1bQKG4EVZ5P6fmsuDznB4=', 'Content-Length': '0'} 2022-09-09T22:33:00.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9c0eec2ebf59917-00631bbf1c-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:00.663 DEBUG:boto:b'jkcgbj-11&1000false&2022-09-09T22:30:44.517Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:00.664 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:00.665 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:00.666 DEBUG:boto:path=/jkcgbj-11/?prefix=%26 2022-09-09T22:33:00.666 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=%26 2022-09-09T22:33:00.667 DEBUG:boto:Method: GET 2022-09-09T22:33:00.667 DEBUG:boto:Path: /jkcgbj-11/?prefix=%26 2022-09-09T22:33:00.669 DEBUG:boto:Data: 2022-09-09T22:33:00.669 DEBUG:boto:Headers: {} 2022-09-09T22:33:00.670 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:00.670 DEBUG:boto:Port: 8000 2022-09-09T22:33:00.671 DEBUG:boto:Params: {} 2022-09-09T22:33:00.672 DEBUG:boto:Token: None 2022-09-09T22:33:00.673 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:00 GMT /jkcgbj-11/ 2022-09-09T22:33:00.673 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8FCpWH1bQKG4EVZ5P6fmsuDznB4= 2022-09-09T22:33:00.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8FCpWH1bQKG4EVZ5P6fmsuDznB4=', 'Content-Length': '0'} 2022-09-09T22:33:00.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad9c20f614022860-00631bbf1c-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:00.680 DEBUG:boto:b'jkcgbj-11&1000false&2022-09-09T22:30:44.517Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:00.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-09T22:33:00.682 DEBUG:boto:path=/jkcgbj-11/%3A 2022-09-09T22:33:00.682 DEBUG:boto:auth_path=/jkcgbj-11/%3A 2022-09-09T22:33:00.683 DEBUG:boto:path=/jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:00.684 DEBUG:boto:auth_path=/jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:00.685 DEBUG:boto:Method: GET 2022-09-09T22:33:00.685 DEBUG:boto:Path: /jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:00.686 DEBUG:boto:Data: 2022-09-09T22:33:00.686 DEBUG:boto:Headers: {} 2022-09-09T22:33:00.687 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:00.687 DEBUG:boto:Port: 8000 2022-09-09T22:33:00.688 DEBUG:boto:Params: {} 2022-09-09T22:33:00.689 DEBUG:boto:Token: None 2022-09-09T22:33:00.691 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:00 GMT /jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:00.692 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OgVXHKPLJgO0/NcuBmG/Cs3tqiQ= 2022-09-09T22:33:00.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OgVXHKPLJgO0/NcuBmG/Cs3tqiQ=', 'Content-Length': '0'} 2022-09-09T22:33:00.695 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea43e6c076b8f793-00631bbf1c-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:00.742 DEBUG:boto:path=/jkcgbj-11/%3A 2022-09-09T22:33:00.743 DEBUG:boto:auth_path=/jkcgbj-11/%3A 2022-09-09T22:33:00.744 DEBUG:boto:path=/jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:00.745 DEBUG:boto:auth_path=/jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:00.746 DEBUG:boto:Method: GET 2022-09-09T22:33:00.747 DEBUG:boto:Path: /jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:00.748 DEBUG:boto:Data: 2022-09-09T22:33:00.749 DEBUG:boto:Headers: {} 2022-09-09T22:33:00.750 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:00.752 DEBUG:boto:Port: 8000 2022-09-09T22:33:00.753 DEBUG:boto:Params: {} 2022-09-09T22:33:00.754 DEBUG:boto:Token: None 2022-09-09T22:33:00.756 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:00 GMT /jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:00.758 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OgVXHKPLJgO0/NcuBmG/Cs3tqiQ= 2022-09-09T22:33:00.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OgVXHKPLJgO0/NcuBmG/Cs3tqiQ=', 'Content-Length': '0'} 2022-09-09T22:33:00.763 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d40cfe31f1ee883-00631bbf1c-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:00.764 DEBUG:boto:path=/jkcgbj-11/%3A 2022-09-09T22:33:00.764 DEBUG:boto:auth_path=/jkcgbj-11/%3A 2022-09-09T22:33:00.765 DEBUG:boto:path=/jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:00.765 DEBUG:boto:auth_path=/jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:00.766 DEBUG:boto:Method: HEAD 2022-09-09T22:33:00.766 DEBUG:boto:Path: /jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:00.767 DEBUG:boto:Data: 2022-09-09T22:33:00.767 DEBUG:boto:Headers: {} 2022-09-09T22:33:00.767 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:00.768 DEBUG:boto:Port: 8000 2022-09-09T22:33:00.768 DEBUG:boto:Params: {} 2022-09-09T22:33:00.769 DEBUG:boto:Token: None 2022-09-09T22:33:00.771 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:00 GMT /jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:00.771 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:uc+wg3BABWkdulqZEWOr8n1FKIM= 2022-09-09T22:33:00.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:uc+wg3BABWkdulqZEWOr8n1FKIM=', 'Content-Length': '0'} 2022-09-09T22:33:00.774 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a9c1aff5af0a38c7-00631bbf1c-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:00.775 DEBUG:boto:path=/jkcgbj-11/%3A 2022-09-09T22:33:00.776 DEBUG:boto:auth_path=/jkcgbj-11/%3A 2022-09-09T22:33:00.777 DEBUG:boto:path=/jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:00.777 DEBUG:boto:auth_path=/jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:00.778 DEBUG:boto:Method: HEAD 2022-09-09T22:33:00.778 DEBUG:boto:Path: /jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:00.779 DEBUG:boto:Data: 2022-09-09T22:33:00.780 DEBUG:boto:Headers: {} 2022-09-09T22:33:00.781 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:00.781 DEBUG:boto:Port: 8000 2022-09-09T22:33:00.782 DEBUG:boto:Params: {} 2022-09-09T22:33:00.782 DEBUG:boto:Token: None 2022-09-09T22:33:00.783 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:00 GMT /jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:00.784 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:uc+wg3BABWkdulqZEWOr8n1FKIM= 2022-09-09T22:33:00.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:uc+wg3BABWkdulqZEWOr8n1FKIM=', 'Content-Length': '0'} 2022-09-09T22:33:00.789 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000057af32237f4e094c-00631bbf1c-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:00.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-09T22:33:00.792 DEBUG:boto:path=/jkcgbj-11/%3A 2022-09-09T22:33:00.793 DEBUG:boto:auth_path=/jkcgbj-11/%3A 2022-09-09T22:33:00.794 DEBUG:boto:path=/jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:00.795 DEBUG:boto:auth_path=/jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:00.796 DEBUG:boto:Method: GET 2022-09-09T22:33:00.797 DEBUG:boto:Path: /jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:00.798 DEBUG:boto:Data: 2022-09-09T22:33:00.799 DEBUG:boto:Headers: {} 2022-09-09T22:33:00.801 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:00.802 DEBUG:boto:Port: 8000 2022-09-09T22:33:00.803 DEBUG:boto:Params: {} 2022-09-09T22:33:00.804 DEBUG:boto:Token: None 2022-09-09T22:33:00.805 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:00 GMT /jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:00.806 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OgVXHKPLJgO0/NcuBmG/Cs3tqiQ= 2022-09-09T22:33:00.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OgVXHKPLJgO0/NcuBmG/Cs3tqiQ=', 'Content-Length': '0'} 2022-09-09T22:33:00.811 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce14bc362e516d7b-00631bbf1c-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:00.813 DEBUG:boto:path=/jkcgbj-11/%3A 2022-09-09T22:33:00.814 DEBUG:boto:auth_path=/jkcgbj-11/%3A 2022-09-09T22:33:00.817 DEBUG:boto:path=/jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:00.818 DEBUG:boto:auth_path=/jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:00.819 DEBUG:boto:Method: GET 2022-09-09T22:33:00.820 DEBUG:boto:Path: /jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:00.821 DEBUG:boto:Data: 2022-09-09T22:33:00.822 DEBUG:boto:Headers: {} 2022-09-09T22:33:00.824 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:00.824 DEBUG:boto:Port: 8000 2022-09-09T22:33:00.825 DEBUG:boto:Params: {} 2022-09-09T22:33:00.826 DEBUG:boto:Token: None 2022-09-09T22:33:00.826 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:00 GMT /jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:00.827 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OgVXHKPLJgO0/NcuBmG/Cs3tqiQ= 2022-09-09T22:33:00.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OgVXHKPLJgO0/NcuBmG/Cs3tqiQ=', 'Content-Length': '0'} 2022-09-09T22:33:00.831 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000929ad5079bc6cc96-00631bbf1c-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:00.878 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:00.879 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:00.880 DEBUG:boto:path=/jkcgbj-11/?prefix=%3A 2022-09-09T22:33:00.881 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=%3A 2022-09-09T22:33:00.882 DEBUG:boto:Method: GET 2022-09-09T22:33:00.882 DEBUG:boto:Path: /jkcgbj-11/?prefix=%3A 2022-09-09T22:33:00.883 DEBUG:boto:Data: 2022-09-09T22:33:00.883 DEBUG:boto:Headers: {} 2022-09-09T22:33:00.884 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:00.887 DEBUG:boto:Port: 8000 2022-09-09T22:33:00.888 DEBUG:boto:Params: {} 2022-09-09T22:33:00.889 DEBUG:boto:Token: None 2022-09-09T22:33:00.892 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:00 GMT /jkcgbj-11/ 2022-09-09T22:33:00.893 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8FCpWH1bQKG4EVZ5P6fmsuDznB4= 2022-09-09T22:33:00.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8FCpWH1bQKG4EVZ5P6fmsuDznB4=', 'Content-Length': '0'} 2022-09-09T22:33:00.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7a8c8035527199f-00631bbf1c-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:00.901 DEBUG:boto:b'jkcgbj-11:1000false:2022-09-09T22:30:44.453Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:00.903 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:00.903 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:00.904 DEBUG:boto:path=/jkcgbj-11/?prefix=%3A 2022-09-09T22:33:00.905 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=%3A 2022-09-09T22:33:00.906 DEBUG:boto:Method: GET 2022-09-09T22:33:00.906 DEBUG:boto:Path: /jkcgbj-11/?prefix=%3A 2022-09-09T22:33:00.907 DEBUG:boto:Data: 2022-09-09T22:33:00.907 DEBUG:boto:Headers: {} 2022-09-09T22:33:00.908 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:00.908 DEBUG:boto:Port: 8000 2022-09-09T22:33:00.909 DEBUG:boto:Params: {} 2022-09-09T22:33:00.909 DEBUG:boto:Token: None 2022-09-09T22:33:00.910 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:00 GMT /jkcgbj-11/ 2022-09-09T22:33:00.911 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8FCpWH1bQKG4EVZ5P6fmsuDznB4= 2022-09-09T22:33:00.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8FCpWH1bQKG4EVZ5P6fmsuDznB4=', 'Content-Length': '0'} 2022-09-09T22:33:00.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f66557e7156e11c-00631bbf1c-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:00.918 DEBUG:boto:b'jkcgbj-11:1000false:2022-09-09T22:30:44.453Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:00.919 DEBUG:boto:path=/jkcgbj-11/%3A 2022-09-09T22:33:00.920 DEBUG:boto:auth_path=/jkcgbj-11/%3A 2022-09-09T22:33:00.921 DEBUG:boto:Method: HEAD 2022-09-09T22:33:00.922 DEBUG:boto:Path: /jkcgbj-11/%3A 2022-09-09T22:33:00.923 DEBUG:boto:Data: 2022-09-09T22:33:00.923 DEBUG:boto:Headers: {} 2022-09-09T22:33:00.924 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:00.925 DEBUG:boto:Port: 8000 2022-09-09T22:33:00.925 DEBUG:boto:Params: {} 2022-09-09T22:33:00.927 DEBUG:boto:Token: None 2022-09-09T22:33:00.928 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:00 GMT /jkcgbj-11/%3A 2022-09-09T22:33:00.929 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JWa0LAQ7bP5ipccJ6Fxd3v/d7YM= 2022-09-09T22:33:00.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JWa0LAQ7bP5ipccJ6Fxd3v/d7YM=', 'Content-Length': '0'} 2022-09-09T22:33:00.934 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc07385b0bf0aff4-00631bbf1c-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:00.935 DEBUG:boto:path=/jkcgbj-11/%3A 2022-09-09T22:33:00.936 DEBUG:boto:auth_path=/jkcgbj-11/%3A 2022-09-09T22:33:00.936 DEBUG:boto:Method: HEAD 2022-09-09T22:33:00.937 DEBUG:boto:Path: /jkcgbj-11/%3A 2022-09-09T22:33:00.937 DEBUG:boto:Data: 2022-09-09T22:33:00.938 DEBUG:boto:Headers: {} 2022-09-09T22:33:00.939 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:00.940 DEBUG:boto:Port: 8000 2022-09-09T22:33:00.941 DEBUG:boto:Params: {} 2022-09-09T22:33:00.942 DEBUG:boto:Token: None 2022-09-09T22:33:00.943 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:00 GMT /jkcgbj-11/%3A 2022-09-09T22:33:00.943 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JWa0LAQ7bP5ipccJ6Fxd3v/d7YM= 2022-09-09T22:33:00.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JWa0LAQ7bP5ipccJ6Fxd3v/d7YM=', 'Content-Length': '0'} 2022-09-09T22:33:00.948 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e122cf6be452df5-00631bbf1c-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:00.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-09T22:33:00.949 DEBUG:boto:path=/jkcgbj-11/%3A 2022-09-09T22:33:00.950 DEBUG:boto:auth_path=/jkcgbj-11/%3A 2022-09-09T22:33:00.951 DEBUG:boto:Method: GET 2022-09-09T22:33:00.952 DEBUG:boto:Path: /jkcgbj-11/%3A 2022-09-09T22:33:00.952 DEBUG:boto:Data: 2022-09-09T22:33:00.953 DEBUG:boto:Headers: {} 2022-09-09T22:33:00.953 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:00.954 DEBUG:boto:Port: 8000 2022-09-09T22:33:00.955 DEBUG:boto:Params: {} 2022-09-09T22:33:00.955 DEBUG:boto:Token: None 2022-09-09T22:33:00.956 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:00 GMT /jkcgbj-11/%3A 2022-09-09T22:33:00.957 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:w3I2xRY/ctPvy38zmMCmNDH4rjo= 2022-09-09T22:33:00.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:00 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:w3I2xRY/ctPvy38zmMCmNDH4rjo=', 'Content-Length': '0'} 2022-09-09T22:33:00.961 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e793779366d4cacf-00631bbf1c-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.002 DEBUG:boto:path=/jkcgbj-11/%3A 2022-09-09T22:33:01.003 DEBUG:boto:auth_path=/jkcgbj-11/%3A 2022-09-09T22:33:01.006 DEBUG:boto:Method: GET 2022-09-09T22:33:01.008 DEBUG:boto:Path: /jkcgbj-11/%3A 2022-09-09T22:33:01.009 DEBUG:boto:Data: 2022-09-09T22:33:01.010 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.011 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:01.012 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.013 DEBUG:boto:Params: {} 2022-09-09T22:33:01.014 DEBUG:boto:Token: None 2022-09-09T22:33:01.015 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/%3A 2022-09-09T22:33:01.016 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:g9i4aZP6rwhPgYGt67/ZHJnMqD4= 2022-09-09T22:33:01.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:g9i4aZP6rwhPgYGt67/ZHJnMqD4=', 'Content-Length': '0'} 2022-09-09T22:33:01.021 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de4617a21e5e70f5-00631bbf1d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.023 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:01.024 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:01.025 DEBUG:boto:path=/jkcgbj-11/?prefix=%3A 2022-09-09T22:33:01.027 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=%3A 2022-09-09T22:33:01.028 DEBUG:boto:Method: GET 2022-09-09T22:33:01.029 DEBUG:boto:Path: /jkcgbj-11/?prefix=%3A 2022-09-09T22:33:01.030 DEBUG:boto:Data: 2022-09-09T22:33:01.031 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.032 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:01.033 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.034 DEBUG:boto:Params: {} 2022-09-09T22:33:01.035 DEBUG:boto:Token: None 2022-09-09T22:33:01.036 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/ 2022-09-09T22:33:01.037 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:uDoN5ZAJeR5MIIaBks4qtXifx9k= 2022-09-09T22:33:01.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:uDoN5ZAJeR5MIIaBks4qtXifx9k=', 'Content-Length': '0'} 2022-09-09T22:33:01.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8d6d3853c58edd1-00631bbf1d-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.047 DEBUG:boto:b'jkcgbj-11:1000false:2022-09-09T22:30:44.453Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:01.048 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:01.049 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:01.050 DEBUG:boto:path=/jkcgbj-11/?prefix=%3A 2022-09-09T22:33:01.051 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=%3A 2022-09-09T22:33:01.052 DEBUG:boto:Method: GET 2022-09-09T22:33:01.053 DEBUG:boto:Path: /jkcgbj-11/?prefix=%3A 2022-09-09T22:33:01.054 DEBUG:boto:Data: 2022-09-09T22:33:01.055 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.056 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:01.057 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.058 DEBUG:boto:Params: {} 2022-09-09T22:33:01.059 DEBUG:boto:Token: None 2022-09-09T22:33:01.060 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/ 2022-09-09T22:33:01.061 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:uDoN5ZAJeR5MIIaBks4qtXifx9k= 2022-09-09T22:33:01.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:uDoN5ZAJeR5MIIaBks4qtXifx9k=', 'Content-Length': '0'} 2022-09-09T22:33:01.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5b294f2c7d9dcbf-00631bbf1d-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.070 DEBUG:boto:b'jkcgbj-11:1000false:2022-09-09T22:30:44.453Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:01.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-09T22:33:01.072 DEBUG:boto:path=/jkcgbj-11/_myobj 2022-09-09T22:33:01.073 DEBUG:boto:auth_path=/jkcgbj-11/_myobj 2022-09-09T22:33:01.075 DEBUG:boto:path=/jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:01.076 DEBUG:boto:auth_path=/jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:01.077 DEBUG:boto:Method: GET 2022-09-09T22:33:01.078 DEBUG:boto:Path: /jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:01.079 DEBUG:boto:Data: 2022-09-09T22:33:01.080 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.081 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:01.082 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.083 DEBUG:boto:Params: {} 2022-09-09T22:33:01.085 DEBUG:boto:Token: None 2022-09-09T22:33:01.086 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:01.087 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UOXxlMC/33qM5AEhKiqym/e4Z2Y= 2022-09-09T22:33:01.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UOXxlMC/33qM5AEhKiqym/e4Z2Y=', 'Content-Length': '0'} 2022-09-09T22:33:01.093 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004ea5e08d9ddcea5-00631bbf1d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.094 DEBUG:boto:path=/jkcgbj-11/_myobj 2022-09-09T22:33:01.095 DEBUG:boto:auth_path=/jkcgbj-11/_myobj 2022-09-09T22:33:01.096 DEBUG:boto:path=/jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:01.097 DEBUG:boto:auth_path=/jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:01.098 DEBUG:boto:Method: GET 2022-09-09T22:33:01.098 DEBUG:boto:Path: /jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:01.099 DEBUG:boto:Data: 2022-09-09T22:33:01.100 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.101 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:01.101 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.102 DEBUG:boto:Params: {} 2022-09-09T22:33:01.102 DEBUG:boto:Token: None 2022-09-09T22:33:01.103 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:01.104 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UOXxlMC/33qM5AEhKiqym/e4Z2Y= 2022-09-09T22:33:01.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UOXxlMC/33qM5AEhKiqym/e4Z2Y=', 'Content-Length': '0'} 2022-09-09T22:33:01.108 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000002bb7cad3afed575-00631bbf1d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.150 DEBUG:boto:path=/jkcgbj-11/_myobj 2022-09-09T22:33:01.151 DEBUG:boto:auth_path=/jkcgbj-11/_myobj 2022-09-09T22:33:01.152 DEBUG:boto:path=/jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:01.153 DEBUG:boto:auth_path=/jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:01.154 DEBUG:boto:Method: HEAD 2022-09-09T22:33:01.155 DEBUG:boto:Path: /jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:01.155 DEBUG:boto:Data: 2022-09-09T22:33:01.156 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.156 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:01.157 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.158 DEBUG:boto:Params: {} 2022-09-09T22:33:01.158 DEBUG:boto:Token: None 2022-09-09T22:33:01.159 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:01.160 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ZQ71YPIJQez4LjmljlFllHyU+m4= 2022-09-09T22:33:01.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ZQ71YPIJQez4LjmljlFllHyU+m4=', 'Content-Length': '0'} 2022-09-09T22:33:01.164 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d287a5fe45341c7-00631bbf1d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.169 DEBUG:boto:path=/jkcgbj-11/_myobj 2022-09-09T22:33:01.170 DEBUG:boto:auth_path=/jkcgbj-11/_myobj 2022-09-09T22:33:01.171 DEBUG:boto:path=/jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:01.171 DEBUG:boto:auth_path=/jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:01.172 DEBUG:boto:Method: HEAD 2022-09-09T22:33:01.173 DEBUG:boto:Path: /jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:01.174 DEBUG:boto:Data: 2022-09-09T22:33:01.175 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.176 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:01.177 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.179 DEBUG:boto:Params: {} 2022-09-09T22:33:01.180 DEBUG:boto:Token: None 2022-09-09T22:33:01.181 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:01.181 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ZQ71YPIJQez4LjmljlFllHyU+m4= 2022-09-09T22:33:01.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ZQ71YPIJQez4LjmljlFllHyU+m4=', 'Content-Length': '0'} 2022-09-09T22:33:01.186 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000339bfce9bd8391e0-00631bbf1d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-09T22:33:01.187 DEBUG:boto:path=/jkcgbj-11/_myobj 2022-09-09T22:33:01.188 DEBUG:boto:auth_path=/jkcgbj-11/_myobj 2022-09-09T22:33:01.189 DEBUG:boto:path=/jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:01.190 DEBUG:boto:auth_path=/jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:01.190 DEBUG:boto:Method: GET 2022-09-09T22:33:01.191 DEBUG:boto:Path: /jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:01.192 DEBUG:boto:Data: 2022-09-09T22:33:01.192 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.193 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:01.194 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.195 DEBUG:boto:Params: {} 2022-09-09T22:33:01.196 DEBUG:boto:Token: None 2022-09-09T22:33:01.196 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:01.197 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UOXxlMC/33qM5AEhKiqym/e4Z2Y= 2022-09-09T22:33:01.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UOXxlMC/33qM5AEhKiqym/e4Z2Y=', 'Content-Length': '0'} 2022-09-09T22:33:01.201 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d109da29777fb367-00631bbf1d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.202 DEBUG:boto:path=/jkcgbj-11/_myobj 2022-09-09T22:33:01.202 DEBUG:boto:auth_path=/jkcgbj-11/_myobj 2022-09-09T22:33:01.203 DEBUG:boto:path=/jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:01.203 DEBUG:boto:auth_path=/jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:01.204 DEBUG:boto:Method: GET 2022-09-09T22:33:01.205 DEBUG:boto:Path: /jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:01.206 DEBUG:boto:Data: 2022-09-09T22:33:01.207 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.208 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:01.211 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.212 DEBUG:boto:Params: {} 2022-09-09T22:33:01.212 DEBUG:boto:Token: None 2022-09-09T22:33:01.213 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:01.214 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UOXxlMC/33qM5AEhKiqym/e4Z2Y= 2022-09-09T22:33:01.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UOXxlMC/33qM5AEhKiqym/e4Z2Y=', 'Content-Length': '0'} 2022-09-09T22:33:01.218 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f049e3e74506d17-00631bbf1d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.222 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:01.223 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:01.224 DEBUG:boto:path=/jkcgbj-11/?prefix=_myobj 2022-09-09T22:33:01.225 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=_myobj 2022-09-09T22:33:01.226 DEBUG:boto:Method: GET 2022-09-09T22:33:01.227 DEBUG:boto:Path: /jkcgbj-11/?prefix=_myobj 2022-09-09T22:33:01.228 DEBUG:boto:Data: 2022-09-09T22:33:01.228 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.230 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:01.231 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.232 DEBUG:boto:Params: {} 2022-09-09T22:33:01.233 DEBUG:boto:Token: None 2022-09-09T22:33:01.234 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/ 2022-09-09T22:33:01.235 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:uDoN5ZAJeR5MIIaBks4qtXifx9k= 2022-09-09T22:33:01.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:uDoN5ZAJeR5MIIaBks4qtXifx9k=', 'Content-Length': '0'} 2022-09-09T22:33:01.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c54c8de434c00ac5-00631bbf1d-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.242 DEBUG:boto:b'jkcgbj-11_myobj1000false_myobj2022-09-09T22:30:44.357Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:01.243 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:01.244 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:01.244 DEBUG:boto:path=/jkcgbj-11/?prefix=_myobj 2022-09-09T22:33:01.245 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=_myobj 2022-09-09T22:33:01.245 DEBUG:boto:Method: GET 2022-09-09T22:33:01.246 DEBUG:boto:Path: /jkcgbj-11/?prefix=_myobj 2022-09-09T22:33:01.247 DEBUG:boto:Data: 2022-09-09T22:33:01.248 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.249 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:01.251 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.252 DEBUG:boto:Params: {} 2022-09-09T22:33:01.254 DEBUG:boto:Token: None 2022-09-09T22:33:01.255 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/ 2022-09-09T22:33:01.256 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:uDoN5ZAJeR5MIIaBks4qtXifx9k= 2022-09-09T22:33:01.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:uDoN5ZAJeR5MIIaBks4qtXifx9k=', 'Content-Length': '0'} 2022-09-09T22:33:01.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f6359ccc9ac2db1-00631bbf1d-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.264 DEBUG:boto:b'jkcgbj-11_myobj1000false_myobj2022-09-09T22:30:44.357Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:01.265 DEBUG:boto:path=/jkcgbj-11/_myobj 2022-09-09T22:33:01.265 DEBUG:boto:auth_path=/jkcgbj-11/_myobj 2022-09-09T22:33:01.266 DEBUG:boto:Method: HEAD 2022-09-09T22:33:01.267 DEBUG:boto:Path: /jkcgbj-11/_myobj 2022-09-09T22:33:01.270 DEBUG:boto:Data: 2022-09-09T22:33:01.271 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.271 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:01.272 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.272 DEBUG:boto:Params: {} 2022-09-09T22:33:01.273 DEBUG:boto:Token: None 2022-09-09T22:33:01.273 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/_myobj 2022-09-09T22:33:01.274 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:AnfXhETElsEKI0UmLHVqS9qzsRc= 2022-09-09T22:33:01.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:AnfXhETElsEKI0UmLHVqS9qzsRc=', 'Content-Length': '0'} 2022-09-09T22:33:01.278 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b75bd5ccfef67d86-00631bbf1d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.278 DEBUG:boto:path=/jkcgbj-11/_myobj 2022-09-09T22:33:01.279 DEBUG:boto:auth_path=/jkcgbj-11/_myobj 2022-09-09T22:33:01.280 DEBUG:boto:Method: HEAD 2022-09-09T22:33:01.280 DEBUG:boto:Path: /jkcgbj-11/_myobj 2022-09-09T22:33:01.281 DEBUG:boto:Data: 2022-09-09T22:33:01.281 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.282 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:01.282 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.283 DEBUG:boto:Params: {} 2022-09-09T22:33:01.283 DEBUG:boto:Token: None 2022-09-09T22:33:01.284 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/_myobj 2022-09-09T22:33:01.285 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:AnfXhETElsEKI0UmLHVqS9qzsRc= 2022-09-09T22:33:01.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:AnfXhETElsEKI0UmLHVqS9qzsRc=', 'Content-Length': '0'} 2022-09-09T22:33:01.288 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008806b5eac10af2f-00631bbf1d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-09T22:33:01.289 DEBUG:boto:path=/jkcgbj-11/_myobj 2022-09-09T22:33:01.290 DEBUG:boto:auth_path=/jkcgbj-11/_myobj 2022-09-09T22:33:01.291 DEBUG:boto:Method: GET 2022-09-09T22:33:01.291 DEBUG:boto:Path: /jkcgbj-11/_myobj 2022-09-09T22:33:01.292 DEBUG:boto:Data: 2022-09-09T22:33:01.292 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.293 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:01.293 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.294 DEBUG:boto:Params: {} 2022-09-09T22:33:01.295 DEBUG:boto:Token: None 2022-09-09T22:33:01.295 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/_myobj 2022-09-09T22:33:01.296 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Vh+5z4ZqLOod3ZzuKVam21ANLL8= 2022-09-09T22:33:01.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Vh+5z4ZqLOod3ZzuKVam21ANLL8=', 'Content-Length': '0'} 2022-09-09T22:33:01.300 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e8ce8fec7f5f60cc-00631bbf1d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.342 DEBUG:boto:path=/jkcgbj-11/_myobj 2022-09-09T22:33:01.343 DEBUG:boto:auth_path=/jkcgbj-11/_myobj 2022-09-09T22:33:01.344 DEBUG:boto:Method: GET 2022-09-09T22:33:01.345 DEBUG:boto:Path: /jkcgbj-11/_myobj 2022-09-09T22:33:01.345 DEBUG:boto:Data: 2022-09-09T22:33:01.346 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.347 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:01.348 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.349 DEBUG:boto:Params: {} 2022-09-09T22:33:01.350 DEBUG:boto:Token: None 2022-09-09T22:33:01.351 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/_myobj 2022-09-09T22:33:01.352 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Vh+5z4ZqLOod3ZzuKVam21ANLL8= 2022-09-09T22:33:01.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Vh+5z4ZqLOod3ZzuKVam21ANLL8=', 'Content-Length': '0'} 2022-09-09T22:33:01.356 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000baf416bb723351a7-00631bbf1d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.357 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:01.358 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:01.358 DEBUG:boto:path=/jkcgbj-11/?prefix=_myobj 2022-09-09T22:33:01.359 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=_myobj 2022-09-09T22:33:01.359 DEBUG:boto:Method: GET 2022-09-09T22:33:01.360 DEBUG:boto:Path: /jkcgbj-11/?prefix=_myobj 2022-09-09T22:33:01.361 DEBUG:boto:Data: 2022-09-09T22:33:01.362 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.362 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:01.363 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.364 DEBUG:boto:Params: {} 2022-09-09T22:33:01.365 DEBUG:boto:Token: None 2022-09-09T22:33:01.366 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/ 2022-09-09T22:33:01.368 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:uDoN5ZAJeR5MIIaBks4qtXifx9k= 2022-09-09T22:33:01.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:uDoN5ZAJeR5MIIaBks4qtXifx9k=', 'Content-Length': '0'} 2022-09-09T22:33:01.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029b6062db3c7bfbb-00631bbf1d-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.376 DEBUG:boto:b'jkcgbj-11_myobj1000false_myobj2022-09-09T22:30:44.357Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:01.377 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:01.378 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:01.378 DEBUG:boto:path=/jkcgbj-11/?prefix=_myobj 2022-09-09T22:33:01.379 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=_myobj 2022-09-09T22:33:01.380 DEBUG:boto:Method: GET 2022-09-09T22:33:01.380 DEBUG:boto:Path: /jkcgbj-11/?prefix=_myobj 2022-09-09T22:33:01.381 DEBUG:boto:Data: 2022-09-09T22:33:01.382 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.382 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:01.383 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.384 DEBUG:boto:Params: {} 2022-09-09T22:33:01.385 DEBUG:boto:Token: None 2022-09-09T22:33:01.385 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/ 2022-09-09T22:33:01.386 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:uDoN5ZAJeR5MIIaBks4qtXifx9k= 2022-09-09T22:33:01.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:uDoN5ZAJeR5MIIaBks4qtXifx9k=', 'Content-Length': '0'} 2022-09-09T22:33:01.392 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009720b0e3e9fa3477-00631bbf1d-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.393 DEBUG:boto:b'jkcgbj-11_myobj1000false_myobj2022-09-09T22:30:44.357Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:01.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:33:01.395 DEBUG:boto:path=/jkcgbj-11/myobj 2022-09-09T22:33:01.395 DEBUG:boto:auth_path=/jkcgbj-11/myobj 2022-09-09T22:33:01.396 DEBUG:boto:path=/jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:01.397 DEBUG:boto:auth_path=/jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:01.397 DEBUG:boto:Method: GET 2022-09-09T22:33:01.398 DEBUG:boto:Path: /jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:01.398 DEBUG:boto:Data: 2022-09-09T22:33:01.399 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.400 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:01.401 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.401 DEBUG:boto:Params: {} 2022-09-09T22:33:01.402 DEBUG:boto:Token: None 2022-09-09T22:33:01.403 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:01.404 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ebInpwWAs1McRevEI9+2KqgfH78= 2022-09-09T22:33:01.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ebInpwWAs1McRevEI9+2KqgfH78=', 'Content-Length': '0'} 2022-09-09T22:33:01.408 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000177fa75f1d8c5836-00631bbf1d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.411 DEBUG:boto:path=/jkcgbj-11/myobj 2022-09-09T22:33:01.411 DEBUG:boto:auth_path=/jkcgbj-11/myobj 2022-09-09T22:33:01.412 DEBUG:boto:path=/jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:01.412 DEBUG:boto:auth_path=/jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:01.413 DEBUG:boto:Method: GET 2022-09-09T22:33:01.414 DEBUG:boto:Path: /jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:01.415 DEBUG:boto:Data: 2022-09-09T22:33:01.415 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.416 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:01.416 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.417 DEBUG:boto:Params: {} 2022-09-09T22:33:01.418 DEBUG:boto:Token: None 2022-09-09T22:33:01.418 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:01.419 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ebInpwWAs1McRevEI9+2KqgfH78= 2022-09-09T22:33:01.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ebInpwWAs1McRevEI9+2KqgfH78=', 'Content-Length': '0'} 2022-09-09T22:33:01.423 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a8a8fbea39c449d4-00631bbf1d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.424 DEBUG:boto:path=/jkcgbj-11/myobj 2022-09-09T22:33:01.425 DEBUG:boto:auth_path=/jkcgbj-11/myobj 2022-09-09T22:33:01.425 DEBUG:boto:path=/jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:01.426 DEBUG:boto:auth_path=/jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:01.426 DEBUG:boto:Method: HEAD 2022-09-09T22:33:01.427 DEBUG:boto:Path: /jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:01.427 DEBUG:boto:Data: 2022-09-09T22:33:01.428 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.428 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:01.429 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.430 DEBUG:boto:Params: {} 2022-09-09T22:33:01.431 DEBUG:boto:Token: None 2022-09-09T22:33:01.432 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:01.432 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:T+hHTOdUk7VBjI9x8ukTVo78xq0= 2022-09-09T22:33:01.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:T+hHTOdUk7VBjI9x8ukTVo78xq0=', 'Content-Length': '0'} 2022-09-09T22:33:01.436 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000003da2244bc973ae6-00631bbf1d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.437 DEBUG:boto:path=/jkcgbj-11/myobj 2022-09-09T22:33:01.438 DEBUG:boto:auth_path=/jkcgbj-11/myobj 2022-09-09T22:33:01.439 DEBUG:boto:path=/jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:01.439 DEBUG:boto:auth_path=/jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:01.440 DEBUG:boto:Method: HEAD 2022-09-09T22:33:01.440 DEBUG:boto:Path: /jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:01.441 DEBUG:boto:Data: 2022-09-09T22:33:01.442 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.443 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:01.444 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.444 DEBUG:boto:Params: {} 2022-09-09T22:33:01.445 DEBUG:boto:Token: None 2022-09-09T22:33:01.446 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:01.447 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:T+hHTOdUk7VBjI9x8ukTVo78xq0= 2022-09-09T22:33:01.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:T+hHTOdUk7VBjI9x8ukTVo78xq0=', 'Content-Length': '0'} 2022-09-09T22:33:01.451 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d34a9c7641a546f9-00631bbf1d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:33:01.454 DEBUG:boto:path=/jkcgbj-11/myobj 2022-09-09T22:33:01.455 DEBUG:boto:auth_path=/jkcgbj-11/myobj 2022-09-09T22:33:01.456 DEBUG:boto:path=/jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:01.456 DEBUG:boto:auth_path=/jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:01.457 DEBUG:boto:Method: GET 2022-09-09T22:33:01.459 DEBUG:boto:Path: /jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:01.459 DEBUG:boto:Data: 2022-09-09T22:33:01.460 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.461 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:01.461 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.462 DEBUG:boto:Params: {} 2022-09-09T22:33:01.463 DEBUG:boto:Token: None 2022-09-09T22:33:01.464 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:01.464 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ebInpwWAs1McRevEI9+2KqgfH78= 2022-09-09T22:33:01.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ebInpwWAs1McRevEI9+2KqgfH78=', 'Content-Length': '0'} 2022-09-09T22:33:01.470 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005358eb839839ba99-00631bbf1d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.510 DEBUG:boto:path=/jkcgbj-11/myobj 2022-09-09T22:33:01.511 DEBUG:boto:auth_path=/jkcgbj-11/myobj 2022-09-09T22:33:01.512 DEBUG:boto:path=/jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:01.513 DEBUG:boto:auth_path=/jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:01.514 DEBUG:boto:Method: GET 2022-09-09T22:33:01.516 DEBUG:boto:Path: /jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:01.517 DEBUG:boto:Data: 2022-09-09T22:33:01.518 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.519 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:01.520 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.521 DEBUG:boto:Params: {} 2022-09-09T22:33:01.522 DEBUG:boto:Token: None 2022-09-09T22:33:01.524 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:01.526 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ebInpwWAs1McRevEI9+2KqgfH78= 2022-09-09T22:33:01.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ebInpwWAs1McRevEI9+2KqgfH78=', 'Content-Length': '0'} 2022-09-09T22:33:01.530 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079f81f78b9d5c954-00631bbf1d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.531 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:01.532 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:01.533 DEBUG:boto:path=/jkcgbj-11/?prefix=myobj 2022-09-09T22:33:01.533 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=myobj 2022-09-09T22:33:01.534 DEBUG:boto:Method: GET 2022-09-09T22:33:01.534 DEBUG:boto:Path: /jkcgbj-11/?prefix=myobj 2022-09-09T22:33:01.535 DEBUG:boto:Data: 2022-09-09T22:33:01.536 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.536 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:01.537 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.537 DEBUG:boto:Params: {} 2022-09-09T22:33:01.538 DEBUG:boto:Token: None 2022-09-09T22:33:01.538 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/ 2022-09-09T22:33:01.539 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:uDoN5ZAJeR5MIIaBks4qtXifx9k= 2022-09-09T22:33:01.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:uDoN5ZAJeR5MIIaBks4qtXifx9k=', 'Content-Length': '0'} 2022-09-09T22:33:01.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f039b6a27927bb04-00631bbf1d-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.546 DEBUG:boto:b'jkcgbj-11myobj1000falsemyobj2022-09-09T22:30:44.302Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:01.547 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:01.548 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:01.548 DEBUG:boto:path=/jkcgbj-11/?prefix=myobj 2022-09-09T22:33:01.549 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=myobj 2022-09-09T22:33:01.549 DEBUG:boto:Method: GET 2022-09-09T22:33:01.550 DEBUG:boto:Path: /jkcgbj-11/?prefix=myobj 2022-09-09T22:33:01.551 DEBUG:boto:Data: 2022-09-09T22:33:01.551 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.552 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:01.553 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.553 DEBUG:boto:Params: {} 2022-09-09T22:33:01.554 DEBUG:boto:Token: None 2022-09-09T22:33:01.555 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/ 2022-09-09T22:33:01.556 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:uDoN5ZAJeR5MIIaBks4qtXifx9k= 2022-09-09T22:33:01.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:uDoN5ZAJeR5MIIaBks4qtXifx9k=', 'Content-Length': '0'} 2022-09-09T22:33:01.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000071d5c8894d6e1e7-00631bbf1d-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.616 DEBUG:boto:b'jkcgbj-11myobj1000falsemyobj2022-09-09T22:30:44.302Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:01.617 DEBUG:boto:path=/jkcgbj-11/myobj 2022-09-09T22:33:01.618 DEBUG:boto:auth_path=/jkcgbj-11/myobj 2022-09-09T22:33:01.619 DEBUG:boto:Method: HEAD 2022-09-09T22:33:01.621 DEBUG:boto:Path: /jkcgbj-11/myobj 2022-09-09T22:33:01.621 DEBUG:boto:Data: 2022-09-09T22:33:01.622 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.622 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:01.623 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.623 DEBUG:boto:Params: {} 2022-09-09T22:33:01.624 DEBUG:boto:Token: None 2022-09-09T22:33:01.625 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/myobj 2022-09-09T22:33:01.626 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7ZF4ge0S3OKKA4V6RADFidPJK8Q= 2022-09-09T22:33:01.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7ZF4ge0S3OKKA4V6RADFidPJK8Q=', 'Content-Length': '0'} 2022-09-09T22:33:01.630 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e67f6ee9d6b25ee-00631bbf1d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.631 DEBUG:boto:path=/jkcgbj-11/myobj 2022-09-09T22:33:01.632 DEBUG:boto:auth_path=/jkcgbj-11/myobj 2022-09-09T22:33:01.633 DEBUG:boto:Method: HEAD 2022-09-09T22:33:01.633 DEBUG:boto:Path: /jkcgbj-11/myobj 2022-09-09T22:33:01.634 DEBUG:boto:Data: 2022-09-09T22:33:01.635 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.635 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:01.636 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.636 DEBUG:boto:Params: {} 2022-09-09T22:33:01.637 DEBUG:boto:Token: None 2022-09-09T22:33:01.638 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/myobj 2022-09-09T22:33:01.638 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7ZF4ge0S3OKKA4V6RADFidPJK8Q= 2022-09-09T22:33:01.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7ZF4ge0S3OKKA4V6RADFidPJK8Q=', 'Content-Length': '0'} 2022-09-09T22:33:01.642 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013b9f5605c51a547-00631bbf1d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:33:01.643 DEBUG:boto:path=/jkcgbj-11/myobj 2022-09-09T22:33:01.644 DEBUG:boto:auth_path=/jkcgbj-11/myobj 2022-09-09T22:33:01.644 DEBUG:boto:Method: GET 2022-09-09T22:33:01.645 DEBUG:boto:Path: /jkcgbj-11/myobj 2022-09-09T22:33:01.646 DEBUG:boto:Data: 2022-09-09T22:33:01.646 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.647 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:01.648 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.648 DEBUG:boto:Params: {} 2022-09-09T22:33:01.649 DEBUG:boto:Token: None 2022-09-09T22:33:01.650 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/myobj 2022-09-09T22:33:01.650 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Xqq8ZNTsj6ik8q/KHhBOcWW+D9w= 2022-09-09T22:33:01.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Xqq8ZNTsj6ik8q/KHhBOcWW+D9w=', 'Content-Length': '0'} 2022-09-09T22:33:01.654 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001b9d2224e3e52bab-00631bbf1d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.655 DEBUG:boto:path=/jkcgbj-11/myobj 2022-09-09T22:33:01.656 DEBUG:boto:auth_path=/jkcgbj-11/myobj 2022-09-09T22:33:01.656 DEBUG:boto:Method: GET 2022-09-09T22:33:01.657 DEBUG:boto:Path: /jkcgbj-11/myobj 2022-09-09T22:33:01.657 DEBUG:boto:Data: 2022-09-09T22:33:01.658 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.659 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:01.659 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.661 DEBUG:boto:Params: {} 2022-09-09T22:33:01.663 DEBUG:boto:Token: None 2022-09-09T22:33:01.663 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/myobj 2022-09-09T22:33:01.664 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Xqq8ZNTsj6ik8q/KHhBOcWW+D9w= 2022-09-09T22:33:01.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Xqq8ZNTsj6ik8q/KHhBOcWW+D9w=', 'Content-Length': '0'} 2022-09-09T22:33:01.668 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006aaec48fc51bf5a6-00631bbf1d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.669 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:01.670 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:01.671 DEBUG:boto:path=/jkcgbj-11/?prefix=myobj 2022-09-09T22:33:01.671 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=myobj 2022-09-09T22:33:01.672 DEBUG:boto:Method: GET 2022-09-09T22:33:01.673 DEBUG:boto:Path: /jkcgbj-11/?prefix=myobj 2022-09-09T22:33:01.673 DEBUG:boto:Data: 2022-09-09T22:33:01.674 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.675 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:01.675 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.676 DEBUG:boto:Params: {} 2022-09-09T22:33:01.677 DEBUG:boto:Token: None 2022-09-09T22:33:01.678 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/ 2022-09-09T22:33:01.678 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:uDoN5ZAJeR5MIIaBks4qtXifx9k= 2022-09-09T22:33:01.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:uDoN5ZAJeR5MIIaBks4qtXifx9k=', 'Content-Length': '0'} 2022-09-09T22:33:01.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ea7d218bd05e752-00631bbf1d-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.728 DEBUG:boto:b'jkcgbj-11myobj1000falsemyobj2022-09-09T22:30:44.302Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:01.729 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:01.730 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:01.730 DEBUG:boto:path=/jkcgbj-11/?prefix=myobj 2022-09-09T22:33:01.731 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=myobj 2022-09-09T22:33:01.732 DEBUG:boto:Method: GET 2022-09-09T22:33:01.732 DEBUG:boto:Path: /jkcgbj-11/?prefix=myobj 2022-09-09T22:33:01.733 DEBUG:boto:Data: 2022-09-09T22:33:01.733 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.734 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:01.734 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.735 DEBUG:boto:Params: {} 2022-09-09T22:33:01.735 DEBUG:boto:Token: None 2022-09-09T22:33:01.736 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/ 2022-09-09T22:33:01.737 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:uDoN5ZAJeR5MIIaBks4qtXifx9k= 2022-09-09T22:33:01.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:uDoN5ZAJeR5MIIaBks4qtXifx9k=', 'Content-Length': '0'} 2022-09-09T22:33:01.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000876e794f2e04d44b-00631bbf1d-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.744 DEBUG:boto:b'jkcgbj-11myobj1000falsemyobj2022-09-09T22:30:44.302Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:01.745 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-11 zones={test-zone1, test-zone2} 2022-09-09T22:33:01.746 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:33:01.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:33:01.747 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:33:01.800 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:33:01.801 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:33:01.870 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:33:01.868+0000 7f2b3396ddc0 1 bucket sync caught up with source: 2022-09-09T22:33:01.871 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000001.17.5, , 00000000001.12.5, , , , 00000000001.25.5, 00000000001.16.5, , ] 2022-09-09T22:33:01.871 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000001.17.5, , 00000000001.12.5, , , , 00000000001.25.5, 00000000001.16.5, , ] 2022-09-09T22:33:01.871 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:33:01.868+0000 7f2b3396ddc0 0 bucket checkpoint complete 2022-09-09T22:33:01.897 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-11 zones={test-zone3, test-zone2} 2022-09-09T22:33:01.897 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:01.898 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:01.899 DEBUG:boto:Method: HEAD 2022-09-09T22:33:01.899 DEBUG:boto:Path: /jkcgbj-11/ 2022-09-09T22:33:01.900 DEBUG:boto:Data: 2022-09-09T22:33:01.901 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.901 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:01.902 DEBUG:boto:Port: 8001 2022-09-09T22:33:01.902 DEBUG:boto:Params: {} 2022-09-09T22:33:01.903 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-09T22:33:01.903 DEBUG:boto:Token: None 2022-09-09T22:33:01.904 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/ 2022-09-09T22:33:01.905 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OrIyPP7pQrzyuPx1V9yBLSdOrbc= 2022-09-09T22:33:01.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OrIyPP7pQrzyuPx1V9yBLSdOrbc=', 'Content-Length': '0'} 2022-09-09T22:33:01.911 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002bd06559ce1ac89b-00631bbf1d-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.912 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:01.912 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:01.913 DEBUG:boto:Method: HEAD 2022-09-09T22:33:01.913 DEBUG:boto:Path: /jkcgbj-11/ 2022-09-09T22:33:01.914 DEBUG:boto:Data: 2022-09-09T22:33:01.914 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.915 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:01.915 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.916 DEBUG:boto:Params: {} 2022-09-09T22:33:01.916 DEBUG:boto:Token: None 2022-09-09T22:33:01.917 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/ 2022-09-09T22:33:01.917 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OrIyPP7pQrzyuPx1V9yBLSdOrbc= 2022-09-09T22:33:01.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OrIyPP7pQrzyuPx1V9yBLSdOrbc=', 'Content-Length': '0'} 2022-09-09T22:33:01.923 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ec192b28205fb5df-00631bbf1d-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.924 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:33:01.925 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:01.926 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:01.926 DEBUG:boto:path=/jkcgbj-11/?versions&max-keys=999 2022-09-09T22:33:01.927 DEBUG:boto:auth_path=/jkcgbj-11/?versions&max-keys=999 2022-09-09T22:33:01.927 DEBUG:boto:Method: GET 2022-09-09T22:33:01.928 DEBUG:boto:Path: /jkcgbj-11/?versions&max-keys=999 2022-09-09T22:33:01.928 DEBUG:boto:Data: 2022-09-09T22:33:01.929 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.929 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:01.930 DEBUG:boto:Port: 8001 2022-09-09T22:33:01.931 DEBUG:boto:Params: {} 2022-09-09T22:33:01.931 DEBUG:boto:Token: None 2022-09-09T22:33:01.932 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/?versions 2022-09-09T22:33:01.933 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:f/XEUsaa1kGUH28AkZmsP59qP8Q= 2022-09-09T22:33:01.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:f/XEUsaa1kGUH28AkZmsP59qP8Q=', 'Content-Length': '0'} 2022-09-09T22:33:01.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e8dc7ec824a82e9-00631bbf1d-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.943 DEBUG:boto:b'jkcgbj-11999false&nulltrue2022-09-09T22:30:44.517Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-09T22:30:44.453Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-09T22:30:44.357Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-09T22:30:44.302Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:01.944 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-09T22:33:01.945 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-09T22:33:01.946 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-09T22:33:01.946 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-09T22:33:01.947 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:33:01.947 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:01.948 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:01.949 DEBUG:boto:path=/jkcgbj-11/?versions&max-keys=999 2022-09-09T22:33:01.949 DEBUG:boto:auth_path=/jkcgbj-11/?versions&max-keys=999 2022-09-09T22:33:01.950 DEBUG:boto:Method: GET 2022-09-09T22:33:01.951 DEBUG:boto:Path: /jkcgbj-11/?versions&max-keys=999 2022-09-09T22:33:01.952 DEBUG:boto:Data: 2022-09-09T22:33:01.953 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.954 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:01.955 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.956 DEBUG:boto:Params: {} 2022-09-09T22:33:01.957 DEBUG:boto:Token: None 2022-09-09T22:33:01.959 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/?versions 2022-09-09T22:33:01.959 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:f/XEUsaa1kGUH28AkZmsP59qP8Q= 2022-09-09T22:33:01.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:f/XEUsaa1kGUH28AkZmsP59qP8Q=', 'Content-Length': '0'} 2022-09-09T22:33:01.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000726d39e9f8373941-00631bbf1d-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.968 DEBUG:boto:b'jkcgbj-11999false&nulltrue2022-09-09T22:30:44.517Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-09T22:30:44.453Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-09T22:30:44.357Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-09T22:30:44.302Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:01.969 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-09T22:33:01.969 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-09T22:33:01.970 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-09T22:33:01.970 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-09T22:33:01.971 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:01.971 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:01.972 DEBUG:boto:path=/jkcgbj-11/?versions&max-keys=999 2022-09-09T22:33:01.972 DEBUG:boto:auth_path=/jkcgbj-11/?versions&max-keys=999 2022-09-09T22:33:01.973 DEBUG:boto:Method: GET 2022-09-09T22:33:01.973 DEBUG:boto:Path: /jkcgbj-11/?versions&max-keys=999 2022-09-09T22:33:01.974 DEBUG:boto:Data: 2022-09-09T22:33:01.974 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.975 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:01.976 DEBUG:boto:Port: 8001 2022-09-09T22:33:01.977 DEBUG:boto:Params: {} 2022-09-09T22:33:01.977 DEBUG:boto:Token: None 2022-09-09T22:33:01.978 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/?versions 2022-09-09T22:33:01.978 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:f/XEUsaa1kGUH28AkZmsP59qP8Q= 2022-09-09T22:33:01.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:f/XEUsaa1kGUH28AkZmsP59qP8Q=', 'Content-Length': '0'} 2022-09-09T22:33:01.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4b6b1a379b96e70-00631bbf1d-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:01.986 DEBUG:boto:b'jkcgbj-11999false&nulltrue2022-09-09T22:30:44.517Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-09T22:30:44.453Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-09T22:30:44.357Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-09T22:30:44.302Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:01.987 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:01.988 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:01.988 DEBUG:boto:path=/jkcgbj-11/?versions&max-keys=999 2022-09-09T22:33:01.989 DEBUG:boto:auth_path=/jkcgbj-11/?versions&max-keys=999 2022-09-09T22:33:01.989 DEBUG:boto:Method: GET 2022-09-09T22:33:01.990 DEBUG:boto:Path: /jkcgbj-11/?versions&max-keys=999 2022-09-09T22:33:01.990 DEBUG:boto:Data: 2022-09-09T22:33:01.991 DEBUG:boto:Headers: {} 2022-09-09T22:33:01.991 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:01.992 DEBUG:boto:Port: 8000 2022-09-09T22:33:01.992 DEBUG:boto:Params: {} 2022-09-09T22:33:01.993 DEBUG:boto:Token: None 2022-09-09T22:33:01.993 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:01 GMT /jkcgbj-11/?versions 2022-09-09T22:33:01.994 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:f/XEUsaa1kGUH28AkZmsP59qP8Q= 2022-09-09T22:33:01.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:f/XEUsaa1kGUH28AkZmsP59qP8Q=', 'Content-Length': '0'} 2022-09-09T22:33:02.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ac88c61ddcf1773-00631bbf1d-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.001 DEBUG:boto:b'jkcgbj-11999false&nulltrue2022-09-09T22:30:44.517Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-09T22:30:44.453Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-09T22:30:44.357Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-09T22:30:44.302Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:02.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-09T22:33:02.002 DEBUG:boto:path=/jkcgbj-11/%26 2022-09-09T22:33:02.003 DEBUG:boto:auth_path=/jkcgbj-11/%26 2022-09-09T22:33:02.003 DEBUG:boto:path=/jkcgbj-11/%26?versionId=null 2022-09-09T22:33:02.004 DEBUG:boto:auth_path=/jkcgbj-11/%26?versionId=null 2022-09-09T22:33:02.004 DEBUG:boto:Method: GET 2022-09-09T22:33:02.005 DEBUG:boto:Path: /jkcgbj-11/%26?versionId=null 2022-09-09T22:33:02.005 DEBUG:boto:Data: 2022-09-09T22:33:02.006 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.006 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:02.007 DEBUG:boto:Port: 8001 2022-09-09T22:33:02.007 DEBUG:boto:Params: {} 2022-09-09T22:33:02.008 DEBUG:boto:Token: None 2022-09-09T22:33:02.009 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/%26?versionId=null 2022-09-09T22:33:02.009 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R3RZgYGBZPc397xt0mfqpa/pXnI= 2022-09-09T22:33:02.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R3RZgYGBZPc397xt0mfqpa/pXnI=', 'Content-Length': '0'} 2022-09-09T22:33:02.013 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dbb2c44cfc239719-00631bbf1e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.014 DEBUG:boto:path=/jkcgbj-11/%26 2022-09-09T22:33:02.015 DEBUG:boto:auth_path=/jkcgbj-11/%26 2022-09-09T22:33:02.015 DEBUG:boto:path=/jkcgbj-11/%26?versionId=null 2022-09-09T22:33:02.016 DEBUG:boto:auth_path=/jkcgbj-11/%26?versionId=null 2022-09-09T22:33:02.016 DEBUG:boto:Method: GET 2022-09-09T22:33:02.017 DEBUG:boto:Path: /jkcgbj-11/%26?versionId=null 2022-09-09T22:33:02.017 DEBUG:boto:Data: 2022-09-09T22:33:02.018 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.018 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:02.019 DEBUG:boto:Port: 8000 2022-09-09T22:33:02.019 DEBUG:boto:Params: {} 2022-09-09T22:33:02.020 DEBUG:boto:Token: None 2022-09-09T22:33:02.020 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/%26?versionId=null 2022-09-09T22:33:02.021 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R3RZgYGBZPc397xt0mfqpa/pXnI= 2022-09-09T22:33:02.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R3RZgYGBZPc397xt0mfqpa/pXnI=', 'Content-Length': '0'} 2022-09-09T22:33:02.025 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008315c136062acd9c-00631bbf1e-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.066 DEBUG:boto:path=/jkcgbj-11/%26 2022-09-09T22:33:02.067 DEBUG:boto:auth_path=/jkcgbj-11/%26 2022-09-09T22:33:02.068 DEBUG:boto:path=/jkcgbj-11/%26?versionId=null 2022-09-09T22:33:02.068 DEBUG:boto:auth_path=/jkcgbj-11/%26?versionId=null 2022-09-09T22:33:02.069 DEBUG:boto:Method: HEAD 2022-09-09T22:33:02.069 DEBUG:boto:Path: /jkcgbj-11/%26?versionId=null 2022-09-09T22:33:02.070 DEBUG:boto:Data: 2022-09-09T22:33:02.070 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.071 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:02.071 DEBUG:boto:Port: 8001 2022-09-09T22:33:02.072 DEBUG:boto:Params: {} 2022-09-09T22:33:02.073 DEBUG:boto:Token: None 2022-09-09T22:33:02.074 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/%26?versionId=null 2022-09-09T22:33:02.074 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:G2z+YYuinVOJvndCxOnVGNTvBXg= 2022-09-09T22:33:02.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:G2z+YYuinVOJvndCxOnVGNTvBXg=', 'Content-Length': '0'} 2022-09-09T22:33:02.078 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008eae63c6fb7a0a6b-00631bbf1e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.079 DEBUG:boto:path=/jkcgbj-11/%26 2022-09-09T22:33:02.079 DEBUG:boto:auth_path=/jkcgbj-11/%26 2022-09-09T22:33:02.080 DEBUG:boto:path=/jkcgbj-11/%26?versionId=null 2022-09-09T22:33:02.080 DEBUG:boto:auth_path=/jkcgbj-11/%26?versionId=null 2022-09-09T22:33:02.081 DEBUG:boto:Method: HEAD 2022-09-09T22:33:02.081 DEBUG:boto:Path: /jkcgbj-11/%26?versionId=null 2022-09-09T22:33:02.082 DEBUG:boto:Data: 2022-09-09T22:33:02.082 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.083 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:02.083 DEBUG:boto:Port: 8000 2022-09-09T22:33:02.084 DEBUG:boto:Params: {} 2022-09-09T22:33:02.084 DEBUG:boto:Token: None 2022-09-09T22:33:02.085 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/%26?versionId=null 2022-09-09T22:33:02.086 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:G2z+YYuinVOJvndCxOnVGNTvBXg= 2022-09-09T22:33:02.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:G2z+YYuinVOJvndCxOnVGNTvBXg=', 'Content-Length': '0'} 2022-09-09T22:33:02.089 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076fad3c5bee14b85-00631bbf1e-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-09T22:33:02.091 DEBUG:boto:path=/jkcgbj-11/%26 2022-09-09T22:33:02.091 DEBUG:boto:auth_path=/jkcgbj-11/%26 2022-09-09T22:33:02.092 DEBUG:boto:path=/jkcgbj-11/%26?versionId=null 2022-09-09T22:33:02.093 DEBUG:boto:auth_path=/jkcgbj-11/%26?versionId=null 2022-09-09T22:33:02.093 DEBUG:boto:Method: GET 2022-09-09T22:33:02.094 DEBUG:boto:Path: /jkcgbj-11/%26?versionId=null 2022-09-09T22:33:02.095 DEBUG:boto:Data: 2022-09-09T22:33:02.096 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.097 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:02.098 DEBUG:boto:Port: 8001 2022-09-09T22:33:02.099 DEBUG:boto:Params: {} 2022-09-09T22:33:02.100 DEBUG:boto:Token: None 2022-09-09T22:33:02.101 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/%26?versionId=null 2022-09-09T22:33:02.102 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R3RZgYGBZPc397xt0mfqpa/pXnI= 2022-09-09T22:33:02.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R3RZgYGBZPc397xt0mfqpa/pXnI=', 'Content-Length': '0'} 2022-09-09T22:33:02.107 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae08c01848d6d049-00631bbf1e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.154 DEBUG:boto:path=/jkcgbj-11/%26 2022-09-09T22:33:02.155 DEBUG:boto:auth_path=/jkcgbj-11/%26 2022-09-09T22:33:02.155 DEBUG:boto:path=/jkcgbj-11/%26?versionId=null 2022-09-09T22:33:02.156 DEBUG:boto:auth_path=/jkcgbj-11/%26?versionId=null 2022-09-09T22:33:02.157 DEBUG:boto:Method: GET 2022-09-09T22:33:02.158 DEBUG:boto:Path: /jkcgbj-11/%26?versionId=null 2022-09-09T22:33:02.158 DEBUG:boto:Data: 2022-09-09T22:33:02.159 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.159 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:02.160 DEBUG:boto:Port: 8000 2022-09-09T22:33:02.160 DEBUG:boto:Params: {} 2022-09-09T22:33:02.161 DEBUG:boto:Token: None 2022-09-09T22:33:02.162 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/%26?versionId=null 2022-09-09T22:33:02.163 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R3RZgYGBZPc397xt0mfqpa/pXnI= 2022-09-09T22:33:02.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R3RZgYGBZPc397xt0mfqpa/pXnI=', 'Content-Length': '0'} 2022-09-09T22:33:02.168 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ce18d8f04a4e0ac-00631bbf1e-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.168 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:02.169 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:02.170 DEBUG:boto:path=/jkcgbj-11/?prefix=%26 2022-09-09T22:33:02.170 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=%26 2022-09-09T22:33:02.171 DEBUG:boto:Method: GET 2022-09-09T22:33:02.171 DEBUG:boto:Path: /jkcgbj-11/?prefix=%26 2022-09-09T22:33:02.172 DEBUG:boto:Data: 2022-09-09T22:33:02.173 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.174 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:02.175 DEBUG:boto:Port: 8001 2022-09-09T22:33:02.175 DEBUG:boto:Params: {} 2022-09-09T22:33:02.176 DEBUG:boto:Token: None 2022-09-09T22:33:02.177 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/ 2022-09-09T22:33:02.178 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qNdIR7/XPPHXh+21qpmZNgQakj8= 2022-09-09T22:33:02.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qNdIR7/XPPHXh+21qpmZNgQakj8=', 'Content-Length': '0'} 2022-09-09T22:33:02.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008580a3716ad4a7ca-00631bbf1e-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.185 DEBUG:boto:b'jkcgbj-11&1000false&2022-09-09T22:30:44.517Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:02.186 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:02.187 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:02.187 DEBUG:boto:path=/jkcgbj-11/?prefix=%26 2022-09-09T22:33:02.188 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=%26 2022-09-09T22:33:02.189 DEBUG:boto:Method: GET 2022-09-09T22:33:02.189 DEBUG:boto:Path: /jkcgbj-11/?prefix=%26 2022-09-09T22:33:02.190 DEBUG:boto:Data: 2022-09-09T22:33:02.190 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.191 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:02.191 DEBUG:boto:Port: 8000 2022-09-09T22:33:02.192 DEBUG:boto:Params: {} 2022-09-09T22:33:02.193 DEBUG:boto:Token: None 2022-09-09T22:33:02.194 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/ 2022-09-09T22:33:02.194 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qNdIR7/XPPHXh+21qpmZNgQakj8= 2022-09-09T22:33:02.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qNdIR7/XPPHXh+21qpmZNgQakj8=', 'Content-Length': '0'} 2022-09-09T22:33:02.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009b6607422138924-00631bbf1e-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.202 DEBUG:boto:b'jkcgbj-11&1000false&2022-09-09T22:30:44.517Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:02.203 DEBUG:boto:path=/jkcgbj-11/%26 2022-09-09T22:33:02.203 DEBUG:boto:auth_path=/jkcgbj-11/%26 2022-09-09T22:33:02.204 DEBUG:boto:Method: HEAD 2022-09-09T22:33:02.205 DEBUG:boto:Path: /jkcgbj-11/%26 2022-09-09T22:33:02.205 DEBUG:boto:Data: 2022-09-09T22:33:02.206 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.206 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:02.207 DEBUG:boto:Port: 8001 2022-09-09T22:33:02.208 DEBUG:boto:Params: {} 2022-09-09T22:33:02.208 DEBUG:boto:Token: None 2022-09-09T22:33:02.209 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/%26 2022-09-09T22:33:02.209 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8TmOcQjuatGhbkRGYjdY+i1460Y= 2022-09-09T22:33:02.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8TmOcQjuatGhbkRGYjdY+i1460Y=', 'Content-Length': '0'} 2022-09-09T22:33:02.213 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000028ab248ed52e29b3-00631bbf1e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.214 DEBUG:boto:path=/jkcgbj-11/%26 2022-09-09T22:33:02.215 DEBUG:boto:auth_path=/jkcgbj-11/%26 2022-09-09T22:33:02.216 DEBUG:boto:Method: HEAD 2022-09-09T22:33:02.216 DEBUG:boto:Path: /jkcgbj-11/%26 2022-09-09T22:33:02.217 DEBUG:boto:Data: 2022-09-09T22:33:02.217 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.218 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:02.218 DEBUG:boto:Port: 8000 2022-09-09T22:33:02.219 DEBUG:boto:Params: {} 2022-09-09T22:33:02.220 DEBUG:boto:Token: None 2022-09-09T22:33:02.220 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/%26 2022-09-09T22:33:02.221 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8TmOcQjuatGhbkRGYjdY+i1460Y= 2022-09-09T22:33:02.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8TmOcQjuatGhbkRGYjdY+i1460Y=', 'Content-Length': '0'} 2022-09-09T22:33:02.225 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ae389bf1544e837-00631bbf1e-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-09T22:33:02.226 DEBUG:boto:path=/jkcgbj-11/%26 2022-09-09T22:33:02.227 DEBUG:boto:auth_path=/jkcgbj-11/%26 2022-09-09T22:33:02.227 DEBUG:boto:Method: GET 2022-09-09T22:33:02.228 DEBUG:boto:Path: /jkcgbj-11/%26 2022-09-09T22:33:02.228 DEBUG:boto:Data: 2022-09-09T22:33:02.229 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.230 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:02.230 DEBUG:boto:Port: 8001 2022-09-09T22:33:02.231 DEBUG:boto:Params: {} 2022-09-09T22:33:02.231 DEBUG:boto:Token: None 2022-09-09T22:33:02.232 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/%26 2022-09-09T22:33:02.233 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:lilAgvavMda21+1T4bntAxLUWd0= 2022-09-09T22:33:02.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:lilAgvavMda21+1T4bntAxLUWd0=', 'Content-Length': '0'} 2022-09-09T22:33:02.237 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007251e4b073b42a06-00631bbf1e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.238 DEBUG:boto:path=/jkcgbj-11/%26 2022-09-09T22:33:02.238 DEBUG:boto:auth_path=/jkcgbj-11/%26 2022-09-09T22:33:02.239 DEBUG:boto:Method: GET 2022-09-09T22:33:02.239 DEBUG:boto:Path: /jkcgbj-11/%26 2022-09-09T22:33:02.240 DEBUG:boto:Data: 2022-09-09T22:33:02.240 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.241 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:02.242 DEBUG:boto:Port: 8000 2022-09-09T22:33:02.242 DEBUG:boto:Params: {} 2022-09-09T22:33:02.243 DEBUG:boto:Token: None 2022-09-09T22:33:02.244 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/%26 2022-09-09T22:33:02.244 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:lilAgvavMda21+1T4bntAxLUWd0= 2022-09-09T22:33:02.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:lilAgvavMda21+1T4bntAxLUWd0=', 'Content-Length': '0'} 2022-09-09T22:33:02.248 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000035a3d2ec564ab82e-00631bbf1e-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.291 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:02.292 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:02.293 DEBUG:boto:path=/jkcgbj-11/?prefix=%26 2022-09-09T22:33:02.294 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=%26 2022-09-09T22:33:02.295 DEBUG:boto:Method: GET 2022-09-09T22:33:02.296 DEBUG:boto:Path: /jkcgbj-11/?prefix=%26 2022-09-09T22:33:02.297 DEBUG:boto:Data: 2022-09-09T22:33:02.298 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.299 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:02.300 DEBUG:boto:Port: 8001 2022-09-09T22:33:02.302 DEBUG:boto:Params: {} 2022-09-09T22:33:02.303 DEBUG:boto:Token: None 2022-09-09T22:33:02.304 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/ 2022-09-09T22:33:02.305 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qNdIR7/XPPHXh+21qpmZNgQakj8= 2022-09-09T22:33:02.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qNdIR7/XPPHXh+21qpmZNgQakj8=', 'Content-Length': '0'} 2022-09-09T22:33:02.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b6117eb7950c715-00631bbf1e-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.313 DEBUG:boto:b'jkcgbj-11&1000false&2022-09-09T22:30:44.517Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:02.315 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:02.316 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:02.317 DEBUG:boto:path=/jkcgbj-11/?prefix=%26 2022-09-09T22:33:02.317 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=%26 2022-09-09T22:33:02.318 DEBUG:boto:Method: GET 2022-09-09T22:33:02.319 DEBUG:boto:Path: /jkcgbj-11/?prefix=%26 2022-09-09T22:33:02.320 DEBUG:boto:Data: 2022-09-09T22:33:02.320 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.321 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:02.321 DEBUG:boto:Port: 8000 2022-09-09T22:33:02.322 DEBUG:boto:Params: {} 2022-09-09T22:33:02.323 DEBUG:boto:Token: None 2022-09-09T22:33:02.324 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/ 2022-09-09T22:33:02.325 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qNdIR7/XPPHXh+21qpmZNgQakj8= 2022-09-09T22:33:02.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qNdIR7/XPPHXh+21qpmZNgQakj8=', 'Content-Length': '0'} 2022-09-09T22:33:02.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3a80b1dec465624-00631bbf1e-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.334 DEBUG:boto:b'jkcgbj-11&1000false&2022-09-09T22:30:44.517Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:02.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-09T22:33:02.336 DEBUG:boto:path=/jkcgbj-11/%3A 2022-09-09T22:33:02.337 DEBUG:boto:auth_path=/jkcgbj-11/%3A 2022-09-09T22:33:02.338 DEBUG:boto:path=/jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:02.339 DEBUG:boto:auth_path=/jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:02.340 DEBUG:boto:Method: GET 2022-09-09T22:33:02.342 DEBUG:boto:Path: /jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:02.344 DEBUG:boto:Data: 2022-09-09T22:33:02.344 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.345 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:02.346 DEBUG:boto:Port: 8001 2022-09-09T22:33:02.346 DEBUG:boto:Params: {} 2022-09-09T22:33:02.347 DEBUG:boto:Token: None 2022-09-09T22:33:02.347 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:02.348 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:T/2r70mxPbneb4PZI16vzrIgZpE= 2022-09-09T22:33:02.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:T/2r70mxPbneb4PZI16vzrIgZpE=', 'Content-Length': '0'} 2022-09-09T22:33:02.353 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001fd17e9d66599450-00631bbf1e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.354 DEBUG:boto:path=/jkcgbj-11/%3A 2022-09-09T22:33:02.355 DEBUG:boto:auth_path=/jkcgbj-11/%3A 2022-09-09T22:33:02.356 DEBUG:boto:path=/jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:02.357 DEBUG:boto:auth_path=/jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:02.358 DEBUG:boto:Method: GET 2022-09-09T22:33:02.359 DEBUG:boto:Path: /jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:02.360 DEBUG:boto:Data: 2022-09-09T22:33:02.361 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.362 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:02.363 DEBUG:boto:Port: 8000 2022-09-09T22:33:02.364 DEBUG:boto:Params: {} 2022-09-09T22:33:02.365 DEBUG:boto:Token: None 2022-09-09T22:33:02.366 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:02.367 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:T/2r70mxPbneb4PZI16vzrIgZpE= 2022-09-09T22:33:02.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:T/2r70mxPbneb4PZI16vzrIgZpE=', 'Content-Length': '0'} 2022-09-09T22:33:02.371 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077f8e8fc8d2e51ca-00631bbf1e-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.372 DEBUG:boto:path=/jkcgbj-11/%3A 2022-09-09T22:33:02.373 DEBUG:boto:auth_path=/jkcgbj-11/%3A 2022-09-09T22:33:02.373 DEBUG:boto:path=/jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:02.374 DEBUG:boto:auth_path=/jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:02.375 DEBUG:boto:Method: HEAD 2022-09-09T22:33:02.375 DEBUG:boto:Path: /jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:02.376 DEBUG:boto:Data: 2022-09-09T22:33:02.377 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.377 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:02.378 DEBUG:boto:Port: 8001 2022-09-09T22:33:02.379 DEBUG:boto:Params: {} 2022-09-09T22:33:02.380 DEBUG:boto:Token: None 2022-09-09T22:33:02.380 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:02.381 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:aGOkPCCBBapFDTr5KdqPPPJVTxc= 2022-09-09T22:33:02.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:aGOkPCCBBapFDTr5KdqPPPJVTxc=', 'Content-Length': '0'} 2022-09-09T22:33:02.388 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a111f34abed3bd6-00631bbf1e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.389 DEBUG:boto:path=/jkcgbj-11/%3A 2022-09-09T22:33:02.389 DEBUG:boto:auth_path=/jkcgbj-11/%3A 2022-09-09T22:33:02.390 DEBUG:boto:path=/jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:02.390 DEBUG:boto:auth_path=/jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:02.391 DEBUG:boto:Method: HEAD 2022-09-09T22:33:02.391 DEBUG:boto:Path: /jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:02.392 DEBUG:boto:Data: 2022-09-09T22:33:02.392 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.393 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:02.393 DEBUG:boto:Port: 8000 2022-09-09T22:33:02.394 DEBUG:boto:Params: {} 2022-09-09T22:33:02.394 DEBUG:boto:Token: None 2022-09-09T22:33:02.395 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:02.396 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:aGOkPCCBBapFDTr5KdqPPPJVTxc= 2022-09-09T22:33:02.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:aGOkPCCBBapFDTr5KdqPPPJVTxc=', 'Content-Length': '0'} 2022-09-09T22:33:02.399 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062362c7c9c6e3e78-00631bbf1e-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-09T22:33:02.400 DEBUG:boto:path=/jkcgbj-11/%3A 2022-09-09T22:33:02.401 DEBUG:boto:auth_path=/jkcgbj-11/%3A 2022-09-09T22:33:02.402 DEBUG:boto:path=/jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:02.402 DEBUG:boto:auth_path=/jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:02.403 DEBUG:boto:Method: GET 2022-09-09T22:33:02.403 DEBUG:boto:Path: /jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:02.404 DEBUG:boto:Data: 2022-09-09T22:33:02.404 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.405 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:02.405 DEBUG:boto:Port: 8001 2022-09-09T22:33:02.406 DEBUG:boto:Params: {} 2022-09-09T22:33:02.407 DEBUG:boto:Token: None 2022-09-09T22:33:02.407 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:02.408 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:T/2r70mxPbneb4PZI16vzrIgZpE= 2022-09-09T22:33:02.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:T/2r70mxPbneb4PZI16vzrIgZpE=', 'Content-Length': '0'} 2022-09-09T22:33:02.417 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000083c0d37841f5db5f-00631bbf1e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.458 DEBUG:boto:path=/jkcgbj-11/%3A 2022-09-09T22:33:02.459 DEBUG:boto:auth_path=/jkcgbj-11/%3A 2022-09-09T22:33:02.460 DEBUG:boto:path=/jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:02.461 DEBUG:boto:auth_path=/jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:02.462 DEBUG:boto:Method: GET 2022-09-09T22:33:02.463 DEBUG:boto:Path: /jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:02.464 DEBUG:boto:Data: 2022-09-09T22:33:02.465 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.465 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:02.466 DEBUG:boto:Port: 8000 2022-09-09T22:33:02.466 DEBUG:boto:Params: {} 2022-09-09T22:33:02.467 DEBUG:boto:Token: None 2022-09-09T22:33:02.469 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/%3A?versionId=null 2022-09-09T22:33:02.470 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:T/2r70mxPbneb4PZI16vzrIgZpE= 2022-09-09T22:33:02.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:T/2r70mxPbneb4PZI16vzrIgZpE=', 'Content-Length': '0'} 2022-09-09T22:33:02.475 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c6f9eeab6984ae0-00631bbf1e-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.477 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:02.478 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:02.480 DEBUG:boto:path=/jkcgbj-11/?prefix=%3A 2022-09-09T22:33:02.481 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=%3A 2022-09-09T22:33:02.482 DEBUG:boto:Method: GET 2022-09-09T22:33:02.482 DEBUG:boto:Path: /jkcgbj-11/?prefix=%3A 2022-09-09T22:33:02.484 DEBUG:boto:Data: 2022-09-09T22:33:02.484 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.485 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:02.486 DEBUG:boto:Port: 8001 2022-09-09T22:33:02.486 DEBUG:boto:Params: {} 2022-09-09T22:33:02.487 DEBUG:boto:Token: None 2022-09-09T22:33:02.488 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/ 2022-09-09T22:33:02.488 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qNdIR7/XPPHXh+21qpmZNgQakj8= 2022-09-09T22:33:02.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qNdIR7/XPPHXh+21qpmZNgQakj8=', 'Content-Length': '0'} 2022-09-09T22:33:02.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000465546c377ddcdc0-00631bbf1e-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.496 DEBUG:boto:b'jkcgbj-11:1000false:2022-09-09T22:30:44.453Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:02.497 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:02.497 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:02.498 DEBUG:boto:path=/jkcgbj-11/?prefix=%3A 2022-09-09T22:33:02.498 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=%3A 2022-09-09T22:33:02.499 DEBUG:boto:Method: GET 2022-09-09T22:33:02.499 DEBUG:boto:Path: /jkcgbj-11/?prefix=%3A 2022-09-09T22:33:02.500 DEBUG:boto:Data: 2022-09-09T22:33:02.500 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.501 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:02.502 DEBUG:boto:Port: 8000 2022-09-09T22:33:02.502 DEBUG:boto:Params: {} 2022-09-09T22:33:02.503 DEBUG:boto:Token: None 2022-09-09T22:33:02.503 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/ 2022-09-09T22:33:02.504 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qNdIR7/XPPHXh+21qpmZNgQakj8= 2022-09-09T22:33:02.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qNdIR7/XPPHXh+21qpmZNgQakj8=', 'Content-Length': '0'} 2022-09-09T22:33:02.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032acaf45839bf1dd-00631bbf1e-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.512 DEBUG:boto:b'jkcgbj-11:1000false:2022-09-09T22:30:44.453Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:02.512 DEBUG:boto:path=/jkcgbj-11/%3A 2022-09-09T22:33:02.513 DEBUG:boto:auth_path=/jkcgbj-11/%3A 2022-09-09T22:33:02.513 DEBUG:boto:Method: HEAD 2022-09-09T22:33:02.514 DEBUG:boto:Path: /jkcgbj-11/%3A 2022-09-09T22:33:02.515 DEBUG:boto:Data: 2022-09-09T22:33:02.515 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.516 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:02.516 DEBUG:boto:Port: 8001 2022-09-09T22:33:02.516 DEBUG:boto:Params: {} 2022-09-09T22:33:02.517 DEBUG:boto:Token: None 2022-09-09T22:33:02.518 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/%3A 2022-09-09T22:33:02.518 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Ivt54qiKtxH2HFExMusT/Mc6mdY= 2022-09-09T22:33:02.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Ivt54qiKtxH2HFExMusT/Mc6mdY=', 'Content-Length': '0'} 2022-09-09T22:33:02.522 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4060e4c0313bb67-00631bbf1e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.524 DEBUG:boto:path=/jkcgbj-11/%3A 2022-09-09T22:33:02.525 DEBUG:boto:auth_path=/jkcgbj-11/%3A 2022-09-09T22:33:02.525 DEBUG:boto:Method: HEAD 2022-09-09T22:33:02.527 DEBUG:boto:Path: /jkcgbj-11/%3A 2022-09-09T22:33:02.528 DEBUG:boto:Data: 2022-09-09T22:33:02.529 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.530 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:02.531 DEBUG:boto:Port: 8000 2022-09-09T22:33:02.532 DEBUG:boto:Params: {} 2022-09-09T22:33:02.533 DEBUG:boto:Token: None 2022-09-09T22:33:02.534 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/%3A 2022-09-09T22:33:02.535 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Ivt54qiKtxH2HFExMusT/Mc6mdY= 2022-09-09T22:33:02.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Ivt54qiKtxH2HFExMusT/Mc6mdY=', 'Content-Length': '0'} 2022-09-09T22:33:02.539 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000356ee32c304e2a78-00631bbf1e-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-09T22:33:02.540 DEBUG:boto:path=/jkcgbj-11/%3A 2022-09-09T22:33:02.541 DEBUG:boto:auth_path=/jkcgbj-11/%3A 2022-09-09T22:33:02.541 DEBUG:boto:Method: GET 2022-09-09T22:33:02.542 DEBUG:boto:Path: /jkcgbj-11/%3A 2022-09-09T22:33:02.542 DEBUG:boto:Data: 2022-09-09T22:33:02.543 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.543 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:02.544 DEBUG:boto:Port: 8001 2022-09-09T22:33:02.545 DEBUG:boto:Params: {} 2022-09-09T22:33:02.545 DEBUG:boto:Token: None 2022-09-09T22:33:02.546 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/%3A 2022-09-09T22:33:02.546 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2YccUtQg0lyFmferp5ENYfNViMc= 2022-09-09T22:33:02.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2YccUtQg0lyFmferp5ENYfNViMc=', 'Content-Length': '0'} 2022-09-09T22:33:02.550 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c292005af48bf75-00631bbf1e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.551 DEBUG:boto:path=/jkcgbj-11/%3A 2022-09-09T22:33:02.551 DEBUG:boto:auth_path=/jkcgbj-11/%3A 2022-09-09T22:33:02.552 DEBUG:boto:Method: GET 2022-09-09T22:33:02.553 DEBUG:boto:Path: /jkcgbj-11/%3A 2022-09-09T22:33:02.553 DEBUG:boto:Data: 2022-09-09T22:33:02.554 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.555 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:02.555 DEBUG:boto:Port: 8000 2022-09-09T22:33:02.556 DEBUG:boto:Params: {} 2022-09-09T22:33:02.557 DEBUG:boto:Token: None 2022-09-09T22:33:02.558 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/%3A 2022-09-09T22:33:02.559 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2YccUtQg0lyFmferp5ENYfNViMc= 2022-09-09T22:33:02.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2YccUtQg0lyFmferp5ENYfNViMc=', 'Content-Length': '0'} 2022-09-09T22:33:02.563 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000879b66e2b893fb9c-00631bbf1e-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.606 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:02.608 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:02.609 DEBUG:boto:path=/jkcgbj-11/?prefix=%3A 2022-09-09T22:33:02.610 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=%3A 2022-09-09T22:33:02.612 DEBUG:boto:Method: GET 2022-09-09T22:33:02.613 DEBUG:boto:Path: /jkcgbj-11/?prefix=%3A 2022-09-09T22:33:02.614 DEBUG:boto:Data: 2022-09-09T22:33:02.615 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.616 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:02.617 DEBUG:boto:Port: 8001 2022-09-09T22:33:02.618 DEBUG:boto:Params: {} 2022-09-09T22:33:02.619 DEBUG:boto:Token: None 2022-09-09T22:33:02.620 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/ 2022-09-09T22:33:02.621 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qNdIR7/XPPHXh+21qpmZNgQakj8= 2022-09-09T22:33:02.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qNdIR7/XPPHXh+21qpmZNgQakj8=', 'Content-Length': '0'} 2022-09-09T22:33:02.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b5356fbab9a86ef-00631bbf1e-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.632 DEBUG:boto:b'jkcgbj-11:1000false:2022-09-09T22:30:44.453Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:02.633 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:02.634 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:02.635 DEBUG:boto:path=/jkcgbj-11/?prefix=%3A 2022-09-09T22:33:02.636 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=%3A 2022-09-09T22:33:02.637 DEBUG:boto:Method: GET 2022-09-09T22:33:02.638 DEBUG:boto:Path: /jkcgbj-11/?prefix=%3A 2022-09-09T22:33:02.639 DEBUG:boto:Data: 2022-09-09T22:33:02.640 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.641 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:02.642 DEBUG:boto:Port: 8000 2022-09-09T22:33:02.643 DEBUG:boto:Params: {} 2022-09-09T22:33:02.644 DEBUG:boto:Token: None 2022-09-09T22:33:02.645 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/ 2022-09-09T22:33:02.646 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qNdIR7/XPPHXh+21qpmZNgQakj8= 2022-09-09T22:33:02.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qNdIR7/XPPHXh+21qpmZNgQakj8=', 'Content-Length': '0'} 2022-09-09T22:33:02.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000180111c0fc653f0f-00631bbf1e-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.655 DEBUG:boto:b'jkcgbj-11:1000false:2022-09-09T22:30:44.453Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:02.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-09T22:33:02.658 DEBUG:boto:path=/jkcgbj-11/_myobj 2022-09-09T22:33:02.659 DEBUG:boto:auth_path=/jkcgbj-11/_myobj 2022-09-09T22:33:02.660 DEBUG:boto:path=/jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:02.661 DEBUG:boto:auth_path=/jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:02.662 DEBUG:boto:Method: GET 2022-09-09T22:33:02.663 DEBUG:boto:Path: /jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:02.664 DEBUG:boto:Data: 2022-09-09T22:33:02.665 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.665 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:02.666 DEBUG:boto:Port: 8001 2022-09-09T22:33:02.667 DEBUG:boto:Params: {} 2022-09-09T22:33:02.668 DEBUG:boto:Token: None 2022-09-09T22:33:02.669 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:02.670 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LRJnzqb5PnhNtSB64Pb9AYja2qQ= 2022-09-09T22:33:02.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LRJnzqb5PnhNtSB64Pb9AYja2qQ=', 'Content-Length': '0'} 2022-09-09T22:33:02.675 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000950a055435a22c58-00631bbf1e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.719 DEBUG:boto:path=/jkcgbj-11/_myobj 2022-09-09T22:33:02.720 DEBUG:boto:auth_path=/jkcgbj-11/_myobj 2022-09-09T22:33:02.720 DEBUG:boto:path=/jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:02.721 DEBUG:boto:auth_path=/jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:02.722 DEBUG:boto:Method: GET 2022-09-09T22:33:02.723 DEBUG:boto:Path: /jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:02.724 DEBUG:boto:Data: 2022-09-09T22:33:02.725 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.726 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:02.727 DEBUG:boto:Port: 8000 2022-09-09T22:33:02.727 DEBUG:boto:Params: {} 2022-09-09T22:33:02.729 DEBUG:boto:Token: None 2022-09-09T22:33:02.730 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:02.731 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LRJnzqb5PnhNtSB64Pb9AYja2qQ= 2022-09-09T22:33:02.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LRJnzqb5PnhNtSB64Pb9AYja2qQ=', 'Content-Length': '0'} 2022-09-09T22:33:02.735 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a34b6b26cba7337-00631bbf1e-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.736 DEBUG:boto:path=/jkcgbj-11/_myobj 2022-09-09T22:33:02.737 DEBUG:boto:auth_path=/jkcgbj-11/_myobj 2022-09-09T22:33:02.737 DEBUG:boto:path=/jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:02.739 DEBUG:boto:auth_path=/jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:02.741 DEBUG:boto:Method: HEAD 2022-09-09T22:33:02.743 DEBUG:boto:Path: /jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:02.743 DEBUG:boto:Data: 2022-09-09T22:33:02.744 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.745 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:02.746 DEBUG:boto:Port: 8001 2022-09-09T22:33:02.747 DEBUG:boto:Params: {} 2022-09-09T22:33:02.748 DEBUG:boto:Token: None 2022-09-09T22:33:02.749 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:02.750 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9+bu/nxu1/JDBCgKkzSPRCStvQ0= 2022-09-09T22:33:02.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9+bu/nxu1/JDBCgKkzSPRCStvQ0=', 'Content-Length': '0'} 2022-09-09T22:33:02.756 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d2fd6c1e2b980394-00631bbf1e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.757 DEBUG:boto:path=/jkcgbj-11/_myobj 2022-09-09T22:33:02.758 DEBUG:boto:auth_path=/jkcgbj-11/_myobj 2022-09-09T22:33:02.759 DEBUG:boto:path=/jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:02.760 DEBUG:boto:auth_path=/jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:02.761 DEBUG:boto:Method: HEAD 2022-09-09T22:33:02.762 DEBUG:boto:Path: /jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:02.763 DEBUG:boto:Data: 2022-09-09T22:33:02.764 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.765 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:02.766 DEBUG:boto:Port: 8000 2022-09-09T22:33:02.767 DEBUG:boto:Params: {} 2022-09-09T22:33:02.768 DEBUG:boto:Token: None 2022-09-09T22:33:02.769 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:02.771 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9+bu/nxu1/JDBCgKkzSPRCStvQ0= 2022-09-09T22:33:02.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9+bu/nxu1/JDBCgKkzSPRCStvQ0=', 'Content-Length': '0'} 2022-09-09T22:33:02.776 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a3dc944fbdaa42bf-00631bbf1e-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-09T22:33:02.778 DEBUG:boto:path=/jkcgbj-11/_myobj 2022-09-09T22:33:02.779 DEBUG:boto:auth_path=/jkcgbj-11/_myobj 2022-09-09T22:33:02.780 DEBUG:boto:path=/jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:02.781 DEBUG:boto:auth_path=/jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:02.782 DEBUG:boto:Method: GET 2022-09-09T22:33:02.783 DEBUG:boto:Path: /jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:02.784 DEBUG:boto:Data: 2022-09-09T22:33:02.784 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.785 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:02.786 DEBUG:boto:Port: 8001 2022-09-09T22:33:02.786 DEBUG:boto:Params: {} 2022-09-09T22:33:02.787 DEBUG:boto:Token: None 2022-09-09T22:33:02.788 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:02.789 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LRJnzqb5PnhNtSB64Pb9AYja2qQ= 2022-09-09T22:33:02.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LRJnzqb5PnhNtSB64Pb9AYja2qQ=', 'Content-Length': '0'} 2022-09-09T22:33:02.793 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d4c714a1611e05ed-00631bbf1e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.794 DEBUG:boto:path=/jkcgbj-11/_myobj 2022-09-09T22:33:02.794 DEBUG:boto:auth_path=/jkcgbj-11/_myobj 2022-09-09T22:33:02.795 DEBUG:boto:path=/jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:02.796 DEBUG:boto:auth_path=/jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:02.796 DEBUG:boto:Method: GET 2022-09-09T22:33:02.797 DEBUG:boto:Path: /jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:02.797 DEBUG:boto:Data: 2022-09-09T22:33:02.798 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.798 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:02.799 DEBUG:boto:Port: 8000 2022-09-09T22:33:02.800 DEBUG:boto:Params: {} 2022-09-09T22:33:02.800 DEBUG:boto:Token: None 2022-09-09T22:33:02.801 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/_myobj?versionId=null 2022-09-09T22:33:02.802 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LRJnzqb5PnhNtSB64Pb9AYja2qQ= 2022-09-09T22:33:02.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LRJnzqb5PnhNtSB64Pb9AYja2qQ=', 'Content-Length': '0'} 2022-09-09T22:33:02.806 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d739a2abb4ea76f9-00631bbf1e-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.807 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:02.807 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:02.808 DEBUG:boto:path=/jkcgbj-11/?prefix=_myobj 2022-09-09T22:33:02.808 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=_myobj 2022-09-09T22:33:02.809 DEBUG:boto:Method: GET 2022-09-09T22:33:02.809 DEBUG:boto:Path: /jkcgbj-11/?prefix=_myobj 2022-09-09T22:33:02.810 DEBUG:boto:Data: 2022-09-09T22:33:02.810 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.811 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:02.812 DEBUG:boto:Port: 8001 2022-09-09T22:33:02.812 DEBUG:boto:Params: {} 2022-09-09T22:33:02.813 DEBUG:boto:Token: None 2022-09-09T22:33:02.814 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/ 2022-09-09T22:33:02.814 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qNdIR7/XPPHXh+21qpmZNgQakj8= 2022-09-09T22:33:02.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qNdIR7/XPPHXh+21qpmZNgQakj8=', 'Content-Length': '0'} 2022-09-09T22:33:02.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006177eaa489be309-00631bbf1e-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.824 DEBUG:boto:b'jkcgbj-11_myobj1000false_myobj2022-09-09T22:30:44.357Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:02.825 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:02.825 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:02.826 DEBUG:boto:path=/jkcgbj-11/?prefix=_myobj 2022-09-09T22:33:02.827 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=_myobj 2022-09-09T22:33:02.827 DEBUG:boto:Method: GET 2022-09-09T22:33:02.828 DEBUG:boto:Path: /jkcgbj-11/?prefix=_myobj 2022-09-09T22:33:02.829 DEBUG:boto:Data: 2022-09-09T22:33:02.829 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.830 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:02.830 DEBUG:boto:Port: 8000 2022-09-09T22:33:02.831 DEBUG:boto:Params: {} 2022-09-09T22:33:02.831 DEBUG:boto:Token: None 2022-09-09T22:33:02.832 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/ 2022-09-09T22:33:02.833 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qNdIR7/XPPHXh+21qpmZNgQakj8= 2022-09-09T22:33:02.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qNdIR7/XPPHXh+21qpmZNgQakj8=', 'Content-Length': '0'} 2022-09-09T22:33:02.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d227b81312b67e1-00631bbf1e-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.840 DEBUG:boto:b'jkcgbj-11_myobj1000false_myobj2022-09-09T22:30:44.357Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:02.841 DEBUG:boto:path=/jkcgbj-11/_myobj 2022-09-09T22:33:02.842 DEBUG:boto:auth_path=/jkcgbj-11/_myobj 2022-09-09T22:33:02.842 DEBUG:boto:Method: HEAD 2022-09-09T22:33:02.843 DEBUG:boto:Path: /jkcgbj-11/_myobj 2022-09-09T22:33:02.843 DEBUG:boto:Data: 2022-09-09T22:33:02.844 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.845 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:02.846 DEBUG:boto:Port: 8001 2022-09-09T22:33:02.847 DEBUG:boto:Params: {} 2022-09-09T22:33:02.848 DEBUG:boto:Token: None 2022-09-09T22:33:02.849 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/_myobj 2022-09-09T22:33:02.850 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:d+o3dhqpc5JCY1mOo6n1FO8o5eU= 2022-09-09T22:33:02.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:d+o3dhqpc5JCY1mOo6n1FO8o5eU=', 'Content-Length': '0'} 2022-09-09T22:33:02.855 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000228845e5b7294b8e-00631bbf1e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.857 DEBUG:boto:path=/jkcgbj-11/_myobj 2022-09-09T22:33:02.858 DEBUG:boto:auth_path=/jkcgbj-11/_myobj 2022-09-09T22:33:02.859 DEBUG:boto:Method: HEAD 2022-09-09T22:33:02.860 DEBUG:boto:Path: /jkcgbj-11/_myobj 2022-09-09T22:33:02.861 DEBUG:boto:Data: 2022-09-09T22:33:02.862 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.863 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:02.864 DEBUG:boto:Port: 8000 2022-09-09T22:33:02.865 DEBUG:boto:Params: {} 2022-09-09T22:33:02.866 DEBUG:boto:Token: None 2022-09-09T22:33:02.867 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/_myobj 2022-09-09T22:33:02.868 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:d+o3dhqpc5JCY1mOo6n1FO8o5eU= 2022-09-09T22:33:02.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:d+o3dhqpc5JCY1mOo6n1FO8o5eU=', 'Content-Length': '0'} 2022-09-09T22:33:02.873 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001709517aed63d7d5-00631bbf1e-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-09T22:33:02.876 DEBUG:boto:path=/jkcgbj-11/_myobj 2022-09-09T22:33:02.877 DEBUG:boto:auth_path=/jkcgbj-11/_myobj 2022-09-09T22:33:02.878 DEBUG:boto:Method: GET 2022-09-09T22:33:02.879 DEBUG:boto:Path: /jkcgbj-11/_myobj 2022-09-09T22:33:02.880 DEBUG:boto:Data: 2022-09-09T22:33:02.881 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.883 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:02.884 DEBUG:boto:Port: 8001 2022-09-09T22:33:02.885 DEBUG:boto:Params: {} 2022-09-09T22:33:02.886 DEBUG:boto:Token: None 2022-09-09T22:33:02.888 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/_myobj 2022-09-09T22:33:02.889 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:iSPrbiKkPuDMMo/6U9Ims61A914= 2022-09-09T22:33:02.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:iSPrbiKkPuDMMo/6U9Ims61A914=', 'Content-Length': '0'} 2022-09-09T22:33:02.895 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006894797435132b0b-00631bbf1e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.940 DEBUG:boto:path=/jkcgbj-11/_myobj 2022-09-09T22:33:02.941 DEBUG:boto:auth_path=/jkcgbj-11/_myobj 2022-09-09T22:33:02.942 DEBUG:boto:Method: GET 2022-09-09T22:33:02.943 DEBUG:boto:Path: /jkcgbj-11/_myobj 2022-09-09T22:33:02.944 DEBUG:boto:Data: 2022-09-09T22:33:02.945 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.947 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:02.948 DEBUG:boto:Port: 8000 2022-09-09T22:33:02.948 DEBUG:boto:Params: {} 2022-09-09T22:33:02.949 DEBUG:boto:Token: None 2022-09-09T22:33:02.950 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/_myobj 2022-09-09T22:33:02.950 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:iSPrbiKkPuDMMo/6U9Ims61A914= 2022-09-09T22:33:02.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:iSPrbiKkPuDMMo/6U9Ims61A914=', 'Content-Length': '0'} 2022-09-09T22:33:02.959 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa4f212038d5af21-00631bbf1e-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.960 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:02.961 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:02.961 DEBUG:boto:path=/jkcgbj-11/?prefix=_myobj 2022-09-09T22:33:02.962 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=_myobj 2022-09-09T22:33:02.963 DEBUG:boto:Method: GET 2022-09-09T22:33:02.963 DEBUG:boto:Path: /jkcgbj-11/?prefix=_myobj 2022-09-09T22:33:02.964 DEBUG:boto:Data: 2022-09-09T22:33:02.965 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.965 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:02.966 DEBUG:boto:Port: 8001 2022-09-09T22:33:02.966 DEBUG:boto:Params: {} 2022-09-09T22:33:02.969 DEBUG:boto:Token: None 2022-09-09T22:33:02.970 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/ 2022-09-09T22:33:02.971 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qNdIR7/XPPHXh+21qpmZNgQakj8= 2022-09-09T22:33:02.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qNdIR7/XPPHXh+21qpmZNgQakj8=', 'Content-Length': '0'} 2022-09-09T22:33:02.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000959e15392490b27c-00631bbf1e-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:02.979 DEBUG:boto:b'jkcgbj-11_myobj1000false_myobj2022-09-09T22:30:44.357Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:02.981 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:02.982 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:02.983 DEBUG:boto:path=/jkcgbj-11/?prefix=_myobj 2022-09-09T22:33:02.986 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=_myobj 2022-09-09T22:33:02.986 DEBUG:boto:Method: GET 2022-09-09T22:33:02.987 DEBUG:boto:Path: /jkcgbj-11/?prefix=_myobj 2022-09-09T22:33:02.988 DEBUG:boto:Data: 2022-09-09T22:33:02.988 DEBUG:boto:Headers: {} 2022-09-09T22:33:02.989 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:02.990 DEBUG:boto:Port: 8000 2022-09-09T22:33:02.991 DEBUG:boto:Params: {} 2022-09-09T22:33:02.992 DEBUG:boto:Token: None 2022-09-09T22:33:02.993 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:02 GMT /jkcgbj-11/ 2022-09-09T22:33:02.994 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qNdIR7/XPPHXh+21qpmZNgQakj8= 2022-09-09T22:33:02.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qNdIR7/XPPHXh+21qpmZNgQakj8=', 'Content-Length': '0'} 2022-09-09T22:33:03.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d3ebf99f0308658-00631bbf1e-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.003 DEBUG:boto:b'jkcgbj-11_myobj1000false_myobj2022-09-09T22:30:44.357Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:03.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:33:03.006 DEBUG:boto:path=/jkcgbj-11/myobj 2022-09-09T22:33:03.007 DEBUG:boto:auth_path=/jkcgbj-11/myobj 2022-09-09T22:33:03.008 DEBUG:boto:path=/jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:03.010 DEBUG:boto:auth_path=/jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:03.011 DEBUG:boto:Method: GET 2022-09-09T22:33:03.014 DEBUG:boto:Path: /jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:03.016 DEBUG:boto:Data: 2022-09-09T22:33:03.017 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.018 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:03.019 DEBUG:boto:Port: 8001 2022-09-09T22:33:03.020 DEBUG:boto:Params: {} 2022-09-09T22:33:03.021 DEBUG:boto:Token: None 2022-09-09T22:33:03.022 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:03.023 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ka43o3P3ZfpJnhIwbhRtWUDifYg= 2022-09-09T22:33:03.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ka43o3P3ZfpJnhIwbhRtWUDifYg=', 'Content-Length': '0'} 2022-09-09T22:33:03.028 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0684a137c107856-00631bbf1f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.030 DEBUG:boto:path=/jkcgbj-11/myobj 2022-09-09T22:33:03.031 DEBUG:boto:auth_path=/jkcgbj-11/myobj 2022-09-09T22:33:03.031 DEBUG:boto:path=/jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:03.032 DEBUG:boto:auth_path=/jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:03.033 DEBUG:boto:Method: GET 2022-09-09T22:33:03.035 DEBUG:boto:Path: /jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:03.036 DEBUG:boto:Data: 2022-09-09T22:33:03.036 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.037 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:03.039 DEBUG:boto:Port: 8000 2022-09-09T22:33:03.040 DEBUG:boto:Params: {} 2022-09-09T22:33:03.041 DEBUG:boto:Token: None 2022-09-09T22:33:03.042 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:03.043 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ka43o3P3ZfpJnhIwbhRtWUDifYg= 2022-09-09T22:33:03.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ka43o3P3ZfpJnhIwbhRtWUDifYg=', 'Content-Length': '0'} 2022-09-09T22:33:03.049 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000edfbf525b94ad395-00631bbf1f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.051 DEBUG:boto:path=/jkcgbj-11/myobj 2022-09-09T22:33:03.052 DEBUG:boto:auth_path=/jkcgbj-11/myobj 2022-09-09T22:33:03.053 DEBUG:boto:path=/jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:03.053 DEBUG:boto:auth_path=/jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:03.054 DEBUG:boto:Method: HEAD 2022-09-09T22:33:03.055 DEBUG:boto:Path: /jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:03.055 DEBUG:boto:Data: 2022-09-09T22:33:03.056 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.057 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:03.057 DEBUG:boto:Port: 8001 2022-09-09T22:33:03.058 DEBUG:boto:Params: {} 2022-09-09T22:33:03.058 DEBUG:boto:Token: None 2022-09-09T22:33:03.059 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:03.060 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0Z3NdgTM44CFSGROVYmZyODxvb4= 2022-09-09T22:33:03.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0Z3NdgTM44CFSGROVYmZyODxvb4=', 'Content-Length': '0'} 2022-09-09T22:33:03.066 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000588bc70e1f45571d-00631bbf1f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.068 DEBUG:boto:path=/jkcgbj-11/myobj 2022-09-09T22:33:03.069 DEBUG:boto:auth_path=/jkcgbj-11/myobj 2022-09-09T22:33:03.070 DEBUG:boto:path=/jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:03.071 DEBUG:boto:auth_path=/jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:03.073 DEBUG:boto:Method: HEAD 2022-09-09T22:33:03.074 DEBUG:boto:Path: /jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:03.075 DEBUG:boto:Data: 2022-09-09T22:33:03.075 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.076 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:03.076 DEBUG:boto:Port: 8000 2022-09-09T22:33:03.077 DEBUG:boto:Params: {} 2022-09-09T22:33:03.077 DEBUG:boto:Token: None 2022-09-09T22:33:03.078 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:03.078 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0Z3NdgTM44CFSGROVYmZyODxvb4= 2022-09-09T22:33:03.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0Z3NdgTM44CFSGROVYmZyODxvb4=', 'Content-Length': '0'} 2022-09-09T22:33:03.082 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000023aaf6e8cbc50096-00631bbf1f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:33:03.084 DEBUG:boto:path=/jkcgbj-11/myobj 2022-09-09T22:33:03.085 DEBUG:boto:auth_path=/jkcgbj-11/myobj 2022-09-09T22:33:03.085 DEBUG:boto:path=/jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:03.086 DEBUG:boto:auth_path=/jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:03.087 DEBUG:boto:Method: GET 2022-09-09T22:33:03.087 DEBUG:boto:Path: /jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:03.088 DEBUG:boto:Data: 2022-09-09T22:33:03.089 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.090 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:03.091 DEBUG:boto:Port: 8001 2022-09-09T22:33:03.092 DEBUG:boto:Params: {} 2022-09-09T22:33:03.092 DEBUG:boto:Token: None 2022-09-09T22:33:03.095 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:03.096 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ka43o3P3ZfpJnhIwbhRtWUDifYg= 2022-09-09T22:33:03.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ka43o3P3ZfpJnhIwbhRtWUDifYg=', 'Content-Length': '0'} 2022-09-09T22:33:03.104 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000599708c0af24ad48-00631bbf1f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.105 DEBUG:boto:path=/jkcgbj-11/myobj 2022-09-09T22:33:03.106 DEBUG:boto:auth_path=/jkcgbj-11/myobj 2022-09-09T22:33:03.107 DEBUG:boto:path=/jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:03.107 DEBUG:boto:auth_path=/jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:03.108 DEBUG:boto:Method: GET 2022-09-09T22:33:03.109 DEBUG:boto:Path: /jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:03.110 DEBUG:boto:Data: 2022-09-09T22:33:03.110 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.111 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:03.111 DEBUG:boto:Port: 8000 2022-09-09T22:33:03.112 DEBUG:boto:Params: {} 2022-09-09T22:33:03.113 DEBUG:boto:Token: None 2022-09-09T22:33:03.113 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-11/myobj?versionId=null 2022-09-09T22:33:03.115 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ka43o3P3ZfpJnhIwbhRtWUDifYg= 2022-09-09T22:33:03.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ka43o3P3ZfpJnhIwbhRtWUDifYg=', 'Content-Length': '0'} 2022-09-09T22:33:03.126 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0b756ebc67c549d-00631bbf1f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.174 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:03.175 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:03.176 DEBUG:boto:path=/jkcgbj-11/?prefix=myobj 2022-09-09T22:33:03.177 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=myobj 2022-09-09T22:33:03.178 DEBUG:boto:Method: GET 2022-09-09T22:33:03.179 DEBUG:boto:Path: /jkcgbj-11/?prefix=myobj 2022-09-09T22:33:03.180 DEBUG:boto:Data: 2022-09-09T22:33:03.181 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.183 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:03.184 DEBUG:boto:Port: 8001 2022-09-09T22:33:03.185 DEBUG:boto:Params: {} 2022-09-09T22:33:03.186 DEBUG:boto:Token: None 2022-09-09T22:33:03.187 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-11/ 2022-09-09T22:33:03.188 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wk/I0Ay5DB3ZAT2JPoXZrSbDgWQ= 2022-09-09T22:33:03.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wk/I0Ay5DB3ZAT2JPoXZrSbDgWQ=', 'Content-Length': '0'} 2022-09-09T22:33:03.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7f2f75c60d28fb3-00631bbf1f-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.203 DEBUG:boto:b'jkcgbj-11myobj1000falsemyobj2022-09-09T22:30:44.302Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:03.204 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:03.205 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:03.205 DEBUG:boto:path=/jkcgbj-11/?prefix=myobj 2022-09-09T22:33:03.206 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=myobj 2022-09-09T22:33:03.206 DEBUG:boto:Method: GET 2022-09-09T22:33:03.207 DEBUG:boto:Path: /jkcgbj-11/?prefix=myobj 2022-09-09T22:33:03.207 DEBUG:boto:Data: 2022-09-09T22:33:03.208 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.209 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:03.210 DEBUG:boto:Port: 8000 2022-09-09T22:33:03.210 DEBUG:boto:Params: {} 2022-09-09T22:33:03.212 DEBUG:boto:Token: None 2022-09-09T22:33:03.213 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-11/ 2022-09-09T22:33:03.214 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wk/I0Ay5DB3ZAT2JPoXZrSbDgWQ= 2022-09-09T22:33:03.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wk/I0Ay5DB3ZAT2JPoXZrSbDgWQ=', 'Content-Length': '0'} 2022-09-09T22:33:03.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2f523592bf68e12-00631bbf1f-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.223 DEBUG:boto:b'jkcgbj-11myobj1000falsemyobj2022-09-09T22:30:44.302Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:03.224 DEBUG:boto:path=/jkcgbj-11/myobj 2022-09-09T22:33:03.224 DEBUG:boto:auth_path=/jkcgbj-11/myobj 2022-09-09T22:33:03.225 DEBUG:boto:Method: HEAD 2022-09-09T22:33:03.225 DEBUG:boto:Path: /jkcgbj-11/myobj 2022-09-09T22:33:03.226 DEBUG:boto:Data: 2022-09-09T22:33:03.226 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.227 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:03.227 DEBUG:boto:Port: 8001 2022-09-09T22:33:03.228 DEBUG:boto:Params: {} 2022-09-09T22:33:03.228 DEBUG:boto:Token: None 2022-09-09T22:33:03.229 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-11/myobj 2022-09-09T22:33:03.229 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:l1Tyx0dEmUpifnivZ1OAGlvEP54= 2022-09-09T22:33:03.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:l1Tyx0dEmUpifnivZ1OAGlvEP54=', 'Content-Length': '0'} 2022-09-09T22:33:03.234 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081238bff04ec44b7-00631bbf1f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.235 DEBUG:boto:path=/jkcgbj-11/myobj 2022-09-09T22:33:03.235 DEBUG:boto:auth_path=/jkcgbj-11/myobj 2022-09-09T22:33:03.236 DEBUG:boto:Method: HEAD 2022-09-09T22:33:03.236 DEBUG:boto:Path: /jkcgbj-11/myobj 2022-09-09T22:33:03.237 DEBUG:boto:Data: 2022-09-09T22:33:03.238 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.238 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:03.239 DEBUG:boto:Port: 8000 2022-09-09T22:33:03.239 DEBUG:boto:Params: {} 2022-09-09T22:33:03.240 DEBUG:boto:Token: None 2022-09-09T22:33:03.240 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-11/myobj 2022-09-09T22:33:03.241 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:l1Tyx0dEmUpifnivZ1OAGlvEP54= 2022-09-09T22:33:03.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:l1Tyx0dEmUpifnivZ1OAGlvEP54=', 'Content-Length': '0'} 2022-09-09T22:33:03.245 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a3c703842e564556-00631bbf1f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:33:03.246 DEBUG:boto:path=/jkcgbj-11/myobj 2022-09-09T22:33:03.247 DEBUG:boto:auth_path=/jkcgbj-11/myobj 2022-09-09T22:33:03.247 DEBUG:boto:Method: GET 2022-09-09T22:33:03.248 DEBUG:boto:Path: /jkcgbj-11/myobj 2022-09-09T22:33:03.249 DEBUG:boto:Data: 2022-09-09T22:33:03.250 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.251 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:03.252 DEBUG:boto:Port: 8001 2022-09-09T22:33:03.253 DEBUG:boto:Params: {} 2022-09-09T22:33:03.254 DEBUG:boto:Token: None 2022-09-09T22:33:03.254 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-11/myobj 2022-09-09T22:33:03.255 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:a7Kgif/dZDMy/+DXhRQ44tcq/OQ= 2022-09-09T22:33:03.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:a7Kgif/dZDMy/+DXhRQ44tcq/OQ=', 'Content-Length': '0'} 2022-09-09T22:33:03.260 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d6ce4ea9aa087e4-00631bbf1f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.306 DEBUG:boto:path=/jkcgbj-11/myobj 2022-09-09T22:33:03.307 DEBUG:boto:auth_path=/jkcgbj-11/myobj 2022-09-09T22:33:03.308 DEBUG:boto:Method: GET 2022-09-09T22:33:03.308 DEBUG:boto:Path: /jkcgbj-11/myobj 2022-09-09T22:33:03.309 DEBUG:boto:Data: 2022-09-09T22:33:03.310 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.310 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:03.311 DEBUG:boto:Port: 8000 2022-09-09T22:33:03.312 DEBUG:boto:Params: {} 2022-09-09T22:33:03.313 DEBUG:boto:Token: None 2022-09-09T22:33:03.314 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-11/myobj 2022-09-09T22:33:03.315 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:a7Kgif/dZDMy/+DXhRQ44tcq/OQ= 2022-09-09T22:33:03.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:a7Kgif/dZDMy/+DXhRQ44tcq/OQ=', 'Content-Length': '0'} 2022-09-09T22:33:03.320 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dbf80aad832d50c5-00631bbf1f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.321 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:03.322 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:03.323 DEBUG:boto:path=/jkcgbj-11/?prefix=myobj 2022-09-09T22:33:03.324 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=myobj 2022-09-09T22:33:03.325 DEBUG:boto:Method: GET 2022-09-09T22:33:03.326 DEBUG:boto:Path: /jkcgbj-11/?prefix=myobj 2022-09-09T22:33:03.327 DEBUG:boto:Data: 2022-09-09T22:33:03.327 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.328 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:03.329 DEBUG:boto:Port: 8001 2022-09-09T22:33:03.330 DEBUG:boto:Params: {} 2022-09-09T22:33:03.331 DEBUG:boto:Token: None 2022-09-09T22:33:03.332 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-11/ 2022-09-09T22:33:03.333 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wk/I0Ay5DB3ZAT2JPoXZrSbDgWQ= 2022-09-09T22:33:03.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wk/I0Ay5DB3ZAT2JPoXZrSbDgWQ=', 'Content-Length': '0'} 2022-09-09T22:33:03.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b656da2a26c19802-00631bbf1f-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.341 DEBUG:boto:b'jkcgbj-11myobj1000falsemyobj2022-09-09T22:30:44.302Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:03.342 DEBUG:boto:path=/jkcgbj-11/ 2022-09-09T22:33:03.342 DEBUG:boto:auth_path=/jkcgbj-11/ 2022-09-09T22:33:03.343 DEBUG:boto:path=/jkcgbj-11/?prefix=myobj 2022-09-09T22:33:03.344 DEBUG:boto:auth_path=/jkcgbj-11/?prefix=myobj 2022-09-09T22:33:03.344 DEBUG:boto:Method: GET 2022-09-09T22:33:03.345 DEBUG:boto:Path: /jkcgbj-11/?prefix=myobj 2022-09-09T22:33:03.345 DEBUG:boto:Data: 2022-09-09T22:33:03.346 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.346 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:03.347 DEBUG:boto:Port: 8000 2022-09-09T22:33:03.347 DEBUG:boto:Params: {} 2022-09-09T22:33:03.348 DEBUG:boto:Token: None 2022-09-09T22:33:03.349 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-11/ 2022-09-09T22:33:03.349 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wk/I0Ay5DB3ZAT2JPoXZrSbDgWQ= 2022-09-09T22:33:03.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wk/I0Ay5DB3ZAT2JPoXZrSbDgWQ=', 'Content-Length': '0'} 2022-09-09T22:33:03.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c05ebf20973436b4-00631bbf1f-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.356 DEBUG:boto:b'jkcgbj-11myobj1000falsemyobj2022-09-09T22:30:44.302Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:03.357 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-11 zones={test-zone3, test-zone2} 2022-09-09T22:33:03.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:33:03.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:33:03.359 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:33:03.412 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:33:03.412 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:33:03.482 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:33:03.480+0000 7fe784b65dc0 1 bucket sync caught up with source: 2022-09-09T22:33:03.483 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000001.21.5, , 00000000001.15.5, , , , 00000000001.23.5, 00000000001.20.5, , ] 2022-09-09T22:33:03.483 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000001.21.5, , 00000000001.15.5, , , , 00000000001.23.5, 00000000001.20.5, , ] 2022-09-09T22:33:03.483 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:33:03.480+0000 7fe784b65dc0 0 bucket checkpoint complete 2022-09-09T22:33:03.508 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-12 zones={test-zone1, test-zone3} 2022-09-09T22:33:03.509 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:03.509 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:03.510 DEBUG:boto:Method: HEAD 2022-09-09T22:33:03.511 DEBUG:boto:Path: /jkcgbj-12/ 2022-09-09T22:33:03.512 DEBUG:boto:Data: 2022-09-09T22:33:03.513 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.514 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:03.515 DEBUG:boto:Port: 8000 2022-09-09T22:33:03.515 DEBUG:boto:Params: {} 2022-09-09T22:33:03.516 DEBUG:boto:Token: None 2022-09-09T22:33:03.517 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-12/ 2022-09-09T22:33:03.518 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ZVQdxstwV3bAmCPdXzo5XbKSwKI= 2022-09-09T22:33:03.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ZVQdxstwV3bAmCPdXzo5XbKSwKI=', 'Content-Length': '0'} 2022-09-09T22:33:03.524 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8e7c2318b614841-00631bbf1f-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.526 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:03.527 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:03.528 DEBUG:boto:Method: HEAD 2022-09-09T22:33:03.529 DEBUG:boto:Path: /jkcgbj-12/ 2022-09-09T22:33:03.530 DEBUG:boto:Data: 2022-09-09T22:33:03.531 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.531 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:03.532 DEBUG:boto:Port: 8001 2022-09-09T22:33:03.533 DEBUG:boto:Params: {} 2022-09-09T22:33:03.534 DEBUG:boto:Token: None 2022-09-09T22:33:03.536 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-12/ 2022-09-09T22:33:03.537 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ZVQdxstwV3bAmCPdXzo5XbKSwKI= 2022-09-09T22:33:03.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ZVQdxstwV3bAmCPdXzo5XbKSwKI=', 'Content-Length': '0'} 2022-09-09T22:33:03.543 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1cc7215237d1058-00631bbf1f-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.544 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:33:03.545 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:03.545 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:03.546 DEBUG:boto:path=/jkcgbj-12/?versions&max-keys=999 2022-09-09T22:33:03.547 DEBUG:boto:auth_path=/jkcgbj-12/?versions&max-keys=999 2022-09-09T22:33:03.548 DEBUG:boto:Method: GET 2022-09-09T22:33:03.550 DEBUG:boto:Path: /jkcgbj-12/?versions&max-keys=999 2022-09-09T22:33:03.550 DEBUG:boto:Data: 2022-09-09T22:33:03.551 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.552 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:03.553 DEBUG:boto:Port: 8000 2022-09-09T22:33:03.553 DEBUG:boto:Params: {} 2022-09-09T22:33:03.554 DEBUG:boto:Token: None 2022-09-09T22:33:03.555 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-12/?versions 2022-09-09T22:33:03.555 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CrM3ZNDcy2sfftlFkbyvqDMZFao= 2022-09-09T22:33:03.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CrM3ZNDcy2sfftlFkbyvqDMZFao=', 'Content-Length': '0'} 2022-09-09T22:33:03.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000976a54592de8953a-00631bbf1f-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.563 DEBUG:boto:b'jkcgbj-12999false&nulltrue2022-09-09T22:30:44.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-09T22:30:44.766Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-09T22:30:44.684Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-09T22:30:44.625Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:03.565 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-09T22:33:03.565 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-09T22:33:03.566 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-09T22:33:03.566 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-09T22:33:03.567 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:33:03.568 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:03.568 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:03.569 DEBUG:boto:path=/jkcgbj-12/?versions&max-keys=999 2022-09-09T22:33:03.570 DEBUG:boto:auth_path=/jkcgbj-12/?versions&max-keys=999 2022-09-09T22:33:03.570 DEBUG:boto:Method: GET 2022-09-09T22:33:03.571 DEBUG:boto:Path: /jkcgbj-12/?versions&max-keys=999 2022-09-09T22:33:03.571 DEBUG:boto:Data: 2022-09-09T22:33:03.572 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.572 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:03.573 DEBUG:boto:Port: 8001 2022-09-09T22:33:03.574 DEBUG:boto:Params: {} 2022-09-09T22:33:03.574 DEBUG:boto:Token: None 2022-09-09T22:33:03.575 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-12/?versions 2022-09-09T22:33:03.576 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CrM3ZNDcy2sfftlFkbyvqDMZFao= 2022-09-09T22:33:03.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CrM3ZNDcy2sfftlFkbyvqDMZFao=', 'Content-Length': '0'} 2022-09-09T22:33:03.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c6b747df3ace7b1-00631bbf1f-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.583 DEBUG:boto:b'jkcgbj-12999false&nulltrue2022-09-09T22:30:44.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-09T22:30:44.766Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-09T22:30:44.684Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-09T22:30:44.625Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:03.584 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-09T22:33:03.584 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-09T22:33:03.585 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-09T22:33:03.585 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-09T22:33:03.586 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:03.587 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:03.587 DEBUG:boto:path=/jkcgbj-12/?versions&max-keys=999 2022-09-09T22:33:03.588 DEBUG:boto:auth_path=/jkcgbj-12/?versions&max-keys=999 2022-09-09T22:33:03.589 DEBUG:boto:Method: GET 2022-09-09T22:33:03.589 DEBUG:boto:Path: /jkcgbj-12/?versions&max-keys=999 2022-09-09T22:33:03.590 DEBUG:boto:Data: 2022-09-09T22:33:03.591 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.592 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:03.592 DEBUG:boto:Port: 8000 2022-09-09T22:33:03.593 DEBUG:boto:Params: {} 2022-09-09T22:33:03.593 DEBUG:boto:Token: None 2022-09-09T22:33:03.594 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-12/?versions 2022-09-09T22:33:03.595 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CrM3ZNDcy2sfftlFkbyvqDMZFao= 2022-09-09T22:33:03.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CrM3ZNDcy2sfftlFkbyvqDMZFao=', 'Content-Length': '0'} 2022-09-09T22:33:03.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006192525a9471c9fc-00631bbf1f-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.601 DEBUG:boto:b'jkcgbj-12999false&nulltrue2022-09-09T22:30:44.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-09T22:30:44.766Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-09T22:30:44.684Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-09T22:30:44.625Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:03.602 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:03.603 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:03.603 DEBUG:boto:path=/jkcgbj-12/?versions&max-keys=999 2022-09-09T22:33:03.604 DEBUG:boto:auth_path=/jkcgbj-12/?versions&max-keys=999 2022-09-09T22:33:03.604 DEBUG:boto:Method: GET 2022-09-09T22:33:03.605 DEBUG:boto:Path: /jkcgbj-12/?versions&max-keys=999 2022-09-09T22:33:03.605 DEBUG:boto:Data: 2022-09-09T22:33:03.606 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.607 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:03.607 DEBUG:boto:Port: 8001 2022-09-09T22:33:03.608 DEBUG:boto:Params: {} 2022-09-09T22:33:03.608 DEBUG:boto:Token: None 2022-09-09T22:33:03.609 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-12/?versions 2022-09-09T22:33:03.610 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CrM3ZNDcy2sfftlFkbyvqDMZFao= 2022-09-09T22:33:03.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CrM3ZNDcy2sfftlFkbyvqDMZFao=', 'Content-Length': '0'} 2022-09-09T22:33:03.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003097dae1c0e7a078-00631bbf1f-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.627 DEBUG:boto:b'jkcgbj-12999false&nulltrue2022-09-09T22:30:44.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-09T22:30:44.766Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-09T22:30:44.684Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-09T22:30:44.625Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:03.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-09T22:33:03.629 DEBUG:boto:path=/jkcgbj-12/%26 2022-09-09T22:33:03.630 DEBUG:boto:auth_path=/jkcgbj-12/%26 2022-09-09T22:33:03.631 DEBUG:boto:path=/jkcgbj-12/%26?versionId=null 2022-09-09T22:33:03.634 DEBUG:boto:auth_path=/jkcgbj-12/%26?versionId=null 2022-09-09T22:33:03.635 DEBUG:boto:Method: GET 2022-09-09T22:33:03.635 DEBUG:boto:Path: /jkcgbj-12/%26?versionId=null 2022-09-09T22:33:03.636 DEBUG:boto:Data: 2022-09-09T22:33:03.637 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.637 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:03.638 DEBUG:boto:Port: 8000 2022-09-09T22:33:03.638 DEBUG:boto:Params: {} 2022-09-09T22:33:03.639 DEBUG:boto:Token: None 2022-09-09T22:33:03.640 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-12/%26?versionId=null 2022-09-09T22:33:03.641 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CbOIuUvZzsui7QsfuBwPKTfxX2k= 2022-09-09T22:33:03.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CbOIuUvZzsui7QsfuBwPKTfxX2k=', 'Content-Length': '0'} 2022-09-09T22:33:03.645 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb95c5905d66dadf-00631bbf1f-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.647 DEBUG:boto:path=/jkcgbj-12/%26 2022-09-09T22:33:03.648 DEBUG:boto:auth_path=/jkcgbj-12/%26 2022-09-09T22:33:03.649 DEBUG:boto:path=/jkcgbj-12/%26?versionId=null 2022-09-09T22:33:03.650 DEBUG:boto:auth_path=/jkcgbj-12/%26?versionId=null 2022-09-09T22:33:03.651 DEBUG:boto:Method: GET 2022-09-09T22:33:03.652 DEBUG:boto:Path: /jkcgbj-12/%26?versionId=null 2022-09-09T22:33:03.653 DEBUG:boto:Data: 2022-09-09T22:33:03.654 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.655 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:03.656 DEBUG:boto:Port: 8001 2022-09-09T22:33:03.657 DEBUG:boto:Params: {} 2022-09-09T22:33:03.658 DEBUG:boto:Token: None 2022-09-09T22:33:03.659 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-12/%26?versionId=null 2022-09-09T22:33:03.660 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CbOIuUvZzsui7QsfuBwPKTfxX2k= 2022-09-09T22:33:03.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CbOIuUvZzsui7QsfuBwPKTfxX2k=', 'Content-Length': '0'} 2022-09-09T22:33:03.664 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000575b00a051a1b018-00631bbf1f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.706 DEBUG:boto:path=/jkcgbj-12/%26 2022-09-09T22:33:03.707 DEBUG:boto:auth_path=/jkcgbj-12/%26 2022-09-09T22:33:03.707 DEBUG:boto:path=/jkcgbj-12/%26?versionId=null 2022-09-09T22:33:03.708 DEBUG:boto:auth_path=/jkcgbj-12/%26?versionId=null 2022-09-09T22:33:03.709 DEBUG:boto:Method: HEAD 2022-09-09T22:33:03.709 DEBUG:boto:Path: /jkcgbj-12/%26?versionId=null 2022-09-09T22:33:03.710 DEBUG:boto:Data: 2022-09-09T22:33:03.711 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.711 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:03.712 DEBUG:boto:Port: 8000 2022-09-09T22:33:03.712 DEBUG:boto:Params: {} 2022-09-09T22:33:03.713 DEBUG:boto:Token: None 2022-09-09T22:33:03.713 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-12/%26?versionId=null 2022-09-09T22:33:03.714 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:3nN3LR3xuSxLI+fPx9dYbenJ97c= 2022-09-09T22:33:03.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:3nN3LR3xuSxLI+fPx9dYbenJ97c=', 'Content-Length': '0'} 2022-09-09T22:33:03.718 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ad5f351355b11129-00631bbf1f-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.718 DEBUG:boto:path=/jkcgbj-12/%26 2022-09-09T22:33:03.719 DEBUG:boto:auth_path=/jkcgbj-12/%26 2022-09-09T22:33:03.720 DEBUG:boto:path=/jkcgbj-12/%26?versionId=null 2022-09-09T22:33:03.720 DEBUG:boto:auth_path=/jkcgbj-12/%26?versionId=null 2022-09-09T22:33:03.721 DEBUG:boto:Method: HEAD 2022-09-09T22:33:03.721 DEBUG:boto:Path: /jkcgbj-12/%26?versionId=null 2022-09-09T22:33:03.722 DEBUG:boto:Data: 2022-09-09T22:33:03.722 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.723 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:03.723 DEBUG:boto:Port: 8001 2022-09-09T22:33:03.724 DEBUG:boto:Params: {} 2022-09-09T22:33:03.724 DEBUG:boto:Token: None 2022-09-09T22:33:03.725 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-12/%26?versionId=null 2022-09-09T22:33:03.726 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:3nN3LR3xuSxLI+fPx9dYbenJ97c= 2022-09-09T22:33:03.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:3nN3LR3xuSxLI+fPx9dYbenJ97c=', 'Content-Length': '0'} 2022-09-09T22:33:03.729 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021077ccae1065735-00631bbf1f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-09T22:33:03.730 DEBUG:boto:path=/jkcgbj-12/%26 2022-09-09T22:33:03.731 DEBUG:boto:auth_path=/jkcgbj-12/%26 2022-09-09T22:33:03.732 DEBUG:boto:path=/jkcgbj-12/%26?versionId=null 2022-09-09T22:33:03.732 DEBUG:boto:auth_path=/jkcgbj-12/%26?versionId=null 2022-09-09T22:33:03.733 DEBUG:boto:Method: GET 2022-09-09T22:33:03.733 DEBUG:boto:Path: /jkcgbj-12/%26?versionId=null 2022-09-09T22:33:03.734 DEBUG:boto:Data: 2022-09-09T22:33:03.734 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.735 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:03.735 DEBUG:boto:Port: 8000 2022-09-09T22:33:03.736 DEBUG:boto:Params: {} 2022-09-09T22:33:03.736 DEBUG:boto:Token: None 2022-09-09T22:33:03.737 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-12/%26?versionId=null 2022-09-09T22:33:03.738 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CbOIuUvZzsui7QsfuBwPKTfxX2k= 2022-09-09T22:33:03.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CbOIuUvZzsui7QsfuBwPKTfxX2k=', 'Content-Length': '0'} 2022-09-09T22:33:03.741 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030713e13f380e2c4-00631bbf1f-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.742 DEBUG:boto:path=/jkcgbj-12/%26 2022-09-09T22:33:03.743 DEBUG:boto:auth_path=/jkcgbj-12/%26 2022-09-09T22:33:03.743 DEBUG:boto:path=/jkcgbj-12/%26?versionId=null 2022-09-09T22:33:03.744 DEBUG:boto:auth_path=/jkcgbj-12/%26?versionId=null 2022-09-09T22:33:03.745 DEBUG:boto:Method: GET 2022-09-09T22:33:03.746 DEBUG:boto:Path: /jkcgbj-12/%26?versionId=null 2022-09-09T22:33:03.746 DEBUG:boto:Data: 2022-09-09T22:33:03.747 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.748 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:03.748 DEBUG:boto:Port: 8001 2022-09-09T22:33:03.749 DEBUG:boto:Params: {} 2022-09-09T22:33:03.750 DEBUG:boto:Token: None 2022-09-09T22:33:03.750 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-12/%26?versionId=null 2022-09-09T22:33:03.751 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CbOIuUvZzsui7QsfuBwPKTfxX2k= 2022-09-09T22:33:03.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CbOIuUvZzsui7QsfuBwPKTfxX2k=', 'Content-Length': '0'} 2022-09-09T22:33:03.754 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000505970bb7e8be1b6-00631bbf1f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.755 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:03.756 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:03.757 DEBUG:boto:path=/jkcgbj-12/?prefix=%26 2022-09-09T22:33:03.757 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=%26 2022-09-09T22:33:03.758 DEBUG:boto:Method: GET 2022-09-09T22:33:03.759 DEBUG:boto:Path: /jkcgbj-12/?prefix=%26 2022-09-09T22:33:03.759 DEBUG:boto:Data: 2022-09-09T22:33:03.760 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.760 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:03.761 DEBUG:boto:Port: 8000 2022-09-09T22:33:03.762 DEBUG:boto:Params: {} 2022-09-09T22:33:03.762 DEBUG:boto:Token: None 2022-09-09T22:33:03.763 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-12/ 2022-09-09T22:33:03.764 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:gUUb+SmpsJGRaiflCzrwlMi1cJg= 2022-09-09T22:33:03.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:gUUb+SmpsJGRaiflCzrwlMi1cJg=', 'Content-Length': '0'} 2022-09-09T22:33:03.770 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b49fde72850dab0f-00631bbf1f-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.771 DEBUG:boto:b'jkcgbj-12&1000false&2022-09-09T22:30:44.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:03.772 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:03.772 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:03.773 DEBUG:boto:path=/jkcgbj-12/?prefix=%26 2022-09-09T22:33:03.774 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=%26 2022-09-09T22:33:03.774 DEBUG:boto:Method: GET 2022-09-09T22:33:03.775 DEBUG:boto:Path: /jkcgbj-12/?prefix=%26 2022-09-09T22:33:03.775 DEBUG:boto:Data: 2022-09-09T22:33:03.776 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.776 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:03.777 DEBUG:boto:Port: 8001 2022-09-09T22:33:03.777 DEBUG:boto:Params: {} 2022-09-09T22:33:03.778 DEBUG:boto:Token: None 2022-09-09T22:33:03.779 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-12/ 2022-09-09T22:33:03.780 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:gUUb+SmpsJGRaiflCzrwlMi1cJg= 2022-09-09T22:33:03.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:gUUb+SmpsJGRaiflCzrwlMi1cJg=', 'Content-Length': '0'} 2022-09-09T22:33:03.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007452196522bbc5c0-00631bbf1f-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.787 DEBUG:boto:b'jkcgbj-12&1000false&2022-09-09T22:30:44.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:03.788 DEBUG:boto:path=/jkcgbj-12/%26 2022-09-09T22:33:03.789 DEBUG:boto:auth_path=/jkcgbj-12/%26 2022-09-09T22:33:03.789 DEBUG:boto:Method: HEAD 2022-09-09T22:33:03.790 DEBUG:boto:Path: /jkcgbj-12/%26 2022-09-09T22:33:03.790 DEBUG:boto:Data: 2022-09-09T22:33:03.791 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.791 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:03.792 DEBUG:boto:Port: 8000 2022-09-09T22:33:03.792 DEBUG:boto:Params: {} 2022-09-09T22:33:03.793 DEBUG:boto:Token: None 2022-09-09T22:33:03.794 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-12/%26 2022-09-09T22:33:03.794 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JyKq0vXh5AY/dmjOiSyYd+h2Zy0= 2022-09-09T22:33:03.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JyKq0vXh5AY/dmjOiSyYd+h2Zy0=', 'Content-Length': '0'} 2022-09-09T22:33:03.797 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c1154656b327ef0b-00631bbf1f-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.798 DEBUG:boto:path=/jkcgbj-12/%26 2022-09-09T22:33:03.799 DEBUG:boto:auth_path=/jkcgbj-12/%26 2022-09-09T22:33:03.799 DEBUG:boto:Method: HEAD 2022-09-09T22:33:03.800 DEBUG:boto:Path: /jkcgbj-12/%26 2022-09-09T22:33:03.800 DEBUG:boto:Data: 2022-09-09T22:33:03.801 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.801 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:03.802 DEBUG:boto:Port: 8001 2022-09-09T22:33:03.802 DEBUG:boto:Params: {} 2022-09-09T22:33:03.803 DEBUG:boto:Token: None 2022-09-09T22:33:03.804 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-12/%26 2022-09-09T22:33:03.804 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JyKq0vXh5AY/dmjOiSyYd+h2Zy0= 2022-09-09T22:33:03.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JyKq0vXh5AY/dmjOiSyYd+h2Zy0=', 'Content-Length': '0'} 2022-09-09T22:33:03.808 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ce533b623b7b58b-00631bbf1f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-09T22:33:03.810 DEBUG:boto:path=/jkcgbj-12/%26 2022-09-09T22:33:03.811 DEBUG:boto:auth_path=/jkcgbj-12/%26 2022-09-09T22:33:03.812 DEBUG:boto:Method: GET 2022-09-09T22:33:03.813 DEBUG:boto:Path: /jkcgbj-12/%26 2022-09-09T22:33:03.814 DEBUG:boto:Data: 2022-09-09T22:33:03.814 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.815 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:03.816 DEBUG:boto:Port: 8000 2022-09-09T22:33:03.816 DEBUG:boto:Params: {} 2022-09-09T22:33:03.817 DEBUG:boto:Token: None 2022-09-09T22:33:03.818 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-12/%26 2022-09-09T22:33:03.819 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:16FReGuslkynvirrcndXQWVe5A4= 2022-09-09T22:33:03.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:16FReGuslkynvirrcndXQWVe5A4=', 'Content-Length': '0'} 2022-09-09T22:33:03.823 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000095958556c079474-00631bbf1f-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.866 DEBUG:boto:path=/jkcgbj-12/%26 2022-09-09T22:33:03.867 DEBUG:boto:auth_path=/jkcgbj-12/%26 2022-09-09T22:33:03.868 DEBUG:boto:Method: GET 2022-09-09T22:33:03.869 DEBUG:boto:Path: /jkcgbj-12/%26 2022-09-09T22:33:03.869 DEBUG:boto:Data: 2022-09-09T22:33:03.870 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.870 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:03.871 DEBUG:boto:Port: 8001 2022-09-09T22:33:03.871 DEBUG:boto:Params: {} 2022-09-09T22:33:03.872 DEBUG:boto:Token: None 2022-09-09T22:33:03.873 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-12/%26 2022-09-09T22:33:03.874 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:16FReGuslkynvirrcndXQWVe5A4= 2022-09-09T22:33:03.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:16FReGuslkynvirrcndXQWVe5A4=', 'Content-Length': '0'} 2022-09-09T22:33:03.878 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000efb9f43673bc4349-00631bbf1f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.879 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:03.879 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:03.880 DEBUG:boto:path=/jkcgbj-12/?prefix=%26 2022-09-09T22:33:03.880 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=%26 2022-09-09T22:33:03.881 DEBUG:boto:Method: GET 2022-09-09T22:33:03.882 DEBUG:boto:Path: /jkcgbj-12/?prefix=%26 2022-09-09T22:33:03.882 DEBUG:boto:Data: 2022-09-09T22:33:03.883 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.883 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:03.884 DEBUG:boto:Port: 8000 2022-09-09T22:33:03.884 DEBUG:boto:Params: {} 2022-09-09T22:33:03.885 DEBUG:boto:Token: None 2022-09-09T22:33:03.886 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-12/ 2022-09-09T22:33:03.887 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:gUUb+SmpsJGRaiflCzrwlMi1cJg= 2022-09-09T22:33:03.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:gUUb+SmpsJGRaiflCzrwlMi1cJg=', 'Content-Length': '0'} 2022-09-09T22:33:03.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ebf498225e9165a-00631bbf1f-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.894 DEBUG:boto:b'jkcgbj-12&1000false&2022-09-09T22:30:44.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:03.895 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:03.896 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:03.896 DEBUG:boto:path=/jkcgbj-12/?prefix=%26 2022-09-09T22:33:03.897 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=%26 2022-09-09T22:33:03.897 DEBUG:boto:Method: GET 2022-09-09T22:33:03.898 DEBUG:boto:Path: /jkcgbj-12/?prefix=%26 2022-09-09T22:33:03.899 DEBUG:boto:Data: 2022-09-09T22:33:03.899 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.900 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:03.900 DEBUG:boto:Port: 8001 2022-09-09T22:33:03.901 DEBUG:boto:Params: {} 2022-09-09T22:33:03.901 DEBUG:boto:Token: None 2022-09-09T22:33:03.902 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-12/ 2022-09-09T22:33:03.902 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:gUUb+SmpsJGRaiflCzrwlMi1cJg= 2022-09-09T22:33:03.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:gUUb+SmpsJGRaiflCzrwlMi1cJg=', 'Content-Length': '0'} 2022-09-09T22:33:03.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008145193339e0b1e1-00631bbf1f-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.909 DEBUG:boto:b'jkcgbj-12&1000false&2022-09-09T22:30:44.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:03.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-09T22:33:03.911 DEBUG:boto:path=/jkcgbj-12/%3A 2022-09-09T22:33:03.911 DEBUG:boto:auth_path=/jkcgbj-12/%3A 2022-09-09T22:33:03.912 DEBUG:boto:path=/jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:03.912 DEBUG:boto:auth_path=/jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:03.913 DEBUG:boto:Method: GET 2022-09-09T22:33:03.913 DEBUG:boto:Path: /jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:03.914 DEBUG:boto:Data: 2022-09-09T22:33:03.914 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.915 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:03.916 DEBUG:boto:Port: 8000 2022-09-09T22:33:03.916 DEBUG:boto:Params: {} 2022-09-09T22:33:03.917 DEBUG:boto:Token: None 2022-09-09T22:33:03.918 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:03.919 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ZeSPTjxIz6QVtyYVbleEjLzysus= 2022-09-09T22:33:03.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ZeSPTjxIz6QVtyYVbleEjLzysus=', 'Content-Length': '0'} 2022-09-09T22:33:03.922 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d4f9de4e6cffd923-00631bbf1f-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.923 DEBUG:boto:path=/jkcgbj-12/%3A 2022-09-09T22:33:03.924 DEBUG:boto:auth_path=/jkcgbj-12/%3A 2022-09-09T22:33:03.924 DEBUG:boto:path=/jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:03.925 DEBUG:boto:auth_path=/jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:03.925 DEBUG:boto:Method: GET 2022-09-09T22:33:03.926 DEBUG:boto:Path: /jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:03.927 DEBUG:boto:Data: 2022-09-09T22:33:03.927 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.928 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:03.928 DEBUG:boto:Port: 8001 2022-09-09T22:33:03.929 DEBUG:boto:Params: {} 2022-09-09T22:33:03.929 DEBUG:boto:Token: None 2022-09-09T22:33:03.930 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:03.930 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ZeSPTjxIz6QVtyYVbleEjLzysus= 2022-09-09T22:33:03.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ZeSPTjxIz6QVtyYVbleEjLzysus=', 'Content-Length': '0'} 2022-09-09T22:33:03.934 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000048e0672965a95fd8-00631bbf1f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.935 DEBUG:boto:path=/jkcgbj-12/%3A 2022-09-09T22:33:03.936 DEBUG:boto:auth_path=/jkcgbj-12/%3A 2022-09-09T22:33:03.937 DEBUG:boto:path=/jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:03.937 DEBUG:boto:auth_path=/jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:03.938 DEBUG:boto:Method: HEAD 2022-09-09T22:33:03.938 DEBUG:boto:Path: /jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:03.939 DEBUG:boto:Data: 2022-09-09T22:33:03.939 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.940 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:03.941 DEBUG:boto:Port: 8000 2022-09-09T22:33:03.941 DEBUG:boto:Params: {} 2022-09-09T22:33:03.942 DEBUG:boto:Token: None 2022-09-09T22:33:03.942 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:03.943 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:xiyHOlZEjsbzufJFf+tq8+rJkuU= 2022-09-09T22:33:03.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:xiyHOlZEjsbzufJFf+tq8+rJkuU=', 'Content-Length': '0'} 2022-09-09T22:33:03.947 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e3c6da949962238-00631bbf1f-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.948 DEBUG:boto:path=/jkcgbj-12/%3A 2022-09-09T22:33:03.949 DEBUG:boto:auth_path=/jkcgbj-12/%3A 2022-09-09T22:33:03.949 DEBUG:boto:path=/jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:03.950 DEBUG:boto:auth_path=/jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:03.950 DEBUG:boto:Method: HEAD 2022-09-09T22:33:03.951 DEBUG:boto:Path: /jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:03.951 DEBUG:boto:Data: 2022-09-09T22:33:03.952 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.952 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:03.953 DEBUG:boto:Port: 8001 2022-09-09T22:33:03.955 DEBUG:boto:Params: {} 2022-09-09T22:33:03.956 DEBUG:boto:Token: None 2022-09-09T22:33:03.956 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:03.957 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:xiyHOlZEjsbzufJFf+tq8+rJkuU= 2022-09-09T22:33:03.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:xiyHOlZEjsbzufJFf+tq8+rJkuU=', 'Content-Length': '0'} 2022-09-09T22:33:03.961 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f90d5c0153f52e88-00631bbf1f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:03.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-09T22:33:03.963 DEBUG:boto:path=/jkcgbj-12/%3A 2022-09-09T22:33:03.963 DEBUG:boto:auth_path=/jkcgbj-12/%3A 2022-09-09T22:33:03.964 DEBUG:boto:path=/jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:03.965 DEBUG:boto:auth_path=/jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:03.965 DEBUG:boto:Method: GET 2022-09-09T22:33:03.966 DEBUG:boto:Path: /jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:03.967 DEBUG:boto:Data: 2022-09-09T22:33:03.968 DEBUG:boto:Headers: {} 2022-09-09T22:33:03.969 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:03.969 DEBUG:boto:Port: 8000 2022-09-09T22:33:03.970 DEBUG:boto:Params: {} 2022-09-09T22:33:03.971 DEBUG:boto:Token: None 2022-09-09T22:33:03.971 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:03 GMT /jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:03.973 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ZeSPTjxIz6QVtyYVbleEjLzysus= 2022-09-09T22:33:03.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ZeSPTjxIz6QVtyYVbleEjLzysus=', 'Content-Length': '0'} 2022-09-09T22:33:03.976 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096edbd214883368a-00631bbf1f-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.018 DEBUG:boto:path=/jkcgbj-12/%3A 2022-09-09T22:33:04.020 DEBUG:boto:auth_path=/jkcgbj-12/%3A 2022-09-09T22:33:04.021 DEBUG:boto:path=/jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:04.023 DEBUG:boto:auth_path=/jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:04.024 DEBUG:boto:Method: GET 2022-09-09T22:33:04.025 DEBUG:boto:Path: /jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:04.027 DEBUG:boto:Data: 2022-09-09T22:33:04.028 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.029 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:04.030 DEBUG:boto:Port: 8001 2022-09-09T22:33:04.031 DEBUG:boto:Params: {} 2022-09-09T22:33:04.032 DEBUG:boto:Token: None 2022-09-09T22:33:04.034 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:04.035 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ZKyiKEBDKHs3tkPSh+jpPI2oO9I= 2022-09-09T22:33:04.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ZKyiKEBDKHs3tkPSh+jpPI2oO9I=', 'Content-Length': '0'} 2022-09-09T22:33:04.044 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a15da4c09062d3da-00631bbf20-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.045 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:04.046 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:04.046 DEBUG:boto:path=/jkcgbj-12/?prefix=%3A 2022-09-09T22:33:04.047 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=%3A 2022-09-09T22:33:04.047 DEBUG:boto:Method: GET 2022-09-09T22:33:04.048 DEBUG:boto:Path: /jkcgbj-12/?prefix=%3A 2022-09-09T22:33:04.048 DEBUG:boto:Data: 2022-09-09T22:33:04.049 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.049 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:04.050 DEBUG:boto:Port: 8000 2022-09-09T22:33:04.051 DEBUG:boto:Params: {} 2022-09-09T22:33:04.051 DEBUG:boto:Token: None 2022-09-09T22:33:04.052 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/ 2022-09-09T22:33:04.052 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bODFBiLf78TRP3I2e5hrcUtrFdA= 2022-09-09T22:33:04.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bODFBiLf78TRP3I2e5hrcUtrFdA=', 'Content-Length': '0'} 2022-09-09T22:33:04.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007da2bc2117514f4c-00631bbf20-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.060 DEBUG:boto:b'jkcgbj-12:1000false:2022-09-09T22:30:44.766Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:04.061 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:04.061 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:04.062 DEBUG:boto:path=/jkcgbj-12/?prefix=%3A 2022-09-09T22:33:04.063 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=%3A 2022-09-09T22:33:04.064 DEBUG:boto:Method: GET 2022-09-09T22:33:04.065 DEBUG:boto:Path: /jkcgbj-12/?prefix=%3A 2022-09-09T22:33:04.066 DEBUG:boto:Data: 2022-09-09T22:33:04.067 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.069 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:04.070 DEBUG:boto:Port: 8001 2022-09-09T22:33:04.071 DEBUG:boto:Params: {} 2022-09-09T22:33:04.072 DEBUG:boto:Token: None 2022-09-09T22:33:04.073 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/ 2022-09-09T22:33:04.074 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bODFBiLf78TRP3I2e5hrcUtrFdA= 2022-09-09T22:33:04.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bODFBiLf78TRP3I2e5hrcUtrFdA=', 'Content-Length': '0'} 2022-09-09T22:33:04.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000412cefb5d5a87469-00631bbf20-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.083 DEBUG:boto:b'jkcgbj-12:1000false:2022-09-09T22:30:44.766Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:04.083 DEBUG:boto:path=/jkcgbj-12/%3A 2022-09-09T22:33:04.084 DEBUG:boto:auth_path=/jkcgbj-12/%3A 2022-09-09T22:33:04.085 DEBUG:boto:Method: HEAD 2022-09-09T22:33:04.085 DEBUG:boto:Path: /jkcgbj-12/%3A 2022-09-09T22:33:04.086 DEBUG:boto:Data: 2022-09-09T22:33:04.086 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.087 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:04.088 DEBUG:boto:Port: 8000 2022-09-09T22:33:04.088 DEBUG:boto:Params: {} 2022-09-09T22:33:04.089 DEBUG:boto:Token: None 2022-09-09T22:33:04.091 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/%3A 2022-09-09T22:33:04.091 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CUHV1G78V4ZPqrUVCRa7L/kjJWQ= 2022-09-09T22:33:04.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CUHV1G78V4ZPqrUVCRa7L/kjJWQ=', 'Content-Length': '0'} 2022-09-09T22:33:04.095 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f986932cfe3a6417-00631bbf20-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.096 DEBUG:boto:path=/jkcgbj-12/%3A 2022-09-09T22:33:04.097 DEBUG:boto:auth_path=/jkcgbj-12/%3A 2022-09-09T22:33:04.097 DEBUG:boto:Method: HEAD 2022-09-09T22:33:04.098 DEBUG:boto:Path: /jkcgbj-12/%3A 2022-09-09T22:33:04.098 DEBUG:boto:Data: 2022-09-09T22:33:04.099 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.099 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:04.100 DEBUG:boto:Port: 8001 2022-09-09T22:33:04.100 DEBUG:boto:Params: {} 2022-09-09T22:33:04.102 DEBUG:boto:Token: None 2022-09-09T22:33:04.102 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/%3A 2022-09-09T22:33:04.103 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CUHV1G78V4ZPqrUVCRa7L/kjJWQ= 2022-09-09T22:33:04.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CUHV1G78V4ZPqrUVCRa7L/kjJWQ=', 'Content-Length': '0'} 2022-09-09T22:33:04.107 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c4a4939f8f20a4cc-00631bbf20-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-09T22:33:04.108 DEBUG:boto:path=/jkcgbj-12/%3A 2022-09-09T22:33:04.109 DEBUG:boto:auth_path=/jkcgbj-12/%3A 2022-09-09T22:33:04.109 DEBUG:boto:Method: GET 2022-09-09T22:33:04.110 DEBUG:boto:Path: /jkcgbj-12/%3A 2022-09-09T22:33:04.110 DEBUG:boto:Data: 2022-09-09T22:33:04.111 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.111 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:04.112 DEBUG:boto:Port: 8000 2022-09-09T22:33:04.112 DEBUG:boto:Params: {} 2022-09-09T22:33:04.113 DEBUG:boto:Token: None 2022-09-09T22:33:04.113 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/%3A 2022-09-09T22:33:04.114 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OteGFruwq/VURX56LvXL63R024M= 2022-09-09T22:33:04.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OteGFruwq/VURX56LvXL63R024M=', 'Content-Length': '0'} 2022-09-09T22:33:04.118 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d5e047709766a3e3-00631bbf20-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.162 DEBUG:boto:path=/jkcgbj-12/%3A 2022-09-09T22:33:04.163 DEBUG:boto:auth_path=/jkcgbj-12/%3A 2022-09-09T22:33:04.164 DEBUG:boto:Method: GET 2022-09-09T22:33:04.164 DEBUG:boto:Path: /jkcgbj-12/%3A 2022-09-09T22:33:04.165 DEBUG:boto:Data: 2022-09-09T22:33:04.165 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.166 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:04.167 DEBUG:boto:Port: 8001 2022-09-09T22:33:04.167 DEBUG:boto:Params: {} 2022-09-09T22:33:04.168 DEBUG:boto:Token: None 2022-09-09T22:33:04.169 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/%3A 2022-09-09T22:33:04.169 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OteGFruwq/VURX56LvXL63R024M= 2022-09-09T22:33:04.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OteGFruwq/VURX56LvXL63R024M=', 'Content-Length': '0'} 2022-09-09T22:33:04.173 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000779f761dad8b419e-00631bbf20-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.174 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:04.175 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:04.175 DEBUG:boto:path=/jkcgbj-12/?prefix=%3A 2022-09-09T22:33:04.176 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=%3A 2022-09-09T22:33:04.177 DEBUG:boto:Method: GET 2022-09-09T22:33:04.177 DEBUG:boto:Path: /jkcgbj-12/?prefix=%3A 2022-09-09T22:33:04.178 DEBUG:boto:Data: 2022-09-09T22:33:04.178 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.179 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:04.180 DEBUG:boto:Port: 8000 2022-09-09T22:33:04.180 DEBUG:boto:Params: {} 2022-09-09T22:33:04.181 DEBUG:boto:Token: None 2022-09-09T22:33:04.181 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/ 2022-09-09T22:33:04.182 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bODFBiLf78TRP3I2e5hrcUtrFdA= 2022-09-09T22:33:04.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bODFBiLf78TRP3I2e5hrcUtrFdA=', 'Content-Length': '0'} 2022-09-09T22:33:04.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c01bc6703e9601ea-00631bbf20-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.190 DEBUG:boto:b'jkcgbj-12:1000false:2022-09-09T22:30:44.766Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:04.191 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:04.191 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:04.192 DEBUG:boto:path=/jkcgbj-12/?prefix=%3A 2022-09-09T22:33:04.192 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=%3A 2022-09-09T22:33:04.193 DEBUG:boto:Method: GET 2022-09-09T22:33:04.194 DEBUG:boto:Path: /jkcgbj-12/?prefix=%3A 2022-09-09T22:33:04.195 DEBUG:boto:Data: 2022-09-09T22:33:04.195 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.196 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:04.196 DEBUG:boto:Port: 8001 2022-09-09T22:33:04.197 DEBUG:boto:Params: {} 2022-09-09T22:33:04.198 DEBUG:boto:Token: None 2022-09-09T22:33:04.198 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/ 2022-09-09T22:33:04.199 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bODFBiLf78TRP3I2e5hrcUtrFdA= 2022-09-09T22:33:04.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bODFBiLf78TRP3I2e5hrcUtrFdA=', 'Content-Length': '0'} 2022-09-09T22:33:04.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b47c9a2def078d0-00631bbf20-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.207 DEBUG:boto:b'jkcgbj-12:1000false:2022-09-09T22:30:44.766Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:04.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-09T22:33:04.209 DEBUG:boto:path=/jkcgbj-12/_myobj 2022-09-09T22:33:04.210 DEBUG:boto:auth_path=/jkcgbj-12/_myobj 2022-09-09T22:33:04.210 DEBUG:boto:path=/jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:04.211 DEBUG:boto:auth_path=/jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:04.212 DEBUG:boto:Method: GET 2022-09-09T22:33:04.213 DEBUG:boto:Path: /jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:04.213 DEBUG:boto:Data: 2022-09-09T22:33:04.214 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.215 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:04.215 DEBUG:boto:Port: 8000 2022-09-09T22:33:04.216 DEBUG:boto:Params: {} 2022-09-09T22:33:04.217 DEBUG:boto:Token: None 2022-09-09T22:33:04.217 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:04.218 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ElpGeRKI5OhCV3K0jof6vxyJ3SE= 2022-09-09T22:33:04.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ElpGeRKI5OhCV3K0jof6vxyJ3SE=', 'Content-Length': '0'} 2022-09-09T22:33:04.222 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047440cd4ec3e0444-00631bbf20-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.223 DEBUG:boto:path=/jkcgbj-12/_myobj 2022-09-09T22:33:04.224 DEBUG:boto:auth_path=/jkcgbj-12/_myobj 2022-09-09T22:33:04.225 DEBUG:boto:path=/jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:04.226 DEBUG:boto:auth_path=/jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:04.227 DEBUG:boto:Method: GET 2022-09-09T22:33:04.228 DEBUG:boto:Path: /jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:04.229 DEBUG:boto:Data: 2022-09-09T22:33:04.230 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.231 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:04.231 DEBUG:boto:Port: 8001 2022-09-09T22:33:04.232 DEBUG:boto:Params: {} 2022-09-09T22:33:04.233 DEBUG:boto:Token: None 2022-09-09T22:33:04.234 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:04.235 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ElpGeRKI5OhCV3K0jof6vxyJ3SE= 2022-09-09T22:33:04.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ElpGeRKI5OhCV3K0jof6vxyJ3SE=', 'Content-Length': '0'} 2022-09-09T22:33:04.239 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000953ffd57e325a812-00631bbf20-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.282 DEBUG:boto:path=/jkcgbj-12/_myobj 2022-09-09T22:33:04.283 DEBUG:boto:auth_path=/jkcgbj-12/_myobj 2022-09-09T22:33:04.284 DEBUG:boto:path=/jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:04.285 DEBUG:boto:auth_path=/jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:04.286 DEBUG:boto:Method: HEAD 2022-09-09T22:33:04.287 DEBUG:boto:Path: /jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:04.288 DEBUG:boto:Data: 2022-09-09T22:33:04.289 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.289 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:04.290 DEBUG:boto:Port: 8000 2022-09-09T22:33:04.291 DEBUG:boto:Params: {} 2022-09-09T22:33:04.292 DEBUG:boto:Token: None 2022-09-09T22:33:04.293 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:04.294 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Hdc9VbUcBiYkONPRDzidD4t1dJ8= 2022-09-09T22:33:04.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Hdc9VbUcBiYkONPRDzidD4t1dJ8=', 'Content-Length': '0'} 2022-09-09T22:33:04.298 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050550b1e3d46e9aa-00631bbf20-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.298 DEBUG:boto:path=/jkcgbj-12/_myobj 2022-09-09T22:33:04.299 DEBUG:boto:auth_path=/jkcgbj-12/_myobj 2022-09-09T22:33:04.300 DEBUG:boto:path=/jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:04.300 DEBUG:boto:auth_path=/jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:04.301 DEBUG:boto:Method: HEAD 2022-09-09T22:33:04.301 DEBUG:boto:Path: /jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:04.302 DEBUG:boto:Data: 2022-09-09T22:33:04.302 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.303 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:04.303 DEBUG:boto:Port: 8001 2022-09-09T22:33:04.304 DEBUG:boto:Params: {} 2022-09-09T22:33:04.305 DEBUG:boto:Token: None 2022-09-09T22:33:04.306 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:04.307 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Hdc9VbUcBiYkONPRDzidD4t1dJ8= 2022-09-09T22:33:04.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Hdc9VbUcBiYkONPRDzidD4t1dJ8=', 'Content-Length': '0'} 2022-09-09T22:33:04.311 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005fa40f1a192855dc-00631bbf20-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-09T22:33:04.312 DEBUG:boto:path=/jkcgbj-12/_myobj 2022-09-09T22:33:04.313 DEBUG:boto:auth_path=/jkcgbj-12/_myobj 2022-09-09T22:33:04.314 DEBUG:boto:path=/jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:04.314 DEBUG:boto:auth_path=/jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:04.315 DEBUG:boto:Method: GET 2022-09-09T22:33:04.315 DEBUG:boto:Path: /jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:04.316 DEBUG:boto:Data: 2022-09-09T22:33:04.317 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.318 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:04.319 DEBUG:boto:Port: 8000 2022-09-09T22:33:04.320 DEBUG:boto:Params: {} 2022-09-09T22:33:04.321 DEBUG:boto:Token: None 2022-09-09T22:33:04.322 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:04.323 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ElpGeRKI5OhCV3K0jof6vxyJ3SE= 2022-09-09T22:33:04.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ElpGeRKI5OhCV3K0jof6vxyJ3SE=', 'Content-Length': '0'} 2022-09-09T22:33:04.328 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006be3d942480b49fc-00631bbf20-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.329 DEBUG:boto:path=/jkcgbj-12/_myobj 2022-09-09T22:33:04.329 DEBUG:boto:auth_path=/jkcgbj-12/_myobj 2022-09-09T22:33:04.330 DEBUG:boto:path=/jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:04.331 DEBUG:boto:auth_path=/jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:04.332 DEBUG:boto:Method: GET 2022-09-09T22:33:04.333 DEBUG:boto:Path: /jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:04.334 DEBUG:boto:Data: 2022-09-09T22:33:04.335 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.336 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:04.337 DEBUG:boto:Port: 8001 2022-09-09T22:33:04.338 DEBUG:boto:Params: {} 2022-09-09T22:33:04.339 DEBUG:boto:Token: None 2022-09-09T22:33:04.340 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:04.341 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ElpGeRKI5OhCV3K0jof6vxyJ3SE= 2022-09-09T22:33:04.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ElpGeRKI5OhCV3K0jof6vxyJ3SE=', 'Content-Length': '0'} 2022-09-09T22:33:04.346 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b193dd137ee1a579-00631bbf20-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.348 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:04.348 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:04.349 DEBUG:boto:path=/jkcgbj-12/?prefix=_myobj 2022-09-09T22:33:04.349 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=_myobj 2022-09-09T22:33:04.350 DEBUG:boto:Method: GET 2022-09-09T22:33:04.351 DEBUG:boto:Path: /jkcgbj-12/?prefix=_myobj 2022-09-09T22:33:04.351 DEBUG:boto:Data: 2022-09-09T22:33:04.352 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.353 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:04.353 DEBUG:boto:Port: 8000 2022-09-09T22:33:04.354 DEBUG:boto:Params: {} 2022-09-09T22:33:04.355 DEBUG:boto:Token: None 2022-09-09T22:33:04.355 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/ 2022-09-09T22:33:04.356 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bODFBiLf78TRP3I2e5hrcUtrFdA= 2022-09-09T22:33:04.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bODFBiLf78TRP3I2e5hrcUtrFdA=', 'Content-Length': '0'} 2022-09-09T22:33:04.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000297c57799eb52378-00631bbf20-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.406 DEBUG:boto:b'jkcgbj-12_myobj1000false_myobj2022-09-09T22:30:44.684Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:04.407 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:04.408 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:04.408 DEBUG:boto:path=/jkcgbj-12/?prefix=_myobj 2022-09-09T22:33:04.409 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=_myobj 2022-09-09T22:33:04.409 DEBUG:boto:Method: GET 2022-09-09T22:33:04.410 DEBUG:boto:Path: /jkcgbj-12/?prefix=_myobj 2022-09-09T22:33:04.410 DEBUG:boto:Data: 2022-09-09T22:33:04.411 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.411 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:04.412 DEBUG:boto:Port: 8001 2022-09-09T22:33:04.412 DEBUG:boto:Params: {} 2022-09-09T22:33:04.413 DEBUG:boto:Token: None 2022-09-09T22:33:04.414 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/ 2022-09-09T22:33:04.414 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bODFBiLf78TRP3I2e5hrcUtrFdA= 2022-09-09T22:33:04.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bODFBiLf78TRP3I2e5hrcUtrFdA=', 'Content-Length': '0'} 2022-09-09T22:33:04.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002845a82f4933573-00631bbf20-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.421 DEBUG:boto:b'jkcgbj-12_myobj1000false_myobj2022-09-09T22:30:44.684Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:04.422 DEBUG:boto:path=/jkcgbj-12/_myobj 2022-09-09T22:33:04.423 DEBUG:boto:auth_path=/jkcgbj-12/_myobj 2022-09-09T22:33:04.423 DEBUG:boto:Method: HEAD 2022-09-09T22:33:04.424 DEBUG:boto:Path: /jkcgbj-12/_myobj 2022-09-09T22:33:04.424 DEBUG:boto:Data: 2022-09-09T22:33:04.425 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.425 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:04.426 DEBUG:boto:Port: 8000 2022-09-09T22:33:04.426 DEBUG:boto:Params: {} 2022-09-09T22:33:04.427 DEBUG:boto:Token: None 2022-09-09T22:33:04.427 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/_myobj 2022-09-09T22:33:04.428 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dfuCsMVdm6mT3UtRJ0C676s99EE= 2022-09-09T22:33:04.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dfuCsMVdm6mT3UtRJ0C676s99EE=', 'Content-Length': '0'} 2022-09-09T22:33:04.431 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca51112bf73bdabb-00631bbf20-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.432 DEBUG:boto:path=/jkcgbj-12/_myobj 2022-09-09T22:33:04.433 DEBUG:boto:auth_path=/jkcgbj-12/_myobj 2022-09-09T22:33:04.433 DEBUG:boto:Method: HEAD 2022-09-09T22:33:04.434 DEBUG:boto:Path: /jkcgbj-12/_myobj 2022-09-09T22:33:04.434 DEBUG:boto:Data: 2022-09-09T22:33:04.435 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.435 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:04.436 DEBUG:boto:Port: 8001 2022-09-09T22:33:04.436 DEBUG:boto:Params: {} 2022-09-09T22:33:04.437 DEBUG:boto:Token: None 2022-09-09T22:33:04.437 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/_myobj 2022-09-09T22:33:04.438 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dfuCsMVdm6mT3UtRJ0C676s99EE= 2022-09-09T22:33:04.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dfuCsMVdm6mT3UtRJ0C676s99EE=', 'Content-Length': '0'} 2022-09-09T22:33:04.442 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005955e0ecc4edadbd-00631bbf20-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-09T22:33:04.444 DEBUG:boto:path=/jkcgbj-12/_myobj 2022-09-09T22:33:04.444 DEBUG:boto:auth_path=/jkcgbj-12/_myobj 2022-09-09T22:33:04.445 DEBUG:boto:Method: GET 2022-09-09T22:33:04.445 DEBUG:boto:Path: /jkcgbj-12/_myobj 2022-09-09T22:33:04.446 DEBUG:boto:Data: 2022-09-09T22:33:04.446 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.447 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:04.447 DEBUG:boto:Port: 8000 2022-09-09T22:33:04.448 DEBUG:boto:Params: {} 2022-09-09T22:33:04.448 DEBUG:boto:Token: None 2022-09-09T22:33:04.449 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/_myobj 2022-09-09T22:33:04.450 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:XH3P/q0Qu0/4lReNjzLL4N1aiF4= 2022-09-09T22:33:04.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:XH3P/q0Qu0/4lReNjzLL4N1aiF4=', 'Content-Length': '0'} 2022-09-09T22:33:04.453 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000042550c16b1c64ca3-00631bbf20-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.455 DEBUG:boto:path=/jkcgbj-12/_myobj 2022-09-09T22:33:04.455 DEBUG:boto:auth_path=/jkcgbj-12/_myobj 2022-09-09T22:33:04.456 DEBUG:boto:Method: GET 2022-09-09T22:33:04.456 DEBUG:boto:Path: /jkcgbj-12/_myobj 2022-09-09T22:33:04.457 DEBUG:boto:Data: 2022-09-09T22:33:04.458 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.459 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:04.460 DEBUG:boto:Port: 8001 2022-09-09T22:33:04.461 DEBUG:boto:Params: {} 2022-09-09T22:33:04.462 DEBUG:boto:Token: None 2022-09-09T22:33:04.462 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/_myobj 2022-09-09T22:33:04.463 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:XH3P/q0Qu0/4lReNjzLL4N1aiF4= 2022-09-09T22:33:04.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:XH3P/q0Qu0/4lReNjzLL4N1aiF4=', 'Content-Length': '0'} 2022-09-09T22:33:04.469 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008266048ee5d48dfc-00631bbf20-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.514 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:04.515 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:04.516 DEBUG:boto:path=/jkcgbj-12/?prefix=_myobj 2022-09-09T22:33:04.517 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=_myobj 2022-09-09T22:33:04.518 DEBUG:boto:Method: GET 2022-09-09T22:33:04.518 DEBUG:boto:Path: /jkcgbj-12/?prefix=_myobj 2022-09-09T22:33:04.519 DEBUG:boto:Data: 2022-09-09T22:33:04.520 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.520 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:04.521 DEBUG:boto:Port: 8000 2022-09-09T22:33:04.521 DEBUG:boto:Params: {} 2022-09-09T22:33:04.522 DEBUG:boto:Token: None 2022-09-09T22:33:04.523 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/ 2022-09-09T22:33:04.524 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bODFBiLf78TRP3I2e5hrcUtrFdA= 2022-09-09T22:33:04.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bODFBiLf78TRP3I2e5hrcUtrFdA=', 'Content-Length': '0'} 2022-09-09T22:33:04.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004479265dc9a87e0-00631bbf20-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.531 DEBUG:boto:b'jkcgbj-12_myobj1000false_myobj2022-09-09T22:30:44.684Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:04.531 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:04.532 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:04.533 DEBUG:boto:path=/jkcgbj-12/?prefix=_myobj 2022-09-09T22:33:04.533 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=_myobj 2022-09-09T22:33:04.534 DEBUG:boto:Method: GET 2022-09-09T22:33:04.534 DEBUG:boto:Path: /jkcgbj-12/?prefix=_myobj 2022-09-09T22:33:04.535 DEBUG:boto:Data: 2022-09-09T22:33:04.535 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.536 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:04.537 DEBUG:boto:Port: 8001 2022-09-09T22:33:04.537 DEBUG:boto:Params: {} 2022-09-09T22:33:04.538 DEBUG:boto:Token: None 2022-09-09T22:33:04.539 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/ 2022-09-09T22:33:04.540 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bODFBiLf78TRP3I2e5hrcUtrFdA= 2022-09-09T22:33:04.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bODFBiLf78TRP3I2e5hrcUtrFdA=', 'Content-Length': '0'} 2022-09-09T22:33:04.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea30bea5a3a98eb1-00631bbf20-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.547 DEBUG:boto:b'jkcgbj-12_myobj1000false_myobj2022-09-09T22:30:44.684Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:04.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:33:04.549 DEBUG:boto:path=/jkcgbj-12/myobj 2022-09-09T22:33:04.550 DEBUG:boto:auth_path=/jkcgbj-12/myobj 2022-09-09T22:33:04.550 DEBUG:boto:path=/jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:04.551 DEBUG:boto:auth_path=/jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:04.551 DEBUG:boto:Method: GET 2022-09-09T22:33:04.552 DEBUG:boto:Path: /jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:04.553 DEBUG:boto:Data: 2022-09-09T22:33:04.553 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.554 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:04.555 DEBUG:boto:Port: 8000 2022-09-09T22:33:04.555 DEBUG:boto:Params: {} 2022-09-09T22:33:04.556 DEBUG:boto:Token: None 2022-09-09T22:33:04.556 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:04.557 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:it+Qro7fQZk7DrpNlrJ3UGwqs+Q= 2022-09-09T22:33:04.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:it+Qro7fQZk7DrpNlrJ3UGwqs+Q=', 'Content-Length': '0'} 2022-09-09T22:33:04.561 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9307dcaf8669c89-00631bbf20-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.602 DEBUG:boto:path=/jkcgbj-12/myobj 2022-09-09T22:33:04.605 DEBUG:boto:auth_path=/jkcgbj-12/myobj 2022-09-09T22:33:04.606 DEBUG:boto:path=/jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:04.607 DEBUG:boto:auth_path=/jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:04.607 DEBUG:boto:Method: GET 2022-09-09T22:33:04.608 DEBUG:boto:Path: /jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:04.609 DEBUG:boto:Data: 2022-09-09T22:33:04.610 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.611 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:04.612 DEBUG:boto:Port: 8001 2022-09-09T22:33:04.613 DEBUG:boto:Params: {} 2022-09-09T22:33:04.614 DEBUG:boto:Token: None 2022-09-09T22:33:04.616 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:04.617 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:it+Qro7fQZk7DrpNlrJ3UGwqs+Q= 2022-09-09T22:33:04.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:it+Qro7fQZk7DrpNlrJ3UGwqs+Q=', 'Content-Length': '0'} 2022-09-09T22:33:04.622 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f6996b39ca5147f-00631bbf20-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.623 DEBUG:boto:path=/jkcgbj-12/myobj 2022-09-09T22:33:04.624 DEBUG:boto:auth_path=/jkcgbj-12/myobj 2022-09-09T22:33:04.625 DEBUG:boto:path=/jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:04.625 DEBUG:boto:auth_path=/jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:04.626 DEBUG:boto:Method: HEAD 2022-09-09T22:33:04.626 DEBUG:boto:Path: /jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:04.627 DEBUG:boto:Data: 2022-09-09T22:33:04.628 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.628 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:04.629 DEBUG:boto:Port: 8000 2022-09-09T22:33:04.629 DEBUG:boto:Params: {} 2022-09-09T22:33:04.630 DEBUG:boto:Token: None 2022-09-09T22:33:04.631 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:04.631 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:im2/aAKMt1kGfdqZ+T0rkoffSgo= 2022-09-09T22:33:04.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:im2/aAKMt1kGfdqZ+T0rkoffSgo=', 'Content-Length': '0'} 2022-09-09T22:33:04.635 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d373859abcc65930-00631bbf20-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.636 DEBUG:boto:path=/jkcgbj-12/myobj 2022-09-09T22:33:04.636 DEBUG:boto:auth_path=/jkcgbj-12/myobj 2022-09-09T22:33:04.637 DEBUG:boto:path=/jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:04.638 DEBUG:boto:auth_path=/jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:04.638 DEBUG:boto:Method: HEAD 2022-09-09T22:33:04.639 DEBUG:boto:Path: /jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:04.639 DEBUG:boto:Data: 2022-09-09T22:33:04.640 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.640 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:04.641 DEBUG:boto:Port: 8001 2022-09-09T22:33:04.641 DEBUG:boto:Params: {} 2022-09-09T22:33:04.642 DEBUG:boto:Token: None 2022-09-09T22:33:04.643 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:04.644 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:im2/aAKMt1kGfdqZ+T0rkoffSgo= 2022-09-09T22:33:04.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:im2/aAKMt1kGfdqZ+T0rkoffSgo=', 'Content-Length': '0'} 2022-09-09T22:33:04.647 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f6f50678c53de8d-00631bbf20-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:33:04.649 DEBUG:boto:path=/jkcgbj-12/myobj 2022-09-09T22:33:04.651 DEBUG:boto:auth_path=/jkcgbj-12/myobj 2022-09-09T22:33:04.652 DEBUG:boto:path=/jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:04.653 DEBUG:boto:auth_path=/jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:04.654 DEBUG:boto:Method: GET 2022-09-09T22:33:04.654 DEBUG:boto:Path: /jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:04.655 DEBUG:boto:Data: 2022-09-09T22:33:04.656 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.657 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:04.657 DEBUG:boto:Port: 8000 2022-09-09T22:33:04.658 DEBUG:boto:Params: {} 2022-09-09T22:33:04.659 DEBUG:boto:Token: None 2022-09-09T22:33:04.660 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:04.660 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:it+Qro7fQZk7DrpNlrJ3UGwqs+Q= 2022-09-09T22:33:04.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:it+Qro7fQZk7DrpNlrJ3UGwqs+Q=', 'Content-Length': '0'} 2022-09-09T22:33:04.664 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aea66dee265f5121-00631bbf20-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.665 DEBUG:boto:path=/jkcgbj-12/myobj 2022-09-09T22:33:04.666 DEBUG:boto:auth_path=/jkcgbj-12/myobj 2022-09-09T22:33:04.666 DEBUG:boto:path=/jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:04.667 DEBUG:boto:auth_path=/jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:04.667 DEBUG:boto:Method: GET 2022-09-09T22:33:04.668 DEBUG:boto:Path: /jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:04.668 DEBUG:boto:Data: 2022-09-09T22:33:04.669 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.669 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:04.670 DEBUG:boto:Port: 8001 2022-09-09T22:33:04.671 DEBUG:boto:Params: {} 2022-09-09T22:33:04.671 DEBUG:boto:Token: None 2022-09-09T22:33:04.672 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:04.672 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:it+Qro7fQZk7DrpNlrJ3UGwqs+Q= 2022-09-09T22:33:04.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:it+Qro7fQZk7DrpNlrJ3UGwqs+Q=', 'Content-Length': '0'} 2022-09-09T22:33:04.677 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0977ca390947416-00631bbf20-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.677 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:04.678 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:04.679 DEBUG:boto:path=/jkcgbj-12/?prefix=myobj 2022-09-09T22:33:04.679 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=myobj 2022-09-09T22:33:04.680 DEBUG:boto:Method: GET 2022-09-09T22:33:04.680 DEBUG:boto:Path: /jkcgbj-12/?prefix=myobj 2022-09-09T22:33:04.681 DEBUG:boto:Data: 2022-09-09T22:33:04.681 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.682 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:04.682 DEBUG:boto:Port: 8000 2022-09-09T22:33:04.683 DEBUG:boto:Params: {} 2022-09-09T22:33:04.684 DEBUG:boto:Token: None 2022-09-09T22:33:04.684 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/ 2022-09-09T22:33:04.685 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bODFBiLf78TRP3I2e5hrcUtrFdA= 2022-09-09T22:33:04.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bODFBiLf78TRP3I2e5hrcUtrFdA=', 'Content-Length': '0'} 2022-09-09T22:33:04.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7f40dc7af9c6fc1-00631bbf20-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.692 DEBUG:boto:b'jkcgbj-12myobj1000falsemyobj2022-09-09T22:30:44.625Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:04.693 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:04.694 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:04.694 DEBUG:boto:path=/jkcgbj-12/?prefix=myobj 2022-09-09T22:33:04.695 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=myobj 2022-09-09T22:33:04.695 DEBUG:boto:Method: GET 2022-09-09T22:33:04.696 DEBUG:boto:Path: /jkcgbj-12/?prefix=myobj 2022-09-09T22:33:04.696 DEBUG:boto:Data: 2022-09-09T22:33:04.697 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.697 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:04.698 DEBUG:boto:Port: 8001 2022-09-09T22:33:04.698 DEBUG:boto:Params: {} 2022-09-09T22:33:04.699 DEBUG:boto:Token: None 2022-09-09T22:33:04.700 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/ 2022-09-09T22:33:04.700 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bODFBiLf78TRP3I2e5hrcUtrFdA= 2022-09-09T22:33:04.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bODFBiLf78TRP3I2e5hrcUtrFdA=', 'Content-Length': '0'} 2022-09-09T22:33:04.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f91e7fcd9e3b2d37-00631bbf20-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.708 DEBUG:boto:b'jkcgbj-12myobj1000falsemyobj2022-09-09T22:30:44.625Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:04.710 DEBUG:boto:path=/jkcgbj-12/myobj 2022-09-09T22:33:04.711 DEBUG:boto:auth_path=/jkcgbj-12/myobj 2022-09-09T22:33:04.711 DEBUG:boto:Method: HEAD 2022-09-09T22:33:04.712 DEBUG:boto:Path: /jkcgbj-12/myobj 2022-09-09T22:33:04.713 DEBUG:boto:Data: 2022-09-09T22:33:04.714 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.715 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:04.716 DEBUG:boto:Port: 8000 2022-09-09T22:33:04.716 DEBUG:boto:Params: {} 2022-09-09T22:33:04.717 DEBUG:boto:Token: None 2022-09-09T22:33:04.719 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/myobj 2022-09-09T22:33:04.720 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:z5USE95uAOowBB2QmsNvGGx4m2A= 2022-09-09T22:33:04.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:z5USE95uAOowBB2QmsNvGGx4m2A=', 'Content-Length': '0'} 2022-09-09T22:33:04.724 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000244aa7f5b187d5fc-00631bbf20-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.725 DEBUG:boto:path=/jkcgbj-12/myobj 2022-09-09T22:33:04.726 DEBUG:boto:auth_path=/jkcgbj-12/myobj 2022-09-09T22:33:04.727 DEBUG:boto:Method: HEAD 2022-09-09T22:33:04.728 DEBUG:boto:Path: /jkcgbj-12/myobj 2022-09-09T22:33:04.729 DEBUG:boto:Data: 2022-09-09T22:33:04.729 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.730 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:04.730 DEBUG:boto:Port: 8001 2022-09-09T22:33:04.731 DEBUG:boto:Params: {} 2022-09-09T22:33:04.731 DEBUG:boto:Token: None 2022-09-09T22:33:04.732 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/myobj 2022-09-09T22:33:04.733 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:z5USE95uAOowBB2QmsNvGGx4m2A= 2022-09-09T22:33:04.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:z5USE95uAOowBB2QmsNvGGx4m2A=', 'Content-Length': '0'} 2022-09-09T22:33:04.737 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c6473ca944a55fb8-00631bbf20-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:33:04.739 DEBUG:boto:path=/jkcgbj-12/myobj 2022-09-09T22:33:04.739 DEBUG:boto:auth_path=/jkcgbj-12/myobj 2022-09-09T22:33:04.740 DEBUG:boto:Method: GET 2022-09-09T22:33:04.741 DEBUG:boto:Path: /jkcgbj-12/myobj 2022-09-09T22:33:04.742 DEBUG:boto:Data: 2022-09-09T22:33:04.742 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.743 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:04.744 DEBUG:boto:Port: 8000 2022-09-09T22:33:04.745 DEBUG:boto:Params: {} 2022-09-09T22:33:04.745 DEBUG:boto:Token: None 2022-09-09T22:33:04.746 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/myobj 2022-09-09T22:33:04.747 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Ene+D+b8MdtjoRkqG+t0ffGsioM= 2022-09-09T22:33:04.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Ene+D+b8MdtjoRkqG+t0ffGsioM=', 'Content-Length': '0'} 2022-09-09T22:33:04.751 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5a78b389eaedef0-00631bbf20-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.798 DEBUG:boto:path=/jkcgbj-12/myobj 2022-09-09T22:33:04.799 DEBUG:boto:auth_path=/jkcgbj-12/myobj 2022-09-09T22:33:04.800 DEBUG:boto:Method: GET 2022-09-09T22:33:04.801 DEBUG:boto:Path: /jkcgbj-12/myobj 2022-09-09T22:33:04.802 DEBUG:boto:Data: 2022-09-09T22:33:04.803 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.804 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:04.804 DEBUG:boto:Port: 8001 2022-09-09T22:33:04.805 DEBUG:boto:Params: {} 2022-09-09T22:33:04.807 DEBUG:boto:Token: None 2022-09-09T22:33:04.808 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/myobj 2022-09-09T22:33:04.809 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Ene+D+b8MdtjoRkqG+t0ffGsioM= 2022-09-09T22:33:04.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Ene+D+b8MdtjoRkqG+t0ffGsioM=', 'Content-Length': '0'} 2022-09-09T22:33:04.816 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b088a77a7536db2d-00631bbf20-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.819 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:04.819 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:04.820 DEBUG:boto:path=/jkcgbj-12/?prefix=myobj 2022-09-09T22:33:04.821 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=myobj 2022-09-09T22:33:04.822 DEBUG:boto:Method: GET 2022-09-09T22:33:04.823 DEBUG:boto:Path: /jkcgbj-12/?prefix=myobj 2022-09-09T22:33:04.824 DEBUG:boto:Data: 2022-09-09T22:33:04.827 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.828 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:04.828 DEBUG:boto:Port: 8000 2022-09-09T22:33:04.830 DEBUG:boto:Params: {} 2022-09-09T22:33:04.832 DEBUG:boto:Token: None 2022-09-09T22:33:04.833 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/ 2022-09-09T22:33:04.834 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bODFBiLf78TRP3I2e5hrcUtrFdA= 2022-09-09T22:33:04.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bODFBiLf78TRP3I2e5hrcUtrFdA=', 'Content-Length': '0'} 2022-09-09T22:33:04.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004abbc0e7adfd8cab-00631bbf20-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.844 DEBUG:boto:b'jkcgbj-12myobj1000falsemyobj2022-09-09T22:30:44.625Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:04.845 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:04.846 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:04.847 DEBUG:boto:path=/jkcgbj-12/?prefix=myobj 2022-09-09T22:33:04.848 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=myobj 2022-09-09T22:33:04.850 DEBUG:boto:Method: GET 2022-09-09T22:33:04.851 DEBUG:boto:Path: /jkcgbj-12/?prefix=myobj 2022-09-09T22:33:04.852 DEBUG:boto:Data: 2022-09-09T22:33:04.853 DEBUG:boto:Headers: {} 2022-09-09T22:33:04.854 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:04.855 DEBUG:boto:Port: 8001 2022-09-09T22:33:04.856 DEBUG:boto:Params: {} 2022-09-09T22:33:04.857 DEBUG:boto:Token: None 2022-09-09T22:33:04.858 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:04 GMT /jkcgbj-12/ 2022-09-09T22:33:04.859 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bODFBiLf78TRP3I2e5hrcUtrFdA= 2022-09-09T22:33:04.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bODFBiLf78TRP3I2e5hrcUtrFdA=', 'Content-Length': '0'} 2022-09-09T22:33:04.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046291160cbc4b5f4-00631bbf20-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:04.868 DEBUG:boto:b'jkcgbj-12myobj1000falsemyobj2022-09-09T22:30:44.625Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:04.871 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-12 zones={test-zone1, test-zone3} 2022-09-09T22:33:04.872 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:33:04.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:33:04.875 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:33:04.930 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:33:04.931 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:33:05.013 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:33:05.008+0000 7f65936c5dc0 1 bucket sync caught up with source: 2022-09-09T22:33:05.013 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000001.21.5, , 00000000001.15.5, , , , 00000000001.23.5, 00000000001.20.5, , ] 2022-09-09T22:33:05.014 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000001.21.5, , 00000000001.15.5, , , , 00000000001.23.5, 00000000001.20.5, , ] 2022-09-09T22:33:05.014 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:33:05.008+0000 7f65936c5dc0 0 bucket checkpoint complete 2022-09-09T22:33:05.041 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-12 zones={test-zone2, test-zone3} 2022-09-09T22:33:05.042 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:05.043 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:05.044 DEBUG:boto:Method: HEAD 2022-09-09T22:33:05.044 DEBUG:boto:Path: /jkcgbj-12/ 2022-09-09T22:33:05.045 DEBUG:boto:Data: 2022-09-09T22:33:05.045 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.046 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:05.046 DEBUG:boto:Port: 8000 2022-09-09T22:33:05.047 DEBUG:boto:Params: {} 2022-09-09T22:33:05.048 DEBUG:boto:Token: None 2022-09-09T22:33:05.049 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/ 2022-09-09T22:33:05.050 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ZDRTMDe5bNkMb0BI0u/HwFZzi1Q= 2022-09-09T22:33:05.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ZDRTMDe5bNkMb0BI0u/HwFZzi1Q=', 'Content-Length': '0'} 2022-09-09T22:33:05.055 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005bee6a90937465cf-00631bbf21-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.056 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:05.057 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:05.057 DEBUG:boto:Method: HEAD 2022-09-09T22:33:05.058 DEBUG:boto:Path: /jkcgbj-12/ 2022-09-09T22:33:05.059 DEBUG:boto:Data: 2022-09-09T22:33:05.060 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.060 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:05.061 DEBUG:boto:Port: 8001 2022-09-09T22:33:05.061 DEBUG:boto:Params: {} 2022-09-09T22:33:05.062 DEBUG:boto:Token: None 2022-09-09T22:33:05.063 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/ 2022-09-09T22:33:05.063 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ZDRTMDe5bNkMb0BI0u/HwFZzi1Q= 2022-09-09T22:33:05.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ZDRTMDe5bNkMb0BI0u/HwFZzi1Q=', 'Content-Length': '0'} 2022-09-09T22:33:05.069 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005fc5989ecc38e3cb-00631bbf21-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.070 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:33:05.071 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:05.071 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:05.072 DEBUG:boto:path=/jkcgbj-12/?versions&max-keys=999 2022-09-09T22:33:05.072 DEBUG:boto:auth_path=/jkcgbj-12/?versions&max-keys=999 2022-09-09T22:33:05.073 DEBUG:boto:Method: GET 2022-09-09T22:33:05.073 DEBUG:boto:Path: /jkcgbj-12/?versions&max-keys=999 2022-09-09T22:33:05.074 DEBUG:boto:Data: 2022-09-09T22:33:05.074 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.075 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:05.075 DEBUG:boto:Port: 8000 2022-09-09T22:33:05.076 DEBUG:boto:Params: {} 2022-09-09T22:33:05.077 DEBUG:boto:Token: None 2022-09-09T22:33:05.078 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/?versions 2022-09-09T22:33:05.078 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Y2pWTdM4OP2uZAv25+xFO3QtXo0= 2022-09-09T22:33:05.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Y2pWTdM4OP2uZAv25+xFO3QtXo0=', 'Content-Length': '0'} 2022-09-09T22:33:05.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ed590e3f2e2583f-00631bbf21-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.086 DEBUG:boto:b'jkcgbj-12999false&nulltrue2022-09-09T22:30:44.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-09T22:30:44.766Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-09T22:30:44.684Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-09T22:30:44.625Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:05.087 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-09T22:33:05.088 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-09T22:33:05.089 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-09T22:33:05.089 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-09T22:33:05.090 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:33:05.090 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:05.091 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:05.092 DEBUG:boto:path=/jkcgbj-12/?versions&max-keys=999 2022-09-09T22:33:05.092 DEBUG:boto:auth_path=/jkcgbj-12/?versions&max-keys=999 2022-09-09T22:33:05.093 DEBUG:boto:Method: GET 2022-09-09T22:33:05.093 DEBUG:boto:Path: /jkcgbj-12/?versions&max-keys=999 2022-09-09T22:33:05.094 DEBUG:boto:Data: 2022-09-09T22:33:05.094 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.095 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:05.095 DEBUG:boto:Port: 8001 2022-09-09T22:33:05.096 DEBUG:boto:Params: {} 2022-09-09T22:33:05.096 DEBUG:boto:Token: None 2022-09-09T22:33:05.097 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/?versions 2022-09-09T22:33:05.098 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Y2pWTdM4OP2uZAv25+xFO3QtXo0= 2022-09-09T22:33:05.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Y2pWTdM4OP2uZAv25+xFO3QtXo0=', 'Content-Length': '0'} 2022-09-09T22:33:05.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000939b092c00848daa-00631bbf21-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.105 DEBUG:boto:b'jkcgbj-12999false&nulltrue2022-09-09T22:30:44.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-09T22:30:44.766Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-09T22:30:44.684Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-09T22:30:44.625Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:05.106 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-09T22:33:05.106 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-09T22:33:05.107 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-09T22:33:05.107 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-09T22:33:05.108 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:05.109 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:05.109 DEBUG:boto:path=/jkcgbj-12/?versions&max-keys=999 2022-09-09T22:33:05.110 DEBUG:boto:auth_path=/jkcgbj-12/?versions&max-keys=999 2022-09-09T22:33:05.111 DEBUG:boto:Method: GET 2022-09-09T22:33:05.111 DEBUG:boto:Path: /jkcgbj-12/?versions&max-keys=999 2022-09-09T22:33:05.112 DEBUG:boto:Data: 2022-09-09T22:33:05.112 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.113 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:05.113 DEBUG:boto:Port: 8000 2022-09-09T22:33:05.114 DEBUG:boto:Params: {} 2022-09-09T22:33:05.115 DEBUG:boto:Token: None 2022-09-09T22:33:05.115 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/?versions 2022-09-09T22:33:05.116 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Y2pWTdM4OP2uZAv25+xFO3QtXo0= 2022-09-09T22:33:05.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Y2pWTdM4OP2uZAv25+xFO3QtXo0=', 'Content-Length': '0'} 2022-09-09T22:33:05.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea2031aaa6bc5d12-00631bbf21-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.124 DEBUG:boto:b'jkcgbj-12999false&nulltrue2022-09-09T22:30:44.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-09T22:30:44.766Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-09T22:30:44.684Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-09T22:30:44.625Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:05.125 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:05.126 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:05.126 DEBUG:boto:path=/jkcgbj-12/?versions&max-keys=999 2022-09-09T22:33:05.127 DEBUG:boto:auth_path=/jkcgbj-12/?versions&max-keys=999 2022-09-09T22:33:05.128 DEBUG:boto:Method: GET 2022-09-09T22:33:05.128 DEBUG:boto:Path: /jkcgbj-12/?versions&max-keys=999 2022-09-09T22:33:05.129 DEBUG:boto:Data: 2022-09-09T22:33:05.130 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.130 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:05.131 DEBUG:boto:Port: 8001 2022-09-09T22:33:05.132 DEBUG:boto:Params: {} 2022-09-09T22:33:05.133 DEBUG:boto:Token: None 2022-09-09T22:33:05.133 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/?versions 2022-09-09T22:33:05.134 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Y2pWTdM4OP2uZAv25+xFO3QtXo0= 2022-09-09T22:33:05.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Y2pWTdM4OP2uZAv25+xFO3QtXo0=', 'Content-Length': '0'} 2022-09-09T22:33:05.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb0c390f52be3a08-00631bbf21-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.143 DEBUG:boto:b'jkcgbj-12999false&nulltrue2022-09-09T22:30:44.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-09T22:30:44.766Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-09T22:30:44.684Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-09T22:30:44.625Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:05.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-09T22:33:05.145 DEBUG:boto:path=/jkcgbj-12/%26 2022-09-09T22:33:05.146 DEBUG:boto:auth_path=/jkcgbj-12/%26 2022-09-09T22:33:05.146 DEBUG:boto:path=/jkcgbj-12/%26?versionId=null 2022-09-09T22:33:05.147 DEBUG:boto:auth_path=/jkcgbj-12/%26?versionId=null 2022-09-09T22:33:05.147 DEBUG:boto:Method: GET 2022-09-09T22:33:05.148 DEBUG:boto:Path: /jkcgbj-12/%26?versionId=null 2022-09-09T22:33:05.149 DEBUG:boto:Data: 2022-09-09T22:33:05.149 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.150 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:05.150 DEBUG:boto:Port: 8000 2022-09-09T22:33:05.151 DEBUG:boto:Params: {} 2022-09-09T22:33:05.151 DEBUG:boto:Token: None 2022-09-09T22:33:05.152 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/%26?versionId=null 2022-09-09T22:33:05.153 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:WN0dAcGNCDWiN4ofVdgkYBghM08= 2022-09-09T22:33:05.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:WN0dAcGNCDWiN4ofVdgkYBghM08=', 'Content-Length': '0'} 2022-09-09T22:33:05.158 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0d806b169e22a84-00631bbf21-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.198 DEBUG:boto:path=/jkcgbj-12/%26 2022-09-09T22:33:05.199 DEBUG:boto:auth_path=/jkcgbj-12/%26 2022-09-09T22:33:05.199 DEBUG:boto:path=/jkcgbj-12/%26?versionId=null 2022-09-09T22:33:05.200 DEBUG:boto:auth_path=/jkcgbj-12/%26?versionId=null 2022-09-09T22:33:05.200 DEBUG:boto:Method: GET 2022-09-09T22:33:05.201 DEBUG:boto:Path: /jkcgbj-12/%26?versionId=null 2022-09-09T22:33:05.201 DEBUG:boto:Data: 2022-09-09T22:33:05.202 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.203 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:05.203 DEBUG:boto:Port: 8001 2022-09-09T22:33:05.204 DEBUG:boto:Params: {} 2022-09-09T22:33:05.204 DEBUG:boto:Token: None 2022-09-09T22:33:05.205 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/%26?versionId=null 2022-09-09T22:33:05.206 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:WN0dAcGNCDWiN4ofVdgkYBghM08= 2022-09-09T22:33:05.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:WN0dAcGNCDWiN4ofVdgkYBghM08=', 'Content-Length': '0'} 2022-09-09T22:33:05.210 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a20a7870b2ab156b-00631bbf21-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.211 DEBUG:boto:path=/jkcgbj-12/%26 2022-09-09T22:33:05.212 DEBUG:boto:auth_path=/jkcgbj-12/%26 2022-09-09T22:33:05.213 DEBUG:boto:path=/jkcgbj-12/%26?versionId=null 2022-09-09T22:33:05.214 DEBUG:boto:auth_path=/jkcgbj-12/%26?versionId=null 2022-09-09T22:33:05.215 DEBUG:boto:Method: HEAD 2022-09-09T22:33:05.216 DEBUG:boto:Path: /jkcgbj-12/%26?versionId=null 2022-09-09T22:33:05.217 DEBUG:boto:Data: 2022-09-09T22:33:05.218 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.219 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:05.219 DEBUG:boto:Port: 8000 2022-09-09T22:33:05.221 DEBUG:boto:Params: {} 2022-09-09T22:33:05.222 DEBUG:boto:Token: None 2022-09-09T22:33:05.223 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/%26?versionId=null 2022-09-09T22:33:05.224 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:XNswFtE17gif8WbVGJ7jCf0UBCY= 2022-09-09T22:33:05.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:XNswFtE17gif8WbVGJ7jCf0UBCY=', 'Content-Length': '0'} 2022-09-09T22:33:05.228 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000085b31a4fcc3e9e29-00631bbf21-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.230 DEBUG:boto:path=/jkcgbj-12/%26 2022-09-09T22:33:05.231 DEBUG:boto:auth_path=/jkcgbj-12/%26 2022-09-09T22:33:05.232 DEBUG:boto:path=/jkcgbj-12/%26?versionId=null 2022-09-09T22:33:05.233 DEBUG:boto:auth_path=/jkcgbj-12/%26?versionId=null 2022-09-09T22:33:05.233 DEBUG:boto:Method: HEAD 2022-09-09T22:33:05.234 DEBUG:boto:Path: /jkcgbj-12/%26?versionId=null 2022-09-09T22:33:05.234 DEBUG:boto:Data: 2022-09-09T22:33:05.235 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.235 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:05.236 DEBUG:boto:Port: 8001 2022-09-09T22:33:05.236 DEBUG:boto:Params: {} 2022-09-09T22:33:05.237 DEBUG:boto:Token: None 2022-09-09T22:33:05.238 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/%26?versionId=null 2022-09-09T22:33:05.239 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:XNswFtE17gif8WbVGJ7jCf0UBCY= 2022-09-09T22:33:05.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:XNswFtE17gif8WbVGJ7jCf0UBCY=', 'Content-Length': '0'} 2022-09-09T22:33:05.243 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000150d154647ab911f-00631bbf21-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-09T22:33:05.244 DEBUG:boto:path=/jkcgbj-12/%26 2022-09-09T22:33:05.245 DEBUG:boto:auth_path=/jkcgbj-12/%26 2022-09-09T22:33:05.245 DEBUG:boto:path=/jkcgbj-12/%26?versionId=null 2022-09-09T22:33:05.246 DEBUG:boto:auth_path=/jkcgbj-12/%26?versionId=null 2022-09-09T22:33:05.246 DEBUG:boto:Method: GET 2022-09-09T22:33:05.247 DEBUG:boto:Path: /jkcgbj-12/%26?versionId=null 2022-09-09T22:33:05.248 DEBUG:boto:Data: 2022-09-09T22:33:05.248 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.249 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:05.249 DEBUG:boto:Port: 8000 2022-09-09T22:33:05.250 DEBUG:boto:Params: {} 2022-09-09T22:33:05.251 DEBUG:boto:Token: None 2022-09-09T22:33:05.252 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/%26?versionId=null 2022-09-09T22:33:05.253 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:WN0dAcGNCDWiN4ofVdgkYBghM08= 2022-09-09T22:33:05.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:WN0dAcGNCDWiN4ofVdgkYBghM08=', 'Content-Length': '0'} 2022-09-09T22:33:05.261 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000035c3e64ec97bba48-00631bbf21-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.262 DEBUG:boto:path=/jkcgbj-12/%26 2022-09-09T22:33:05.262 DEBUG:boto:auth_path=/jkcgbj-12/%26 2022-09-09T22:33:05.263 DEBUG:boto:path=/jkcgbj-12/%26?versionId=null 2022-09-09T22:33:05.264 DEBUG:boto:auth_path=/jkcgbj-12/%26?versionId=null 2022-09-09T22:33:05.264 DEBUG:boto:Method: GET 2022-09-09T22:33:05.265 DEBUG:boto:Path: /jkcgbj-12/%26?versionId=null 2022-09-09T22:33:05.265 DEBUG:boto:Data: 2022-09-09T22:33:05.266 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.266 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:05.267 DEBUG:boto:Port: 8001 2022-09-09T22:33:05.267 DEBUG:boto:Params: {} 2022-09-09T22:33:05.268 DEBUG:boto:Token: None 2022-09-09T22:33:05.269 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/%26?versionId=null 2022-09-09T22:33:05.269 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:WN0dAcGNCDWiN4ofVdgkYBghM08= 2022-09-09T22:33:05.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:WN0dAcGNCDWiN4ofVdgkYBghM08=', 'Content-Length': '0'} 2022-09-09T22:33:05.273 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000060cd678a975d7912-00631bbf21-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.314 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:05.315 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:05.316 DEBUG:boto:path=/jkcgbj-12/?prefix=%26 2022-09-09T22:33:05.316 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=%26 2022-09-09T22:33:05.317 DEBUG:boto:Method: GET 2022-09-09T22:33:05.317 DEBUG:boto:Path: /jkcgbj-12/?prefix=%26 2022-09-09T22:33:05.318 DEBUG:boto:Data: 2022-09-09T22:33:05.318 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.319 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:05.319 DEBUG:boto:Port: 8000 2022-09-09T22:33:05.320 DEBUG:boto:Params: {} 2022-09-09T22:33:05.320 DEBUG:boto:Token: None 2022-09-09T22:33:05.321 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/ 2022-09-09T22:33:05.322 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rEcnr6g1geDvxONsg/Gh8IT/06w= 2022-09-09T22:33:05.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rEcnr6g1geDvxONsg/Gh8IT/06w=', 'Content-Length': '0'} 2022-09-09T22:33:05.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058337895d596fe6d-00631bbf21-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.332 DEBUG:boto:b'jkcgbj-12&1000false&2022-09-09T22:30:44.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:05.333 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:05.334 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:05.334 DEBUG:boto:path=/jkcgbj-12/?prefix=%26 2022-09-09T22:33:05.335 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=%26 2022-09-09T22:33:05.335 DEBUG:boto:Method: GET 2022-09-09T22:33:05.336 DEBUG:boto:Path: /jkcgbj-12/?prefix=%26 2022-09-09T22:33:05.336 DEBUG:boto:Data: 2022-09-09T22:33:05.337 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.337 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:05.338 DEBUG:boto:Port: 8001 2022-09-09T22:33:05.338 DEBUG:boto:Params: {} 2022-09-09T22:33:05.339 DEBUG:boto:Token: None 2022-09-09T22:33:05.340 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/ 2022-09-09T22:33:05.340 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rEcnr6g1geDvxONsg/Gh8IT/06w= 2022-09-09T22:33:05.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rEcnr6g1geDvxONsg/Gh8IT/06w=', 'Content-Length': '0'} 2022-09-09T22:33:05.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df2bac20dbb819f1-00631bbf21-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.349 DEBUG:boto:b'jkcgbj-12&1000false&2022-09-09T22:30:44.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:05.349 DEBUG:boto:path=/jkcgbj-12/%26 2022-09-09T22:33:05.350 DEBUG:boto:auth_path=/jkcgbj-12/%26 2022-09-09T22:33:05.351 DEBUG:boto:Method: HEAD 2022-09-09T22:33:05.351 DEBUG:boto:Path: /jkcgbj-12/%26 2022-09-09T22:33:05.352 DEBUG:boto:Data: 2022-09-09T22:33:05.352 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.353 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:05.353 DEBUG:boto:Port: 8000 2022-09-09T22:33:05.354 DEBUG:boto:Params: {} 2022-09-09T22:33:05.354 DEBUG:boto:Token: None 2022-09-09T22:33:05.355 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/%26 2022-09-09T22:33:05.356 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:P4nOWnjE9FqHSItVGKezJbw77ns= 2022-09-09T22:33:05.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:P4nOWnjE9FqHSItVGKezJbw77ns=', 'Content-Length': '0'} 2022-09-09T22:33:05.360 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a59160c8c08f4a79-00631bbf21-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.361 DEBUG:boto:path=/jkcgbj-12/%26 2022-09-09T22:33:05.361 DEBUG:boto:auth_path=/jkcgbj-12/%26 2022-09-09T22:33:05.362 DEBUG:boto:Method: HEAD 2022-09-09T22:33:05.362 DEBUG:boto:Path: /jkcgbj-12/%26 2022-09-09T22:33:05.363 DEBUG:boto:Data: 2022-09-09T22:33:05.363 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.364 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:05.364 DEBUG:boto:Port: 8001 2022-09-09T22:33:05.365 DEBUG:boto:Params: {} 2022-09-09T22:33:05.366 DEBUG:boto:Token: None 2022-09-09T22:33:05.369 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/%26 2022-09-09T22:33:05.371 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:P4nOWnjE9FqHSItVGKezJbw77ns= 2022-09-09T22:33:05.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:P4nOWnjE9FqHSItVGKezJbw77ns=', 'Content-Length': '0'} 2022-09-09T22:33:05.374 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000376287726dcb7ab4-00631bbf21-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-09T22:33:05.376 DEBUG:boto:path=/jkcgbj-12/%26 2022-09-09T22:33:05.377 DEBUG:boto:auth_path=/jkcgbj-12/%26 2022-09-09T22:33:05.377 DEBUG:boto:Method: GET 2022-09-09T22:33:05.378 DEBUG:boto:Path: /jkcgbj-12/%26 2022-09-09T22:33:05.379 DEBUG:boto:Data: 2022-09-09T22:33:05.380 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.380 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:05.381 DEBUG:boto:Port: 8000 2022-09-09T22:33:05.381 DEBUG:boto:Params: {} 2022-09-09T22:33:05.382 DEBUG:boto:Token: None 2022-09-09T22:33:05.383 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/%26 2022-09-09T22:33:05.383 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FhxmYygUA8pAslJqxd0Nnhmfqb0= 2022-09-09T22:33:05.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FhxmYygUA8pAslJqxd0Nnhmfqb0=', 'Content-Length': '0'} 2022-09-09T22:33:05.387 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063253d0636ff63ac-00631bbf21-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.388 DEBUG:boto:path=/jkcgbj-12/%26 2022-09-09T22:33:05.388 DEBUG:boto:auth_path=/jkcgbj-12/%26 2022-09-09T22:33:05.389 DEBUG:boto:Method: GET 2022-09-09T22:33:05.390 DEBUG:boto:Path: /jkcgbj-12/%26 2022-09-09T22:33:05.390 DEBUG:boto:Data: 2022-09-09T22:33:05.391 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.392 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:05.392 DEBUG:boto:Port: 8001 2022-09-09T22:33:05.393 DEBUG:boto:Params: {} 2022-09-09T22:33:05.393 DEBUG:boto:Token: None 2022-09-09T22:33:05.394 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/%26 2022-09-09T22:33:05.395 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FhxmYygUA8pAslJqxd0Nnhmfqb0= 2022-09-09T22:33:05.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FhxmYygUA8pAslJqxd0Nnhmfqb0=', 'Content-Length': '0'} 2022-09-09T22:33:05.399 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a369992245ec8cc-00631bbf21-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.442 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:05.443 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:05.444 DEBUG:boto:path=/jkcgbj-12/?prefix=%26 2022-09-09T22:33:05.444 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=%26 2022-09-09T22:33:05.445 DEBUG:boto:Method: GET 2022-09-09T22:33:05.445 DEBUG:boto:Path: /jkcgbj-12/?prefix=%26 2022-09-09T22:33:05.446 DEBUG:boto:Data: 2022-09-09T22:33:05.447 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.447 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:05.448 DEBUG:boto:Port: 8000 2022-09-09T22:33:05.448 DEBUG:boto:Params: {} 2022-09-09T22:33:05.449 DEBUG:boto:Token: None 2022-09-09T22:33:05.450 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/ 2022-09-09T22:33:05.451 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rEcnr6g1geDvxONsg/Gh8IT/06w= 2022-09-09T22:33:05.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rEcnr6g1geDvxONsg/Gh8IT/06w=', 'Content-Length': '0'} 2022-09-09T22:33:05.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a8de2c1c80e358f-00631bbf21-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.458 DEBUG:boto:b'jkcgbj-12&1000false&2022-09-09T22:30:44.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:05.459 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:05.459 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:05.460 DEBUG:boto:path=/jkcgbj-12/?prefix=%26 2022-09-09T22:33:05.461 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=%26 2022-09-09T22:33:05.462 DEBUG:boto:Method: GET 2022-09-09T22:33:05.462 DEBUG:boto:Path: /jkcgbj-12/?prefix=%26 2022-09-09T22:33:05.463 DEBUG:boto:Data: 2022-09-09T22:33:05.463 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.464 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:05.464 DEBUG:boto:Port: 8001 2022-09-09T22:33:05.465 DEBUG:boto:Params: {} 2022-09-09T22:33:05.466 DEBUG:boto:Token: None 2022-09-09T22:33:05.466 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/ 2022-09-09T22:33:05.467 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rEcnr6g1geDvxONsg/Gh8IT/06w= 2022-09-09T22:33:05.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rEcnr6g1geDvxONsg/Gh8IT/06w=', 'Content-Length': '0'} 2022-09-09T22:33:05.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018388425ebf5ff7a-00631bbf21-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.474 DEBUG:boto:b'jkcgbj-12&1000false&2022-09-09T22:30:44.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:05.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-09T22:33:05.475 DEBUG:boto:path=/jkcgbj-12/%3A 2022-09-09T22:33:05.476 DEBUG:boto:auth_path=/jkcgbj-12/%3A 2022-09-09T22:33:05.476 DEBUG:boto:path=/jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:05.477 DEBUG:boto:auth_path=/jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:05.477 DEBUG:boto:Method: GET 2022-09-09T22:33:05.478 DEBUG:boto:Path: /jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:05.478 DEBUG:boto:Data: 2022-09-09T22:33:05.479 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.479 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:05.480 DEBUG:boto:Port: 8000 2022-09-09T22:33:05.480 DEBUG:boto:Params: {} 2022-09-09T22:33:05.481 DEBUG:boto:Token: None 2022-09-09T22:33:05.482 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:05.483 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:3U655Yzin32UrLDsOCyCYpGLzZk= 2022-09-09T22:33:05.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:3U655Yzin32UrLDsOCyCYpGLzZk=', 'Content-Length': '0'} 2022-09-09T22:33:05.487 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d5eb71cd51514bf-00631bbf21-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.534 DEBUG:boto:path=/jkcgbj-12/%3A 2022-09-09T22:33:05.535 DEBUG:boto:auth_path=/jkcgbj-12/%3A 2022-09-09T22:33:05.536 DEBUG:boto:path=/jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:05.537 DEBUG:boto:auth_path=/jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:05.537 DEBUG:boto:Method: GET 2022-09-09T22:33:05.538 DEBUG:boto:Path: /jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:05.539 DEBUG:boto:Data: 2022-09-09T22:33:05.539 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.541 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:05.541 DEBUG:boto:Port: 8001 2022-09-09T22:33:05.542 DEBUG:boto:Params: {} 2022-09-09T22:33:05.543 DEBUG:boto:Token: None 2022-09-09T22:33:05.543 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:05.544 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:3U655Yzin32UrLDsOCyCYpGLzZk= 2022-09-09T22:33:05.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:3U655Yzin32UrLDsOCyCYpGLzZk=', 'Content-Length': '0'} 2022-09-09T22:33:05.548 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd17ecf4e72eacf8-00631bbf21-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.560 DEBUG:boto:path=/jkcgbj-12/%3A 2022-09-09T22:33:05.561 DEBUG:boto:auth_path=/jkcgbj-12/%3A 2022-09-09T22:33:05.562 DEBUG:boto:path=/jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:05.563 DEBUG:boto:auth_path=/jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:05.564 DEBUG:boto:Method: HEAD 2022-09-09T22:33:05.565 DEBUG:boto:Path: /jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:05.566 DEBUG:boto:Data: 2022-09-09T22:33:05.567 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.568 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:05.568 DEBUG:boto:Port: 8000 2022-09-09T22:33:05.569 DEBUG:boto:Params: {} 2022-09-09T22:33:05.570 DEBUG:boto:Token: None 2022-09-09T22:33:05.572 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:05.573 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+cDnqkhte6OOmShbYBP47P4lZfk= 2022-09-09T22:33:05.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+cDnqkhte6OOmShbYBP47P4lZfk=', 'Content-Length': '0'} 2022-09-09T22:33:05.577 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004c038f33cb515c3-00631bbf21-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.578 DEBUG:boto:path=/jkcgbj-12/%3A 2022-09-09T22:33:05.579 DEBUG:boto:auth_path=/jkcgbj-12/%3A 2022-09-09T22:33:05.580 DEBUG:boto:path=/jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:05.582 DEBUG:boto:auth_path=/jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:05.583 DEBUG:boto:Method: HEAD 2022-09-09T22:33:05.584 DEBUG:boto:Path: /jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:05.585 DEBUG:boto:Data: 2022-09-09T22:33:05.585 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.586 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:05.587 DEBUG:boto:Port: 8001 2022-09-09T22:33:05.588 DEBUG:boto:Params: {} 2022-09-09T22:33:05.589 DEBUG:boto:Token: None 2022-09-09T22:33:05.591 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:05.592 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+cDnqkhte6OOmShbYBP47P4lZfk= 2022-09-09T22:33:05.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+cDnqkhte6OOmShbYBP47P4lZfk=', 'Content-Length': '0'} 2022-09-09T22:33:05.597 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000098026b212d8783a6-00631bbf21-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-09T22:33:05.600 DEBUG:boto:path=/jkcgbj-12/%3A 2022-09-09T22:33:05.601 DEBUG:boto:auth_path=/jkcgbj-12/%3A 2022-09-09T22:33:05.602 DEBUG:boto:path=/jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:05.603 DEBUG:boto:auth_path=/jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:05.604 DEBUG:boto:Method: GET 2022-09-09T22:33:05.605 DEBUG:boto:Path: /jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:05.606 DEBUG:boto:Data: 2022-09-09T22:33:05.607 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.608 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:05.609 DEBUG:boto:Port: 8000 2022-09-09T22:33:05.611 DEBUG:boto:Params: {} 2022-09-09T22:33:05.612 DEBUG:boto:Token: None 2022-09-09T22:33:05.613 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:05.615 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:3U655Yzin32UrLDsOCyCYpGLzZk= 2022-09-09T22:33:05.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:3U655Yzin32UrLDsOCyCYpGLzZk=', 'Content-Length': '0'} 2022-09-09T22:33:05.626 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063a4de942b8fc13b-00631bbf21-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.628 DEBUG:boto:path=/jkcgbj-12/%3A 2022-09-09T22:33:05.628 DEBUG:boto:auth_path=/jkcgbj-12/%3A 2022-09-09T22:33:05.629 DEBUG:boto:path=/jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:05.630 DEBUG:boto:auth_path=/jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:05.630 DEBUG:boto:Method: GET 2022-09-09T22:33:05.631 DEBUG:boto:Path: /jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:05.631 DEBUG:boto:Data: 2022-09-09T22:33:05.632 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.632 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:05.633 DEBUG:boto:Port: 8001 2022-09-09T22:33:05.633 DEBUG:boto:Params: {} 2022-09-09T22:33:05.634 DEBUG:boto:Token: None 2022-09-09T22:33:05.635 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/%3A?versionId=null 2022-09-09T22:33:05.636 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:3U655Yzin32UrLDsOCyCYpGLzZk= 2022-09-09T22:33:05.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:3U655Yzin32UrLDsOCyCYpGLzZk=', 'Content-Length': '0'} 2022-09-09T22:33:05.639 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001916c6cfe8524382-00631bbf21-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.640 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:05.641 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:05.641 DEBUG:boto:path=/jkcgbj-12/?prefix=%3A 2022-09-09T22:33:05.642 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=%3A 2022-09-09T22:33:05.642 DEBUG:boto:Method: GET 2022-09-09T22:33:05.643 DEBUG:boto:Path: /jkcgbj-12/?prefix=%3A 2022-09-09T22:33:05.644 DEBUG:boto:Data: 2022-09-09T22:33:05.644 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.645 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:05.645 DEBUG:boto:Port: 8000 2022-09-09T22:33:05.646 DEBUG:boto:Params: {} 2022-09-09T22:33:05.646 DEBUG:boto:Token: None 2022-09-09T22:33:05.647 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/ 2022-09-09T22:33:05.647 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rEcnr6g1geDvxONsg/Gh8IT/06w= 2022-09-09T22:33:05.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rEcnr6g1geDvxONsg/Gh8IT/06w=', 'Content-Length': '0'} 2022-09-09T22:33:05.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001dad771dd81928c3-00631bbf21-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.654 DEBUG:boto:b'jkcgbj-12:1000false:2022-09-09T22:30:44.766Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:05.656 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:05.656 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:05.657 DEBUG:boto:path=/jkcgbj-12/?prefix=%3A 2022-09-09T22:33:05.658 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=%3A 2022-09-09T22:33:05.659 DEBUG:boto:Method: GET 2022-09-09T22:33:05.660 DEBUG:boto:Path: /jkcgbj-12/?prefix=%3A 2022-09-09T22:33:05.661 DEBUG:boto:Data: 2022-09-09T22:33:05.662 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.663 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:05.664 DEBUG:boto:Port: 8001 2022-09-09T22:33:05.666 DEBUG:boto:Params: {} 2022-09-09T22:33:05.666 DEBUG:boto:Token: None 2022-09-09T22:33:05.667 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/ 2022-09-09T22:33:05.668 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rEcnr6g1geDvxONsg/Gh8IT/06w= 2022-09-09T22:33:05.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rEcnr6g1geDvxONsg/Gh8IT/06w=', 'Content-Length': '0'} 2022-09-09T22:33:05.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c1c9215bf05e808-00631bbf21-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.674 DEBUG:boto:b'jkcgbj-12:1000false:2022-09-09T22:30:44.766Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:05.675 DEBUG:boto:path=/jkcgbj-12/%3A 2022-09-09T22:33:05.675 DEBUG:boto:auth_path=/jkcgbj-12/%3A 2022-09-09T22:33:05.676 DEBUG:boto:Method: HEAD 2022-09-09T22:33:05.676 DEBUG:boto:Path: /jkcgbj-12/%3A 2022-09-09T22:33:05.677 DEBUG:boto:Data: 2022-09-09T22:33:05.677 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.678 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:05.678 DEBUG:boto:Port: 8000 2022-09-09T22:33:05.679 DEBUG:boto:Params: {} 2022-09-09T22:33:05.679 DEBUG:boto:Token: None 2022-09-09T22:33:05.680 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/%3A 2022-09-09T22:33:05.681 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CmTg4YbJybW3P9jE6+CrhkQdTYM= 2022-09-09T22:33:05.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CmTg4YbJybW3P9jE6+CrhkQdTYM=', 'Content-Length': '0'} 2022-09-09T22:33:05.684 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006c5be75bd9f6abfc-00631bbf21-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.685 DEBUG:boto:path=/jkcgbj-12/%3A 2022-09-09T22:33:05.686 DEBUG:boto:auth_path=/jkcgbj-12/%3A 2022-09-09T22:33:05.686 DEBUG:boto:Method: HEAD 2022-09-09T22:33:05.687 DEBUG:boto:Path: /jkcgbj-12/%3A 2022-09-09T22:33:05.687 DEBUG:boto:Data: 2022-09-09T22:33:05.688 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.688 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:05.689 DEBUG:boto:Port: 8001 2022-09-09T22:33:05.689 DEBUG:boto:Params: {} 2022-09-09T22:33:05.690 DEBUG:boto:Token: None 2022-09-09T22:33:05.691 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/%3A 2022-09-09T22:33:05.691 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CmTg4YbJybW3P9jE6+CrhkQdTYM= 2022-09-09T22:33:05.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CmTg4YbJybW3P9jE6+CrhkQdTYM=', 'Content-Length': '0'} 2022-09-09T22:33:05.695 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091a212c2887554a5-00631bbf21-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-09T22:33:05.696 DEBUG:boto:path=/jkcgbj-12/%3A 2022-09-09T22:33:05.697 DEBUG:boto:auth_path=/jkcgbj-12/%3A 2022-09-09T22:33:05.697 DEBUG:boto:Method: GET 2022-09-09T22:33:05.698 DEBUG:boto:Path: /jkcgbj-12/%3A 2022-09-09T22:33:05.698 DEBUG:boto:Data: 2022-09-09T22:33:05.699 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.699 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:05.700 DEBUG:boto:Port: 8000 2022-09-09T22:33:05.701 DEBUG:boto:Params: {} 2022-09-09T22:33:05.701 DEBUG:boto:Token: None 2022-09-09T22:33:05.702 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/%3A 2022-09-09T22:33:05.703 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:l4fWz4teZe1CMf4QNUjSgD+cw3s= 2022-09-09T22:33:05.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:l4fWz4teZe1CMf4QNUjSgD+cw3s=', 'Content-Length': '0'} 2022-09-09T22:33:05.707 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c15e976fccd92b90-00631bbf21-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.708 DEBUG:boto:path=/jkcgbj-12/%3A 2022-09-09T22:33:05.709 DEBUG:boto:auth_path=/jkcgbj-12/%3A 2022-09-09T22:33:05.710 DEBUG:boto:Method: GET 2022-09-09T22:33:05.710 DEBUG:boto:Path: /jkcgbj-12/%3A 2022-09-09T22:33:05.711 DEBUG:boto:Data: 2022-09-09T22:33:05.711 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.712 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:05.713 DEBUG:boto:Port: 8001 2022-09-09T22:33:05.713 DEBUG:boto:Params: {} 2022-09-09T22:33:05.715 DEBUG:boto:Token: None 2022-09-09T22:33:05.716 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/%3A 2022-09-09T22:33:05.717 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:l4fWz4teZe1CMf4QNUjSgD+cw3s= 2022-09-09T22:33:05.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:l4fWz4teZe1CMf4QNUjSgD+cw3s=', 'Content-Length': '0'} 2022-09-09T22:33:05.721 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a058ebd05a624bc9-00631bbf21-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.762 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:05.764 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:05.765 DEBUG:boto:path=/jkcgbj-12/?prefix=%3A 2022-09-09T22:33:05.766 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=%3A 2022-09-09T22:33:05.766 DEBUG:boto:Method: GET 2022-09-09T22:33:05.768 DEBUG:boto:Path: /jkcgbj-12/?prefix=%3A 2022-09-09T22:33:05.769 DEBUG:boto:Data: 2022-09-09T22:33:05.769 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.770 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:05.771 DEBUG:boto:Port: 8000 2022-09-09T22:33:05.772 DEBUG:boto:Params: {} 2022-09-09T22:33:05.772 DEBUG:boto:Token: None 2022-09-09T22:33:05.773 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/ 2022-09-09T22:33:05.774 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rEcnr6g1geDvxONsg/Gh8IT/06w= 2022-09-09T22:33:05.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rEcnr6g1geDvxONsg/Gh8IT/06w=', 'Content-Length': '0'} 2022-09-09T22:33:05.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000733021836d11b8e8-00631bbf21-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.783 DEBUG:boto:b'jkcgbj-12:1000false:2022-09-09T22:30:44.766Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:05.786 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:05.787 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:05.788 DEBUG:boto:path=/jkcgbj-12/?prefix=%3A 2022-09-09T22:33:05.789 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=%3A 2022-09-09T22:33:05.789 DEBUG:boto:Method: GET 2022-09-09T22:33:05.790 DEBUG:boto:Path: /jkcgbj-12/?prefix=%3A 2022-09-09T22:33:05.791 DEBUG:boto:Data: 2022-09-09T22:33:05.791 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.792 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:05.792 DEBUG:boto:Port: 8001 2022-09-09T22:33:05.793 DEBUG:boto:Params: {} 2022-09-09T22:33:05.794 DEBUG:boto:Token: None 2022-09-09T22:33:05.795 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/ 2022-09-09T22:33:05.796 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rEcnr6g1geDvxONsg/Gh8IT/06w= 2022-09-09T22:33:05.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rEcnr6g1geDvxONsg/Gh8IT/06w=', 'Content-Length': '0'} 2022-09-09T22:33:05.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4822ff9cc31929c-00631bbf21-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.803 DEBUG:boto:b'jkcgbj-12:1000false:2022-09-09T22:30:44.766Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:05.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-09T22:33:05.804 DEBUG:boto:path=/jkcgbj-12/_myobj 2022-09-09T22:33:05.805 DEBUG:boto:auth_path=/jkcgbj-12/_myobj 2022-09-09T22:33:05.806 DEBUG:boto:path=/jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:05.807 DEBUG:boto:auth_path=/jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:05.807 DEBUG:boto:Method: GET 2022-09-09T22:33:05.808 DEBUG:boto:Path: /jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:05.809 DEBUG:boto:Data: 2022-09-09T22:33:05.809 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.810 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:05.811 DEBUG:boto:Port: 8000 2022-09-09T22:33:05.811 DEBUG:boto:Params: {} 2022-09-09T22:33:05.812 DEBUG:boto:Token: None 2022-09-09T22:33:05.813 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:05.814 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Eh5HB4vDuhYWO+1HGrWTw1FdmWg= 2022-09-09T22:33:05.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Eh5HB4vDuhYWO+1HGrWTw1FdmWg=', 'Content-Length': '0'} 2022-09-09T22:33:05.819 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e15a658466179d71-00631bbf21-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.866 DEBUG:boto:path=/jkcgbj-12/_myobj 2022-09-09T22:33:05.867 DEBUG:boto:auth_path=/jkcgbj-12/_myobj 2022-09-09T22:33:05.867 DEBUG:boto:path=/jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:05.868 DEBUG:boto:auth_path=/jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:05.868 DEBUG:boto:Method: GET 2022-09-09T22:33:05.869 DEBUG:boto:Path: /jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:05.870 DEBUG:boto:Data: 2022-09-09T22:33:05.870 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.871 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:05.871 DEBUG:boto:Port: 8001 2022-09-09T22:33:05.872 DEBUG:boto:Params: {} 2022-09-09T22:33:05.872 DEBUG:boto:Token: None 2022-09-09T22:33:05.874 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:05.874 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Eh5HB4vDuhYWO+1HGrWTw1FdmWg= 2022-09-09T22:33:05.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Eh5HB4vDuhYWO+1HGrWTw1FdmWg=', 'Content-Length': '0'} 2022-09-09T22:33:05.878 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000537d4bbdbffb7b28-00631bbf21-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.879 DEBUG:boto:path=/jkcgbj-12/_myobj 2022-09-09T22:33:05.880 DEBUG:boto:auth_path=/jkcgbj-12/_myobj 2022-09-09T22:33:05.880 DEBUG:boto:path=/jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:05.881 DEBUG:boto:auth_path=/jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:05.881 DEBUG:boto:Method: HEAD 2022-09-09T22:33:05.882 DEBUG:boto:Path: /jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:05.882 DEBUG:boto:Data: 2022-09-09T22:33:05.883 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.883 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:05.884 DEBUG:boto:Port: 8000 2022-09-09T22:33:05.885 DEBUG:boto:Params: {} 2022-09-09T22:33:05.886 DEBUG:boto:Token: None 2022-09-09T22:33:05.886 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:05.887 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:DhQIuTUtrFoWhDHKbIF93SohSLQ= 2022-09-09T22:33:05.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:DhQIuTUtrFoWhDHKbIF93SohSLQ=', 'Content-Length': '0'} 2022-09-09T22:33:05.891 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047ca1fe3aebff27e-00631bbf21-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.892 DEBUG:boto:path=/jkcgbj-12/_myobj 2022-09-09T22:33:05.892 DEBUG:boto:auth_path=/jkcgbj-12/_myobj 2022-09-09T22:33:05.893 DEBUG:boto:path=/jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:05.893 DEBUG:boto:auth_path=/jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:05.894 DEBUG:boto:Method: HEAD 2022-09-09T22:33:05.894 DEBUG:boto:Path: /jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:05.895 DEBUG:boto:Data: 2022-09-09T22:33:05.895 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.896 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:05.896 DEBUG:boto:Port: 8001 2022-09-09T22:33:05.897 DEBUG:boto:Params: {} 2022-09-09T22:33:05.897 DEBUG:boto:Token: None 2022-09-09T22:33:05.898 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:05.899 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:DhQIuTUtrFoWhDHKbIF93SohSLQ= 2022-09-09T22:33:05.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:DhQIuTUtrFoWhDHKbIF93SohSLQ=', 'Content-Length': '0'} 2022-09-09T22:33:05.903 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007c7bdf43df101d96-00631bbf21-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-09T22:33:05.904 DEBUG:boto:path=/jkcgbj-12/_myobj 2022-09-09T22:33:05.905 DEBUG:boto:auth_path=/jkcgbj-12/_myobj 2022-09-09T22:33:05.906 DEBUG:boto:path=/jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:05.907 DEBUG:boto:auth_path=/jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:05.908 DEBUG:boto:Method: GET 2022-09-09T22:33:05.909 DEBUG:boto:Path: /jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:05.911 DEBUG:boto:Data: 2022-09-09T22:33:05.912 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.913 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:05.914 DEBUG:boto:Port: 8000 2022-09-09T22:33:05.915 DEBUG:boto:Params: {} 2022-09-09T22:33:05.916 DEBUG:boto:Token: None 2022-09-09T22:33:05.917 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:05.919 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Eh5HB4vDuhYWO+1HGrWTw1FdmWg= 2022-09-09T22:33:05.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Eh5HB4vDuhYWO+1HGrWTw1FdmWg=', 'Content-Length': '0'} 2022-09-09T22:33:05.924 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0f9ca7ffad7d250-00631bbf21-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.926 DEBUG:boto:path=/jkcgbj-12/_myobj 2022-09-09T22:33:05.926 DEBUG:boto:auth_path=/jkcgbj-12/_myobj 2022-09-09T22:33:05.927 DEBUG:boto:path=/jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:05.928 DEBUG:boto:auth_path=/jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:05.928 DEBUG:boto:Method: GET 2022-09-09T22:33:05.929 DEBUG:boto:Path: /jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:05.930 DEBUG:boto:Data: 2022-09-09T22:33:05.930 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.931 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:05.932 DEBUG:boto:Port: 8001 2022-09-09T22:33:05.932 DEBUG:boto:Params: {} 2022-09-09T22:33:05.933 DEBUG:boto:Token: None 2022-09-09T22:33:05.934 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/_myobj?versionId=null 2022-09-09T22:33:05.935 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Eh5HB4vDuhYWO+1HGrWTw1FdmWg= 2022-09-09T22:33:05.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Eh5HB4vDuhYWO+1HGrWTw1FdmWg=', 'Content-Length': '0'} 2022-09-09T22:33:05.938 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001672148d5d7eb213-00631bbf21-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:05.986 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:05.987 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:05.988 DEBUG:boto:path=/jkcgbj-12/?prefix=_myobj 2022-09-09T22:33:05.988 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=_myobj 2022-09-09T22:33:05.989 DEBUG:boto:Method: GET 2022-09-09T22:33:05.989 DEBUG:boto:Path: /jkcgbj-12/?prefix=_myobj 2022-09-09T22:33:05.990 DEBUG:boto:Data: 2022-09-09T22:33:05.990 DEBUG:boto:Headers: {} 2022-09-09T22:33:05.991 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:05.991 DEBUG:boto:Port: 8000 2022-09-09T22:33:05.993 DEBUG:boto:Params: {} 2022-09-09T22:33:05.995 DEBUG:boto:Token: None 2022-09-09T22:33:05.996 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:05 GMT /jkcgbj-12/ 2022-09-09T22:33:05.996 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rEcnr6g1geDvxONsg/Gh8IT/06w= 2022-09-09T22:33:05.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rEcnr6g1geDvxONsg/Gh8IT/06w=', 'Content-Length': '0'} 2022-09-09T22:33:06.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003527e5e6c82e6398-00631bbf22-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:06.012 DEBUG:boto:b'jkcgbj-12_myobj1000false_myobj2022-09-09T22:30:44.684Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:06.014 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:06.016 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:06.017 DEBUG:boto:path=/jkcgbj-12/?prefix=_myobj 2022-09-09T22:33:06.018 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=_myobj 2022-09-09T22:33:06.019 DEBUG:boto:Method: GET 2022-09-09T22:33:06.020 DEBUG:boto:Path: /jkcgbj-12/?prefix=_myobj 2022-09-09T22:33:06.021 DEBUG:boto:Data: 2022-09-09T22:33:06.022 DEBUG:boto:Headers: {} 2022-09-09T22:33:06.023 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:06.025 DEBUG:boto:Port: 8001 2022-09-09T22:33:06.026 DEBUG:boto:Params: {} 2022-09-09T22:33:06.027 DEBUG:boto:Token: None 2022-09-09T22:33:06.028 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:06 GMT /jkcgbj-12/ 2022-09-09T22:33:06.030 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8dOXd09iM9R5mhTyhOrCKLe7Mx4= 2022-09-09T22:33:06.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8dOXd09iM9R5mhTyhOrCKLe7Mx4=', 'Content-Length': '0'} 2022-09-09T22:33:06.037 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000988d3cb5dee716a1-00631bbf22-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:06.039 DEBUG:boto:b'jkcgbj-12_myobj1000false_myobj2022-09-09T22:30:44.684Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:06.040 DEBUG:boto:path=/jkcgbj-12/_myobj 2022-09-09T22:33:06.041 DEBUG:boto:auth_path=/jkcgbj-12/_myobj 2022-09-09T22:33:06.042 DEBUG:boto:Method: HEAD 2022-09-09T22:33:06.043 DEBUG:boto:Path: /jkcgbj-12/_myobj 2022-09-09T22:33:06.044 DEBUG:boto:Data: 2022-09-09T22:33:06.045 DEBUG:boto:Headers: {} 2022-09-09T22:33:06.046 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:06.047 DEBUG:boto:Port: 8000 2022-09-09T22:33:06.048 DEBUG:boto:Params: {} 2022-09-09T22:33:06.048 DEBUG:boto:Token: None 2022-09-09T22:33:06.049 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:06 GMT /jkcgbj-12/_myobj 2022-09-09T22:33:06.050 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:q2ChF2JnNXGRvgNj1RSeWuatamA= 2022-09-09T22:33:06.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:q2ChF2JnNXGRvgNj1RSeWuatamA=', 'Content-Length': '0'} 2022-09-09T22:33:06.053 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b9a4304aae3aee1-00631bbf22-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:06.054 DEBUG:boto:path=/jkcgbj-12/_myobj 2022-09-09T22:33:06.055 DEBUG:boto:auth_path=/jkcgbj-12/_myobj 2022-09-09T22:33:06.055 DEBUG:boto:Method: HEAD 2022-09-09T22:33:06.057 DEBUG:boto:Path: /jkcgbj-12/_myobj 2022-09-09T22:33:06.057 DEBUG:boto:Data: 2022-09-09T22:33:06.058 DEBUG:boto:Headers: {} 2022-09-09T22:33:06.058 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:06.059 DEBUG:boto:Port: 8001 2022-09-09T22:33:06.060 DEBUG:boto:Params: {} 2022-09-09T22:33:06.061 DEBUG:boto:Token: None 2022-09-09T22:33:06.062 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:06 GMT /jkcgbj-12/_myobj 2022-09-09T22:33:06.063 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:q2ChF2JnNXGRvgNj1RSeWuatamA= 2022-09-09T22:33:06.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:q2ChF2JnNXGRvgNj1RSeWuatamA=', 'Content-Length': '0'} 2022-09-09T22:33:06.068 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003630acd012811f01-00631bbf22-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:06.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-09T22:33:06.071 DEBUG:boto:path=/jkcgbj-12/_myobj 2022-09-09T22:33:06.072 DEBUG:boto:auth_path=/jkcgbj-12/_myobj 2022-09-09T22:33:06.073 DEBUG:boto:Method: GET 2022-09-09T22:33:06.073 DEBUG:boto:Path: /jkcgbj-12/_myobj 2022-09-09T22:33:06.074 DEBUG:boto:Data: 2022-09-09T22:33:06.074 DEBUG:boto:Headers: {} 2022-09-09T22:33:06.075 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:06.076 DEBUG:boto:Port: 8000 2022-09-09T22:33:06.076 DEBUG:boto:Params: {} 2022-09-09T22:33:06.077 DEBUG:boto:Token: None 2022-09-09T22:33:06.078 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:06 GMT /jkcgbj-12/_myobj 2022-09-09T22:33:06.078 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:xM8RxOnryKnD8fymGu0A++0DsgA= 2022-09-09T22:33:06.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:xM8RxOnryKnD8fymGu0A++0DsgA=', 'Content-Length': '0'} 2022-09-09T22:33:06.082 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006c6fe6551b37cb0a-00631bbf22-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:06.083 DEBUG:boto:path=/jkcgbj-12/_myobj 2022-09-09T22:33:06.084 DEBUG:boto:auth_path=/jkcgbj-12/_myobj 2022-09-09T22:33:06.084 DEBUG:boto:Method: GET 2022-09-09T22:33:06.085 DEBUG:boto:Path: /jkcgbj-12/_myobj 2022-09-09T22:33:06.086 DEBUG:boto:Data: 2022-09-09T22:33:06.086 DEBUG:boto:Headers: {} 2022-09-09T22:33:06.087 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:06.088 DEBUG:boto:Port: 8001 2022-09-09T22:33:06.088 DEBUG:boto:Params: {} 2022-09-09T22:33:06.089 DEBUG:boto:Token: None 2022-09-09T22:33:06.090 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:06 GMT /jkcgbj-12/_myobj 2022-09-09T22:33:06.090 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:xM8RxOnryKnD8fymGu0A++0DsgA= 2022-09-09T22:33:06.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:xM8RxOnryKnD8fymGu0A++0DsgA=', 'Content-Length': '0'} 2022-09-09T22:33:06.095 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009de06d0afaae93a9-00631bbf22-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:06.096 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:06.097 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:06.097 DEBUG:boto:path=/jkcgbj-12/?prefix=_myobj 2022-09-09T22:33:06.098 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=_myobj 2022-09-09T22:33:06.099 DEBUG:boto:Method: GET 2022-09-09T22:33:06.100 DEBUG:boto:Path: /jkcgbj-12/?prefix=_myobj 2022-09-09T22:33:06.101 DEBUG:boto:Data: 2022-09-09T22:33:06.101 DEBUG:boto:Headers: {} 2022-09-09T22:33:06.102 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:06.103 DEBUG:boto:Port: 8000 2022-09-09T22:33:06.103 DEBUG:boto:Params: {} 2022-09-09T22:33:06.104 DEBUG:boto:Token: None 2022-09-09T22:33:06.105 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:06 GMT /jkcgbj-12/ 2022-09-09T22:33:06.109 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8dOXd09iM9R5mhTyhOrCKLe7Mx4= 2022-09-09T22:33:06.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8dOXd09iM9R5mhTyhOrCKLe7Mx4=', 'Content-Length': '0'} 2022-09-09T22:33:06.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f4a7761069f838f-00631bbf22-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:06.158 DEBUG:boto:b'jkcgbj-12_myobj1000false_myobj2022-09-09T22:30:44.684Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:06.160 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:06.161 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:06.162 DEBUG:boto:path=/jkcgbj-12/?prefix=_myobj 2022-09-09T22:33:06.163 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=_myobj 2022-09-09T22:33:06.164 DEBUG:boto:Method: GET 2022-09-09T22:33:06.165 DEBUG:boto:Path: /jkcgbj-12/?prefix=_myobj 2022-09-09T22:33:06.166 DEBUG:boto:Data: 2022-09-09T22:33:06.167 DEBUG:boto:Headers: {} 2022-09-09T22:33:06.168 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:06.168 DEBUG:boto:Port: 8001 2022-09-09T22:33:06.169 DEBUG:boto:Params: {} 2022-09-09T22:33:06.170 DEBUG:boto:Token: None 2022-09-09T22:33:06.171 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:06 GMT /jkcgbj-12/ 2022-09-09T22:33:06.172 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8dOXd09iM9R5mhTyhOrCKLe7Mx4= 2022-09-09T22:33:06.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8dOXd09iM9R5mhTyhOrCKLe7Mx4=', 'Content-Length': '0'} 2022-09-09T22:33:06.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8d884e79a9b9bcc-00631bbf22-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:06.180 DEBUG:boto:b'jkcgbj-12_myobj1000false_myobj2022-09-09T22:30:44.684Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:06.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:33:06.182 DEBUG:boto:path=/jkcgbj-12/myobj 2022-09-09T22:33:06.182 DEBUG:boto:auth_path=/jkcgbj-12/myobj 2022-09-09T22:33:06.183 DEBUG:boto:path=/jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:06.184 DEBUG:boto:auth_path=/jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:06.185 DEBUG:boto:Method: GET 2022-09-09T22:33:06.185 DEBUG:boto:Path: /jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:06.186 DEBUG:boto:Data: 2022-09-09T22:33:06.187 DEBUG:boto:Headers: {} 2022-09-09T22:33:06.188 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:06.189 DEBUG:boto:Port: 8000 2022-09-09T22:33:06.190 DEBUG:boto:Params: {} 2022-09-09T22:33:06.191 DEBUG:boto:Token: None 2022-09-09T22:33:06.191 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:06 GMT /jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:06.192 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4fEbx92Xa4wVtRfdCwL7SLXfakA= 2022-09-09T22:33:06.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4fEbx92Xa4wVtRfdCwL7SLXfakA=', 'Content-Length': '0'} 2022-09-09T22:33:06.197 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d6867ae853c7ed3e-00631bbf22-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:06.199 DEBUG:boto:path=/jkcgbj-12/myobj 2022-09-09T22:33:06.200 DEBUG:boto:auth_path=/jkcgbj-12/myobj 2022-09-09T22:33:06.201 DEBUG:boto:path=/jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:06.202 DEBUG:boto:auth_path=/jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:06.203 DEBUG:boto:Method: GET 2022-09-09T22:33:06.204 DEBUG:boto:Path: /jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:06.205 DEBUG:boto:Data: 2022-09-09T22:33:06.206 DEBUG:boto:Headers: {} 2022-09-09T22:33:06.207 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:06.208 DEBUG:boto:Port: 8001 2022-09-09T22:33:06.209 DEBUG:boto:Params: {} 2022-09-09T22:33:06.210 DEBUG:boto:Token: None 2022-09-09T22:33:06.211 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:06 GMT /jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:06.212 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4fEbx92Xa4wVtRfdCwL7SLXfakA= 2022-09-09T22:33:06.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4fEbx92Xa4wVtRfdCwL7SLXfakA=', 'Content-Length': '0'} 2022-09-09T22:33:06.217 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b3fe5fa132a14b3-00631bbf22-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:06.270 DEBUG:boto:path=/jkcgbj-12/myobj 2022-09-09T22:33:06.271 DEBUG:boto:auth_path=/jkcgbj-12/myobj 2022-09-09T22:33:06.272 DEBUG:boto:path=/jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:06.273 DEBUG:boto:auth_path=/jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:06.274 DEBUG:boto:Method: HEAD 2022-09-09T22:33:06.275 DEBUG:boto:Path: /jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:06.277 DEBUG:boto:Data: 2022-09-09T22:33:06.278 DEBUG:boto:Headers: {} 2022-09-09T22:33:06.279 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:06.281 DEBUG:boto:Port: 8000 2022-09-09T22:33:06.281 DEBUG:boto:Params: {} 2022-09-09T22:33:06.282 DEBUG:boto:Token: None 2022-09-09T22:33:06.283 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:06 GMT /jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:06.284 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:hxmxJ/PFsd3N5765pjJ2yfgsb84= 2022-09-09T22:33:06.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:hxmxJ/PFsd3N5765pjJ2yfgsb84=', 'Content-Length': '0'} 2022-09-09T22:33:06.288 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008f03c190432750b-00631bbf22-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:06.289 DEBUG:boto:path=/jkcgbj-12/myobj 2022-09-09T22:33:06.290 DEBUG:boto:auth_path=/jkcgbj-12/myobj 2022-09-09T22:33:06.291 DEBUG:boto:path=/jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:06.292 DEBUG:boto:auth_path=/jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:06.292 DEBUG:boto:Method: HEAD 2022-09-09T22:33:06.293 DEBUG:boto:Path: /jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:06.293 DEBUG:boto:Data: 2022-09-09T22:33:06.294 DEBUG:boto:Headers: {} 2022-09-09T22:33:06.295 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:06.295 DEBUG:boto:Port: 8001 2022-09-09T22:33:06.296 DEBUG:boto:Params: {} 2022-09-09T22:33:06.297 DEBUG:boto:Token: None 2022-09-09T22:33:06.298 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:06 GMT /jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:06.300 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:hxmxJ/PFsd3N5765pjJ2yfgsb84= 2022-09-09T22:33:06.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:hxmxJ/PFsd3N5765pjJ2yfgsb84=', 'Content-Length': '0'} 2022-09-09T22:33:06.305 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c48a90bc3821646e-00631bbf22-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:06.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:33:06.308 DEBUG:boto:path=/jkcgbj-12/myobj 2022-09-09T22:33:06.309 DEBUG:boto:auth_path=/jkcgbj-12/myobj 2022-09-09T22:33:06.310 DEBUG:boto:path=/jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:06.311 DEBUG:boto:auth_path=/jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:06.313 DEBUG:boto:Method: GET 2022-09-09T22:33:06.314 DEBUG:boto:Path: /jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:06.315 DEBUG:boto:Data: 2022-09-09T22:33:06.316 DEBUG:boto:Headers: {} 2022-09-09T22:33:06.317 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:06.318 DEBUG:boto:Port: 8000 2022-09-09T22:33:06.319 DEBUG:boto:Params: {} 2022-09-09T22:33:06.320 DEBUG:boto:Token: None 2022-09-09T22:33:06.320 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:06 GMT /jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:06.321 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4fEbx92Xa4wVtRfdCwL7SLXfakA= 2022-09-09T22:33:06.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4fEbx92Xa4wVtRfdCwL7SLXfakA=', 'Content-Length': '0'} 2022-09-09T22:33:06.325 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000decf9b12a4610187-00631bbf22-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:06.326 DEBUG:boto:path=/jkcgbj-12/myobj 2022-09-09T22:33:06.327 DEBUG:boto:auth_path=/jkcgbj-12/myobj 2022-09-09T22:33:06.327 DEBUG:boto:path=/jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:06.328 DEBUG:boto:auth_path=/jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:06.329 DEBUG:boto:Method: GET 2022-09-09T22:33:06.329 DEBUG:boto:Path: /jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:06.330 DEBUG:boto:Data: 2022-09-09T22:33:06.331 DEBUG:boto:Headers: {} 2022-09-09T22:33:06.331 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:06.332 DEBUG:boto:Port: 8001 2022-09-09T22:33:06.333 DEBUG:boto:Params: {} 2022-09-09T22:33:06.333 DEBUG:boto:Token: None 2022-09-09T22:33:06.334 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:06 GMT /jkcgbj-12/myobj?versionId=null 2022-09-09T22:33:06.335 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4fEbx92Xa4wVtRfdCwL7SLXfakA= 2022-09-09T22:33:06.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4fEbx92Xa4wVtRfdCwL7SLXfakA=', 'Content-Length': '0'} 2022-09-09T22:33:06.339 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ca74e012549190a-00631bbf22-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:06.341 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:06.341 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:06.342 DEBUG:boto:path=/jkcgbj-12/?prefix=myobj 2022-09-09T22:33:06.343 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=myobj 2022-09-09T22:33:06.343 DEBUG:boto:Method: GET 2022-09-09T22:33:06.344 DEBUG:boto:Path: /jkcgbj-12/?prefix=myobj 2022-09-09T22:33:06.345 DEBUG:boto:Data: 2022-09-09T22:33:06.345 DEBUG:boto:Headers: {} 2022-09-09T22:33:06.346 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:06.347 DEBUG:boto:Port: 8000 2022-09-09T22:33:06.348 DEBUG:boto:Params: {} 2022-09-09T22:33:06.348 DEBUG:boto:Token: None 2022-09-09T22:33:06.349 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:06 GMT /jkcgbj-12/ 2022-09-09T22:33:06.350 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8dOXd09iM9R5mhTyhOrCKLe7Mx4= 2022-09-09T22:33:06.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8dOXd09iM9R5mhTyhOrCKLe7Mx4=', 'Content-Length': '0'} 2022-09-09T22:33:06.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b778aac04844a687-00631bbf22-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:06.357 DEBUG:boto:b'jkcgbj-12myobj1000falsemyobj2022-09-09T22:30:44.625Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:06.358 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:06.359 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:06.359 DEBUG:boto:path=/jkcgbj-12/?prefix=myobj 2022-09-09T22:33:06.360 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=myobj 2022-09-09T22:33:06.361 DEBUG:boto:Method: GET 2022-09-09T22:33:06.361 DEBUG:boto:Path: /jkcgbj-12/?prefix=myobj 2022-09-09T22:33:06.362 DEBUG:boto:Data: 2022-09-09T22:33:06.362 DEBUG:boto:Headers: {} 2022-09-09T22:33:06.363 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:06.364 DEBUG:boto:Port: 8001 2022-09-09T22:33:06.364 DEBUG:boto:Params: {} 2022-09-09T22:33:06.365 DEBUG:boto:Token: None 2022-09-09T22:33:06.366 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:06 GMT /jkcgbj-12/ 2022-09-09T22:33:06.367 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8dOXd09iM9R5mhTyhOrCKLe7Mx4= 2022-09-09T22:33:06.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8dOXd09iM9R5mhTyhOrCKLe7Mx4=', 'Content-Length': '0'} 2022-09-09T22:33:06.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b8959713e7a53cc-00631bbf22-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:06.374 DEBUG:boto:b'jkcgbj-12myobj1000falsemyobj2022-09-09T22:30:44.625Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:06.375 DEBUG:boto:path=/jkcgbj-12/myobj 2022-09-09T22:33:06.377 DEBUG:boto:auth_path=/jkcgbj-12/myobj 2022-09-09T22:33:06.379 DEBUG:boto:Method: HEAD 2022-09-09T22:33:06.380 DEBUG:boto:Path: /jkcgbj-12/myobj 2022-09-09T22:33:06.381 DEBUG:boto:Data: 2022-09-09T22:33:06.383 DEBUG:boto:Headers: {} 2022-09-09T22:33:06.383 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:06.384 DEBUG:boto:Port: 8000 2022-09-09T22:33:06.386 DEBUG:boto:Params: {} 2022-09-09T22:33:06.387 DEBUG:boto:Token: None 2022-09-09T22:33:06.388 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:06 GMT /jkcgbj-12/myobj 2022-09-09T22:33:06.389 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:fioI/u3KwyFngKXiL3g90iLCcII= 2022-09-09T22:33:06.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:fioI/u3KwyFngKXiL3g90iLCcII=', 'Content-Length': '0'} 2022-09-09T22:33:06.394 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007427c9b1efb9ee6-00631bbf22-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:06.396 DEBUG:boto:path=/jkcgbj-12/myobj 2022-09-09T22:33:06.398 DEBUG:boto:auth_path=/jkcgbj-12/myobj 2022-09-09T22:33:06.399 DEBUG:boto:Method: HEAD 2022-09-09T22:33:06.400 DEBUG:boto:Path: /jkcgbj-12/myobj 2022-09-09T22:33:06.401 DEBUG:boto:Data: 2022-09-09T22:33:06.402 DEBUG:boto:Headers: {} 2022-09-09T22:33:06.404 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:06.404 DEBUG:boto:Port: 8001 2022-09-09T22:33:06.406 DEBUG:boto:Params: {} 2022-09-09T22:33:06.407 DEBUG:boto:Token: None 2022-09-09T22:33:06.408 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:06 GMT /jkcgbj-12/myobj 2022-09-09T22:33:06.409 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:fioI/u3KwyFngKXiL3g90iLCcII= 2022-09-09T22:33:06.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:fioI/u3KwyFngKXiL3g90iLCcII=', 'Content-Length': '0'} 2022-09-09T22:33:06.414 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f4ee9be4e6e6b1b-00631bbf22-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:06.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:33:06.417 DEBUG:boto:path=/jkcgbj-12/myobj 2022-09-09T22:33:06.418 DEBUG:boto:auth_path=/jkcgbj-12/myobj 2022-09-09T22:33:06.419 DEBUG:boto:Method: GET 2022-09-09T22:33:06.420 DEBUG:boto:Path: /jkcgbj-12/myobj 2022-09-09T22:33:06.421 DEBUG:boto:Data: 2022-09-09T22:33:06.422 DEBUG:boto:Headers: {} 2022-09-09T22:33:06.423 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:06.424 DEBUG:boto:Port: 8000 2022-09-09T22:33:06.425 DEBUG:boto:Params: {} 2022-09-09T22:33:06.426 DEBUG:boto:Token: None 2022-09-09T22:33:06.428 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:06 GMT /jkcgbj-12/myobj 2022-09-09T22:33:06.429 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:yz31ni0GoV8DwO3tuF81arYheEY= 2022-09-09T22:33:06.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:yz31ni0GoV8DwO3tuF81arYheEY=', 'Content-Length': '0'} 2022-09-09T22:33:06.434 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f46897662bb8c251-00631bbf22-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:06.474 DEBUG:boto:path=/jkcgbj-12/myobj 2022-09-09T22:33:06.476 DEBUG:boto:auth_path=/jkcgbj-12/myobj 2022-09-09T22:33:06.477 DEBUG:boto:Method: GET 2022-09-09T22:33:06.479 DEBUG:boto:Path: /jkcgbj-12/myobj 2022-09-09T22:33:06.480 DEBUG:boto:Data: 2022-09-09T22:33:06.481 DEBUG:boto:Headers: {} 2022-09-09T22:33:06.482 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:06.483 DEBUG:boto:Port: 8001 2022-09-09T22:33:06.484 DEBUG:boto:Params: {} 2022-09-09T22:33:06.485 DEBUG:boto:Token: None 2022-09-09T22:33:06.486 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:06 GMT /jkcgbj-12/myobj 2022-09-09T22:33:06.487 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:yz31ni0GoV8DwO3tuF81arYheEY= 2022-09-09T22:33:06.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:yz31ni0GoV8DwO3tuF81arYheEY=', 'Content-Length': '0'} 2022-09-09T22:33:06.492 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:30:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd923690be331a9e-00631bbf22-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:06.492 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:06.493 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:06.494 DEBUG:boto:path=/jkcgbj-12/?prefix=myobj 2022-09-09T22:33:06.495 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=myobj 2022-09-09T22:33:06.495 DEBUG:boto:Method: GET 2022-09-09T22:33:06.496 DEBUG:boto:Path: /jkcgbj-12/?prefix=myobj 2022-09-09T22:33:06.497 DEBUG:boto:Data: 2022-09-09T22:33:06.498 DEBUG:boto:Headers: {} 2022-09-09T22:33:06.499 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:06.500 DEBUG:boto:Port: 8000 2022-09-09T22:33:06.501 DEBUG:boto:Params: {} 2022-09-09T22:33:06.502 DEBUG:boto:Token: None 2022-09-09T22:33:06.503 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:06 GMT /jkcgbj-12/ 2022-09-09T22:33:06.504 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8dOXd09iM9R5mhTyhOrCKLe7Mx4= 2022-09-09T22:33:06.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8dOXd09iM9R5mhTyhOrCKLe7Mx4=', 'Content-Length': '0'} 2022-09-09T22:33:06.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c73258c357fcce2d-00631bbf22-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:06.513 DEBUG:boto:b'jkcgbj-12myobj1000falsemyobj2022-09-09T22:30:44.625Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:06.515 DEBUG:boto:path=/jkcgbj-12/ 2022-09-09T22:33:06.516 DEBUG:boto:auth_path=/jkcgbj-12/ 2022-09-09T22:33:06.517 DEBUG:boto:path=/jkcgbj-12/?prefix=myobj 2022-09-09T22:33:06.517 DEBUG:boto:auth_path=/jkcgbj-12/?prefix=myobj 2022-09-09T22:33:06.519 DEBUG:boto:Method: GET 2022-09-09T22:33:06.520 DEBUG:boto:Path: /jkcgbj-12/?prefix=myobj 2022-09-09T22:33:06.520 DEBUG:boto:Data: 2022-09-09T22:33:06.521 DEBUG:boto:Headers: {} 2022-09-09T22:33:06.522 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:06.523 DEBUG:boto:Port: 8001 2022-09-09T22:33:06.524 DEBUG:boto:Params: {} 2022-09-09T22:33:06.526 DEBUG:boto:Token: None 2022-09-09T22:33:06.527 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:06 GMT /jkcgbj-12/ 2022-09-09T22:33:06.529 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8dOXd09iM9R5mhTyhOrCKLe7Mx4= 2022-09-09T22:33:06.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8dOXd09iM9R5mhTyhOrCKLe7Mx4=', 'Content-Length': '0'} 2022-09-09T22:33:06.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be19fe1dc08f9edb-00631bbf22-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:06.578 DEBUG:boto:b'jkcgbj-12myobj1000falsemyobj2022-09-09T22:30:44.625Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:06.580 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-12 zones={test-zone2, test-zone3} 2022-09-09T22:33:06.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2022-09-09T22:33:06.585 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jkcgbj-13 2022-09-09T22:33:06.586 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:33:06.587 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:33:06.588 DEBUG:boto:Method: PUT 2022-09-09T22:33:06.589 DEBUG:boto:Path: /jkcgbj-13/ 2022-09-09T22:33:06.590 DEBUG:boto:Data: 2022-09-09T22:33:06.591 DEBUG:boto:Headers: {} 2022-09-09T22:33:06.592 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:06.593 DEBUG:boto:Port: 8000 2022-09-09T22:33:06.594 DEBUG:boto:Params: {} 2022-09-09T22:33:06.595 DEBUG:boto:Token: None 2022-09-09T22:33:06.596 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:33:06 GMT /jkcgbj-13/ 2022-09-09T22:33:06.597 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Qcm1zKfzAPZqaIx+ciBFlyWReRw= 2022-09-09T22:33:06.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Qcm1zKfzAPZqaIx+ciBFlyWReRw=', 'Content-Length': '0'} 2022-09-09T22:33:06.624 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009bc83a527dbb7eae-00631bbf22-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:06.625 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jkcgbj-14 2022-09-09T22:33:06.625 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:33:06.626 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:33:06.626 DEBUG:boto:Method: PUT 2022-09-09T22:33:06.627 DEBUG:boto:Path: /jkcgbj-14/ 2022-09-09T22:33:06.627 DEBUG:boto:Data: 2022-09-09T22:33:06.628 DEBUG:boto:Headers: {} 2022-09-09T22:33:06.628 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:06.629 DEBUG:boto:Port: 8000 2022-09-09T22:33:06.630 DEBUG:boto:Params: {} 2022-09-09T22:33:06.631 DEBUG:boto:Token: None 2022-09-09T22:33:06.632 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:33:06 GMT /jkcgbj-14/ 2022-09-09T22:33:06.632 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TBaHlQQ4pJ097PMWqUbNtK00B5M= 2022-09-09T22:33:06.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TBaHlQQ4pJ097PMWqUbNtK00B5M=', 'Content-Length': '0'} 2022-09-09T22:33:06.682 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b162a4e040e54d89-00631bbf22-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:06.684 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jkcgbj-15 2022-09-09T22:33:06.685 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:33:06.686 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:33:06.687 DEBUG:boto:Method: PUT 2022-09-09T22:33:06.688 DEBUG:boto:Path: /jkcgbj-15/ 2022-09-09T22:33:06.689 DEBUG:boto:Data: 2022-09-09T22:33:06.689 DEBUG:boto:Headers: {} 2022-09-09T22:33:06.690 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:06.691 DEBUG:boto:Port: 8001 2022-09-09T22:33:06.692 DEBUG:boto:Params: {} 2022-09-09T22:33:06.692 DEBUG:boto:Token: None 2022-09-09T22:33:06.693 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:33:06 GMT /jkcgbj-15/ 2022-09-09T22:33:06.694 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:B8q2rs4fPhJwWsOJUylQmJVESl0= 2022-09-09T22:33:06.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:B8q2rs4fPhJwWsOJUylQmJVESl0=', 'Content-Length': '0'} 2022-09-09T22:33:06.733 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005adb2a9d6a2a4dd1-00631bbf22-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:06.734 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:33:06.735 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:33:06.738 DEBUG:boto:Method: HEAD 2022-09-09T22:33:06.739 DEBUG:boto:Path: /jkcgbj-13/ 2022-09-09T22:33:06.739 DEBUG:boto:Data: 2022-09-09T22:33:06.740 DEBUG:boto:Headers: {} 2022-09-09T22:33:06.740 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:06.741 DEBUG:boto:Port: 8000 2022-09-09T22:33:06.742 DEBUG:boto:Params: {} 2022-09-09T22:33:06.742 DEBUG:boto:Token: None 2022-09-09T22:33:06.743 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:06 GMT /jkcgbj-13/ 2022-09-09T22:33:06.744 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jWJCQRIx5gaHkC3xze+fgmt14OU= 2022-09-09T22:33:06.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jWJCQRIx5gaHkC3xze+fgmt14OU=', 'Content-Length': '0'} 2022-09-09T22:33:06.749 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a9b1fcd2a1d58799-00631bbf22-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:06.750 DEBUG:boto:path=/jkcgbj-13/myobj 2022-09-09T22:33:06.750 DEBUG:boto:auth_path=/jkcgbj-13/myobj 2022-09-09T22:33:06.751 DEBUG:boto:Method: PUT 2022-09-09T22:33:06.751 DEBUG:boto:Path: /jkcgbj-13/myobj 2022-09-09T22:33:06.752 DEBUG:boto:Data: 2022-09-09T22:33:06.753 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-09T22:33:06.753 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:06.754 DEBUG:boto:Port: 8000 2022-09-09T22:33:06.754 DEBUG:boto:Params: {} 2022-09-09T22:33:06.755 DEBUG:boto:Token: None 2022-09-09T22:33:06.756 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 09 Sep 2022 22:33:06 GMT /jkcgbj-13/myobj 2022-09-09T22:33:06.757 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:/ybp9dO2O/4za1vI7diyW5vftdY= 2022-09-09T22:33:06.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:33:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:/ybp9dO2O/4za1vI7diyW5vftdY='} 2022-09-09T22:33:06.768 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000008f7833f3fb5780e-00631bbf22-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:06.770 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:33:06.771 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:33:06.772 DEBUG:boto:Method: HEAD 2022-09-09T22:33:06.773 DEBUG:boto:Path: /jkcgbj-14/ 2022-09-09T22:33:06.774 DEBUG:boto:Data: 2022-09-09T22:33:06.775 DEBUG:boto:Headers: {} 2022-09-09T22:33:06.775 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:06.776 DEBUG:boto:Port: 8000 2022-09-09T22:33:06.777 DEBUG:boto:Params: {} 2022-09-09T22:33:06.778 DEBUG:boto:Token: None 2022-09-09T22:33:06.779 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:06 GMT /jkcgbj-14/ 2022-09-09T22:33:06.780 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:iWXK6DaD+5otak5Wtguxz2YwgP8= 2022-09-09T22:33:06.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:iWXK6DaD+5otak5Wtguxz2YwgP8=', 'Content-Length': '0'} 2022-09-09T22:33:06.786 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7877b31e136fbe5-00631bbf22-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:06.788 DEBUG:boto:path=/jkcgbj-14/myobj 2022-09-09T22:33:06.788 DEBUG:boto:auth_path=/jkcgbj-14/myobj 2022-09-09T22:33:06.789 DEBUG:boto:Method: PUT 2022-09-09T22:33:06.789 DEBUG:boto:Path: /jkcgbj-14/myobj 2022-09-09T22:33:06.790 DEBUG:boto:Data: 2022-09-09T22:33:06.790 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-09T22:33:06.791 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:06.791 DEBUG:boto:Port: 8000 2022-09-09T22:33:06.792 DEBUG:boto:Params: {} 2022-09-09T22:33:06.793 DEBUG:boto:Token: None 2022-09-09T22:33:06.794 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 09 Sep 2022 22:33:06 GMT /jkcgbj-14/myobj 2022-09-09T22:33:06.794 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jEyrwOqRO9EnBvarxe5Z4uhUvNs= 2022-09-09T22:33:06.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:33:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jEyrwOqRO9EnBvarxe5Z4uhUvNs='} 2022-09-09T22:33:06.812 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b484b62617b9ccb-00631bbf22-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:06.813 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:33:06.814 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:33:06.815 DEBUG:boto:Method: HEAD 2022-09-09T22:33:06.815 DEBUG:boto:Path: /jkcgbj-15/ 2022-09-09T22:33:06.816 DEBUG:boto:Data: 2022-09-09T22:33:06.816 DEBUG:boto:Headers: {} 2022-09-09T22:33:06.817 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:06.818 DEBUG:boto:Port: 8001 2022-09-09T22:33:06.818 DEBUG:boto:Params: {} 2022-09-09T22:33:06.819 DEBUG:boto:Token: None 2022-09-09T22:33:06.820 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:06 GMT /jkcgbj-15/ 2022-09-09T22:33:06.820 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OVMVA+ewy4Z/W0AU79W/URM82IY= 2022-09-09T22:33:06.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OVMVA+ewy4Z/W0AU79W/URM82IY=', 'Content-Length': '0'} 2022-09-09T22:33:06.826 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000808a26fbd9e4c9e-00631bbf22-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:06.829 DEBUG:boto:path=/jkcgbj-15/myobj 2022-09-09T22:33:06.830 DEBUG:boto:auth_path=/jkcgbj-15/myobj 2022-09-09T22:33:06.831 DEBUG:boto:Method: PUT 2022-09-09T22:33:06.831 DEBUG:boto:Path: /jkcgbj-15/myobj 2022-09-09T22:33:06.832 DEBUG:boto:Data: 2022-09-09T22:33:06.833 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-09T22:33:06.834 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:06.835 DEBUG:boto:Port: 8001 2022-09-09T22:33:06.835 DEBUG:boto:Params: {} 2022-09-09T22:33:06.836 DEBUG:boto:Token: None 2022-09-09T22:33:06.837 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 09 Sep 2022 22:33:06 GMT /jkcgbj-15/myobj 2022-09-09T22:33:06.837 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:icpJTyD0vlgziLIV8oVKsdUCRmk= 2022-09-09T22:33:06.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:33:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:icpJTyD0vlgziLIV8oVKsdUCRmk='} 2022-09-09T22:33:06.856 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb80b39a2f1f19be-00631bbf22-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:06.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:33:06.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:33:06.861 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:33:06.913 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:33:06.914 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:33:06.963 INFO:teuthology.orchestra.run.smithi005.stderr:No --period given, using current period=503b1d61-4513-48c8-a308-17b76219610c 2022-09-09T22:33:06.990 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T22:33:06.991 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:33:06.991 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:33:06.992 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:28:06.146147Z" 2022-09-09T22:33:06.992 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:33:06.993 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:33:06.993 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:33:06.993 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:33:06.994 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:33:06.994 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:33:06.995 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:33:06.995 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:30:37.757014Z" 2022-09-09T22:33:06.996 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:33:06.997 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:33:06.998 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762786.718887_59.1", 2022-09-09T22:33:06.999 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:33:06.718887Z" 2022-09-09T22:33:06.999 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:33:06.999 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T22:33:07.001 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662762786.718887_59.1'} 2022-09-09T22:33:07.002 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-09T22:33:07.002 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:33:07.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:33:07.004 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:33:07.050 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:33:07.050 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:33:07.118 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:33:07.112+0000 7f1836957dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:33:07.119 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:33:07.116+0000 7f1836957dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:33:07.155 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:33:07.155 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:33:07.156 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:33:07.156 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:33:07.157 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:33:07.157 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:33:07.158 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:33:07.158 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:33:07.159 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:33:07.159 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:33:07.160 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:33:07.160 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:33:07.160 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:33:07.161 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:33:07.161 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:33:07.162 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:33:07.162 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:33:07.163 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:33:07.163 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:33:07.164 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:33:07.164 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:33:07.165 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:33:07.165 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:33:07.165 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:33:07.166 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:33:07.166 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:33:07.167 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:33:07.167 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:33:07.168 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:33:07.168 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:33:07.169 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:33:07.169 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:33:07.170 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:33:07.170 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:33:07.170 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:33:07.174 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:33:07.174 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:33:07.174 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:33:07.175 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:33:07.175 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:33:07.175 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:33:07.176 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:33:07.176 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:33:07.176 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:33:07.177 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:33:07.177 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:33:07.177 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:33:07.177 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:33:07.178 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:33:07.178 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762786.718887_59.1", 2022-09-09T22:33:07.178 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:33:07.179 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:33:07.179 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:33:07.179 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:33:06.718887Z", 2022-09-09T22:33:07.179 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:33:07.180 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:33:07.180 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:33:07.180 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:33:07.181 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:33:07.181 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:33:07.181 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:33:07.186 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:33:07.186 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:33:07.187 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:33:07Z" 2022-09-09T22:33:07.187 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:33:07.188 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662762786.718887_59.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:33:06.718887Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:33:07Z" } 2022-09-09T22:33:07.188 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662762786.718887_59.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:33:06.718887Z', 'realm_epoch': 2}}] 2022-09-09T22:33:07.189 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662762786.718887_59.1'} 2022-09-09T22:33:07.189 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662762786.718887_59.1'} 2022-09-09T22:33:07.190 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-09T22:33:07.190 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-09T22:33:07.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:33:07.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:33:07.194 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:33:07.242 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:33:07.242 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:33:07.295 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:33:07.291+0000 7fe6fcea3dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:33:07.300 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:33:07.295+0000 7fe6fcea3dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:33:07.322 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:33:07.323 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T22:33:07.324 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T22:33:07.324 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T22:33:07.325 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T22:33:07.326 INFO:teuthology.orchestra.run.smithi005.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:33:07.327 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:33:07.328 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:33:07.329 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T22:33:07.329 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:33:07.330 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T22:33:07.331 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:33:07.331 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:33:07.332 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:33:07.332 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:33:07.333 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 1, 2022-09-09T22:33:07.333 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:33:07.334 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:33:07.335 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:33:07.335 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:33:07.336 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:33:07.337 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:33:07.338 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T22:33:07.339 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:33:07.340 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:33:07.340 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:33:07.341 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:33:07.341 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:33:07.342 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:33:07.342 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:33:07.343 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:33:07.343 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:33:07.344 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:33:07.344 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:33:07.345 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T22:33:07.345 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:33:07.346 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:33:07.346 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:33:07.347 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:33:07.347 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:33:07.348 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:33:07.349 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:33:07.349 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:33:07.350 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:33:07.350 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:33:07.351 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:33:07.351 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T22:33:07.352 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:33:07.352 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:33:07.353 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762786.718887_59.1", 2022-09-09T22:33:07.353 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:33:07.354 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:33:07.354 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:33:07.355 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:33:06.718887Z", 2022-09-09T22:33:07.355 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:33:07.356 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:33:07.356 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:33:07.357 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:33:07.357 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:33:07.358 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T22:33:07.358 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 1, 2022-09-09T22:33:07.359 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 1 2022-09-09T22:33:07.360 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:33:07.360 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T22:33:07Z" 2022-09-09T22:33:07.361 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:33:07.363 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662762786.718887_59.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:33:06.718887Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:33:07Z" } 2022-09-09T22:33:07.364 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662762786.718887_59.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:33:06.718887Z', 'realm_epoch': 2}}] 2022-09-09T22:33:07.365 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662762786.718887_59.1'} 2022-09-09T22:33:07.366 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662762786.718887_59.1'} 2022-09-09T22:33:07.367 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-09T22:33:07.368 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-09T22:33:07.369 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:33:07.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:33:07.370 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:33:07.418 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:33:07.419 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:33:07.473 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:33:07.468+0000 7f2b9d388dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:33:07.475 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:33:07.468+0000 7f2b9d388dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:33:07.502 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:33:07.502 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:33:07.502 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:33:07.503 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:33:07.503 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:33:07.503 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:33:07.504 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:33:07.504 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:33:07.504 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:33:07.504 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:33:07.505 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:33:07.505 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:33:07.506 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:33:07.506 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:33:07.506 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:33:07.507 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:33:07.507 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:33:07.507 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:33:07.507 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:33:07.508 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:33:07.508 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:33:07.508 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:33:07.509 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:33:07.509 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:33:07.510 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:33:07.510 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:33:07.511 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:33:07.511 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:33:07.511 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:33:07.512 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:33:07.512 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:33:07.513 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:33:07.513 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:33:07.514 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:33:07.514 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:33:07.515 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:33:07.515 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:33:07.516 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:33:07.516 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:33:07.516 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:33:07.516 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:33:07.517 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:33:07.517 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:33:07.518 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:33:07.518 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:33:07.518 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:33:07.518 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:33:07.519 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:33:07.519 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:33:07.520 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762786.718887_59.1", 2022-09-09T22:33:07.520 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:33:07.521 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:33:07.521 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:33:07.521 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:33:06.718887Z", 2022-09-09T22:33:07.522 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:33:07.522 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:33:07.522 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:33:07.522 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:33:07.523 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:33:07.523 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:33:07.523 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:33:07.523 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:33:07.524 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:33:07.524 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:33:07Z" 2022-09-09T22:33:07.524 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:33:07.525 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662762786.718887_59.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:33:06.718887Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:33:07Z" } 2022-09-09T22:33:07.526 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662762786.718887_59.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:33:06.718887Z', 'realm_epoch': 2}}] 2022-09-09T22:33:07.527 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662762786.718887_59.1'} 2022-09-09T22:33:07.527 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662762786.718887_59.1'} 2022-09-09T22:33:07.528 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-09T22:33:07.528 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:33:07.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:33:07.530 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:33:07.582 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:33:07.583 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:33:07.672 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:33:07.668+0000 7f8fd6306dc0 1 waiting to reach incremental sync.. 2022-09-09T22:33:12.673 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:33:12.668+0000 7f8fd6306dc0 1 waiting to reach incremental sync.. 2022-09-09T22:33:17.674 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:33:17.667+0000 7f8fd6306dc0 1 waiting to reach incremental sync.. 2022-09-09T22:33:22.679 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:33:22.675+0000 7f8fd6306dc0 1 bucket sync caught up with source: 2022-09-09T22:33:22.680 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000001.4.5, , , , , , , , , ] 2022-09-09T22:33:22.680 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000001.4.5, , , , , , , , , ] 2022-09-09T22:33:22.680 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:33:22.675+0000 7f8fd6306dc0 0 bucket checkpoint complete 2022-09-09T22:33:22.707 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-13 zones={test-zone2, test-zone1} 2022-09-09T22:33:22.708 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:33:22.708 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:33:22.709 DEBUG:boto:Method: HEAD 2022-09-09T22:33:22.710 DEBUG:boto:Path: /jkcgbj-13/ 2022-09-09T22:33:22.710 DEBUG:boto:Data: 2022-09-09T22:33:22.711 DEBUG:boto:Headers: {} 2022-09-09T22:33:22.711 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:22.712 DEBUG:boto:Port: 8000 2022-09-09T22:33:22.712 DEBUG:boto:Params: {} 2022-09-09T22:33:22.713 DEBUG:boto:Token: None 2022-09-09T22:33:22.713 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:22 GMT /jkcgbj-13/ 2022-09-09T22:33:22.715 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:DaYZzRDQ9ykF7qePltG668AqrMY= 2022-09-09T22:33:22.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:22 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:DaYZzRDQ9ykF7qePltG668AqrMY=', 'Content-Length': '0'} 2022-09-09T22:33:22.726 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a16cdbb35dfcdb4-00631bbf32-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:33:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:22.727 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:33:22.728 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:33:22.728 DEBUG:boto:Method: HEAD 2022-09-09T22:33:22.729 DEBUG:boto:Path: /jkcgbj-13/ 2022-09-09T22:33:22.730 DEBUG:boto:Data: 2022-09-09T22:33:22.730 DEBUG:boto:Headers: {} 2022-09-09T22:33:22.731 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:22.732 DEBUG:boto:Port: 8000 2022-09-09T22:33:22.732 DEBUG:boto:Params: {} 2022-09-09T22:33:22.733 DEBUG:boto:Token: None 2022-09-09T22:33:22.733 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:22 GMT /jkcgbj-13/ 2022-09-09T22:33:22.734 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:DaYZzRDQ9ykF7qePltG668AqrMY= 2022-09-09T22:33:22.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:22 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:DaYZzRDQ9ykF7qePltG668AqrMY=', 'Content-Length': '0'} 2022-09-09T22:33:22.741 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077e231eae6ef7224-00631bbf32-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:33:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:22.742 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:33:22.743 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:33:22.743 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:33:22.744 DEBUG:boto:path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:33:22.744 DEBUG:boto:auth_path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:33:22.745 DEBUG:boto:Method: GET 2022-09-09T22:33:22.746 DEBUG:boto:Path: /jkcgbj-13/?versions&max-keys=999 2022-09-09T22:33:22.747 DEBUG:boto:Data: 2022-09-09T22:33:22.747 DEBUG:boto:Headers: {} 2022-09-09T22:33:22.748 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:22.748 DEBUG:boto:Port: 8000 2022-09-09T22:33:22.749 DEBUG:boto:Params: {} 2022-09-09T22:33:22.750 DEBUG:boto:Token: None 2022-09-09T22:33:22.751 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:22 GMT /jkcgbj-13/?versions 2022-09-09T22:33:22.751 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:hMlwwqRdERMWFScfM7pGj8bRLKo= 2022-09-09T22:33:22.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:22 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:hMlwwqRdERMWFScfM7pGj8bRLKo=', 'Content-Length': '0'} 2022-09-09T22:33:22.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068c4162ca008e211-00631bbf32-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:22.801 DEBUG:boto:b'jkcgbj-13999falsemyobjnulltrue2022-09-09T22:33:06.761Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:22.803 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-09T22:33:22.805 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:33:22.807 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:33:22.808 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:33:22.809 DEBUG:boto:path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:33:22.810 DEBUG:boto:auth_path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:33:22.811 DEBUG:boto:Method: GET 2022-09-09T22:33:22.812 DEBUG:boto:Path: /jkcgbj-13/?versions&max-keys=999 2022-09-09T22:33:22.813 DEBUG:boto:Data: 2022-09-09T22:33:22.815 DEBUG:boto:Headers: {} 2022-09-09T22:33:22.816 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:22.817 DEBUG:boto:Port: 8000 2022-09-09T22:33:22.818 DEBUG:boto:Params: {} 2022-09-09T22:33:22.819 DEBUG:boto:Token: None 2022-09-09T22:33:22.821 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:22 GMT /jkcgbj-13/?versions 2022-09-09T22:33:22.822 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:hMlwwqRdERMWFScfM7pGj8bRLKo= 2022-09-09T22:33:22.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:22 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:hMlwwqRdERMWFScfM7pGj8bRLKo=', 'Content-Length': '0'} 2022-09-09T22:33:22.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f7f968d04ed665e-00631bbf32-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:22.845 DEBUG:boto:b'jkcgbj-13999falsemyobjnulltrue2022-09-09T22:33:06.761Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:22.847 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-09T22:33:22.848 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:33:22.848 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:33:22.849 DEBUG:boto:path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:33:22.849 DEBUG:boto:auth_path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:33:22.850 DEBUG:boto:Method: GET 2022-09-09T22:33:22.850 DEBUG:boto:Path: /jkcgbj-13/?versions&max-keys=999 2022-09-09T22:33:22.851 DEBUG:boto:Data: 2022-09-09T22:33:22.851 DEBUG:boto:Headers: {} 2022-09-09T22:33:22.853 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:22.853 DEBUG:boto:Port: 8000 2022-09-09T22:33:22.854 DEBUG:boto:Params: {} 2022-09-09T22:33:22.854 DEBUG:boto:Token: None 2022-09-09T22:33:22.855 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:22 GMT /jkcgbj-13/?versions 2022-09-09T22:33:22.856 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:hMlwwqRdERMWFScfM7pGj8bRLKo= 2022-09-09T22:33:22.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:22 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:hMlwwqRdERMWFScfM7pGj8bRLKo=', 'Content-Length': '0'} 2022-09-09T22:33:22.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3a205d5b5d4a53f-00631bbf32-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:22.863 DEBUG:boto:b'jkcgbj-13999falsemyobjnulltrue2022-09-09T22:33:06.761Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:22.863 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:33:22.864 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:33:22.865 DEBUG:boto:path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:33:22.865 DEBUG:boto:auth_path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:33:22.866 DEBUG:boto:Method: GET 2022-09-09T22:33:22.866 DEBUG:boto:Path: /jkcgbj-13/?versions&max-keys=999 2022-09-09T22:33:22.867 DEBUG:boto:Data: 2022-09-09T22:33:22.867 DEBUG:boto:Headers: {} 2022-09-09T22:33:22.868 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:22.868 DEBUG:boto:Port: 8000 2022-09-09T22:33:22.869 DEBUG:boto:Params: {} 2022-09-09T22:33:22.869 DEBUG:boto:Token: None 2022-09-09T22:33:22.872 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:22 GMT /jkcgbj-13/?versions 2022-09-09T22:33:22.873 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:hMlwwqRdERMWFScfM7pGj8bRLKo= 2022-09-09T22:33:22.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:22 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:hMlwwqRdERMWFScfM7pGj8bRLKo=', 'Content-Length': '0'} 2022-09-09T22:33:22.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061cba8e3fc4d2a76-00631bbf32-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:22.922 DEBUG:boto:b'jkcgbj-13999falsemyobjnulltrue2022-09-09T22:33:06.761Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:22.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:33:22.928 DEBUG:boto:path=/jkcgbj-13/myobj 2022-09-09T22:33:22.929 DEBUG:boto:auth_path=/jkcgbj-13/myobj 2022-09-09T22:33:22.929 DEBUG:boto:path=/jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:22.930 DEBUG:boto:auth_path=/jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:22.930 DEBUG:boto:Method: GET 2022-09-09T22:33:22.931 DEBUG:boto:Path: /jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:22.931 DEBUG:boto:Data: 2022-09-09T22:33:22.932 DEBUG:boto:Headers: {} 2022-09-09T22:33:22.932 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:22.933 DEBUG:boto:Port: 8000 2022-09-09T22:33:22.933 DEBUG:boto:Params: {} 2022-09-09T22:33:22.934 DEBUG:boto:Token: None 2022-09-09T22:33:22.935 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:22 GMT /jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:22.935 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1STp6WkuK1Xz/Iz0p/pLUrSxM/U= 2022-09-09T22:33:22.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:22 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1STp6WkuK1Xz/Iz0p/pLUrSxM/U=', 'Content-Length': '0'} 2022-09-09T22:33:22.939 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd6f50eab898746e-00631bbf32-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:22.941 DEBUG:boto:path=/jkcgbj-13/myobj 2022-09-09T22:33:22.942 DEBUG:boto:auth_path=/jkcgbj-13/myobj 2022-09-09T22:33:22.943 DEBUG:boto:path=/jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:22.943 DEBUG:boto:auth_path=/jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:22.944 DEBUG:boto:Method: GET 2022-09-09T22:33:22.945 DEBUG:boto:Path: /jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:22.946 DEBUG:boto:Data: 2022-09-09T22:33:22.947 DEBUG:boto:Headers: {} 2022-09-09T22:33:22.948 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:22.949 DEBUG:boto:Port: 8000 2022-09-09T22:33:22.949 DEBUG:boto:Params: {} 2022-09-09T22:33:22.950 DEBUG:boto:Token: None 2022-09-09T22:33:22.951 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:22 GMT /jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:22.952 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1STp6WkuK1Xz/Iz0p/pLUrSxM/U= 2022-09-09T22:33:22.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:22 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1STp6WkuK1Xz/Iz0p/pLUrSxM/U=', 'Content-Length': '0'} 2022-09-09T22:33:22.958 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e2e9d3073e991b7-00631bbf32-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:22.959 DEBUG:boto:path=/jkcgbj-13/myobj 2022-09-09T22:33:22.960 DEBUG:boto:auth_path=/jkcgbj-13/myobj 2022-09-09T22:33:22.960 DEBUG:boto:path=/jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:22.961 DEBUG:boto:auth_path=/jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:22.962 DEBUG:boto:Method: HEAD 2022-09-09T22:33:22.962 DEBUG:boto:Path: /jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:22.963 DEBUG:boto:Data: 2022-09-09T22:33:22.963 DEBUG:boto:Headers: {} 2022-09-09T22:33:22.964 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:22.964 DEBUG:boto:Port: 8000 2022-09-09T22:33:22.965 DEBUG:boto:Params: {} 2022-09-09T22:33:22.966 DEBUG:boto:Token: None 2022-09-09T22:33:22.966 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:22 GMT /jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:22.967 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bjoPxKLcFjnjQM1zW0WWE6kN5bo= 2022-09-09T22:33:22.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:22 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bjoPxKLcFjnjQM1zW0WWE6kN5bo=', 'Content-Length': '0'} 2022-09-09T22:33:22.971 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001bf8c946e4e9e1dd-00631bbf32-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:22.972 DEBUG:boto:path=/jkcgbj-13/myobj 2022-09-09T22:33:22.972 DEBUG:boto:auth_path=/jkcgbj-13/myobj 2022-09-09T22:33:22.973 DEBUG:boto:path=/jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:22.973 DEBUG:boto:auth_path=/jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:22.974 DEBUG:boto:Method: HEAD 2022-09-09T22:33:22.974 DEBUG:boto:Path: /jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:22.975 DEBUG:boto:Data: 2022-09-09T22:33:22.975 DEBUG:boto:Headers: {} 2022-09-09T22:33:22.976 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:22.976 DEBUG:boto:Port: 8000 2022-09-09T22:33:22.977 DEBUG:boto:Params: {} 2022-09-09T22:33:22.977 DEBUG:boto:Token: None 2022-09-09T22:33:22.978 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:22 GMT /jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:22.979 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bjoPxKLcFjnjQM1zW0WWE6kN5bo= 2022-09-09T22:33:22.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:22 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bjoPxKLcFjnjQM1zW0WWE6kN5bo=', 'Content-Length': '0'} 2022-09-09T22:33:22.984 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae0e7ddbba9a73ac-00631bbf32-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:22.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:33:22.987 DEBUG:boto:path=/jkcgbj-13/myobj 2022-09-09T22:33:22.988 DEBUG:boto:auth_path=/jkcgbj-13/myobj 2022-09-09T22:33:22.988 DEBUG:boto:path=/jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:22.989 DEBUG:boto:auth_path=/jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:22.990 DEBUG:boto:Method: GET 2022-09-09T22:33:22.991 DEBUG:boto:Path: /jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:22.991 DEBUG:boto:Data: 2022-09-09T22:33:22.992 DEBUG:boto:Headers: {} 2022-09-09T22:33:22.992 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:22.993 DEBUG:boto:Port: 8000 2022-09-09T22:33:22.994 DEBUG:boto:Params: {} 2022-09-09T22:33:23.002 DEBUG:boto:Token: None 2022-09-09T22:33:23.003 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:23 GMT /jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:23.004 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2Yp8IawPGfvomWS2uMOaUHHpJNQ= 2022-09-09T22:33:23.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2Yp8IawPGfvomWS2uMOaUHHpJNQ=', 'Content-Length': '0'} 2022-09-09T22:33:23.009 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032804323bb6f8b81-00631bbf33-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:23.011 DEBUG:boto:path=/jkcgbj-13/myobj 2022-09-09T22:33:23.012 DEBUG:boto:auth_path=/jkcgbj-13/myobj 2022-09-09T22:33:23.013 DEBUG:boto:path=/jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:23.014 DEBUG:boto:auth_path=/jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:23.015 DEBUG:boto:Method: GET 2022-09-09T22:33:23.016 DEBUG:boto:Path: /jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:23.016 DEBUG:boto:Data: 2022-09-09T22:33:23.017 DEBUG:boto:Headers: {} 2022-09-09T22:33:23.018 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:23.019 DEBUG:boto:Port: 8000 2022-09-09T22:33:23.020 DEBUG:boto:Params: {} 2022-09-09T22:33:23.021 DEBUG:boto:Token: None 2022-09-09T22:33:23.022 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:23 GMT /jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:23.023 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2Yp8IawPGfvomWS2uMOaUHHpJNQ= 2022-09-09T22:33:23.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2Yp8IawPGfvomWS2uMOaUHHpJNQ=', 'Content-Length': '0'} 2022-09-09T22:33:23.028 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000478b0007cf3328a5-00631bbf33-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:23.029 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:33:23.029 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:33:23.030 DEBUG:boto:path=/jkcgbj-13/?prefix=myobj 2022-09-09T22:33:23.030 DEBUG:boto:auth_path=/jkcgbj-13/?prefix=myobj 2022-09-09T22:33:23.031 DEBUG:boto:Method: GET 2022-09-09T22:33:23.031 DEBUG:boto:Path: /jkcgbj-13/?prefix=myobj 2022-09-09T22:33:23.032 DEBUG:boto:Data: 2022-09-09T22:33:23.032 DEBUG:boto:Headers: {} 2022-09-09T22:33:23.033 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:23.033 DEBUG:boto:Port: 8000 2022-09-09T22:33:23.034 DEBUG:boto:Params: {} 2022-09-09T22:33:23.035 DEBUG:boto:Token: None 2022-09-09T22:33:23.035 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:23 GMT /jkcgbj-13/ 2022-09-09T22:33:23.036 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:AqJ7BZYi5G4+bKHWmDxay6HdUIs= 2022-09-09T22:33:23.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:AqJ7BZYi5G4+bKHWmDxay6HdUIs=', 'Content-Length': '0'} 2022-09-09T22:33:23.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074df173ff164389f-00631bbf33-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:23.043 DEBUG:boto:b'jkcgbj-13myobj1000falsemyobj2022-09-09T22:33:06.761Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:23.044 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:33:23.045 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:33:23.046 DEBUG:boto:path=/jkcgbj-13/?prefix=myobj 2022-09-09T22:33:23.047 DEBUG:boto:auth_path=/jkcgbj-13/?prefix=myobj 2022-09-09T22:33:23.047 DEBUG:boto:Method: GET 2022-09-09T22:33:23.048 DEBUG:boto:Path: /jkcgbj-13/?prefix=myobj 2022-09-09T22:33:23.049 DEBUG:boto:Data: 2022-09-09T22:33:23.049 DEBUG:boto:Headers: {} 2022-09-09T22:33:23.050 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:23.051 DEBUG:boto:Port: 8000 2022-09-09T22:33:23.051 DEBUG:boto:Params: {} 2022-09-09T22:33:23.052 DEBUG:boto:Token: None 2022-09-09T22:33:23.052 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:23 GMT /jkcgbj-13/ 2022-09-09T22:33:23.053 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:AqJ7BZYi5G4+bKHWmDxay6HdUIs= 2022-09-09T22:33:23.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:AqJ7BZYi5G4+bKHWmDxay6HdUIs=', 'Content-Length': '0'} 2022-09-09T22:33:23.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c5d31408324eef2-00631bbf33-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:23.106 DEBUG:boto:b'jkcgbj-13myobj1000falsemyobj2022-09-09T22:33:06.761Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:23.107 DEBUG:boto:path=/jkcgbj-13/myobj 2022-09-09T22:33:23.108 DEBUG:boto:auth_path=/jkcgbj-13/myobj 2022-09-09T22:33:23.108 DEBUG:boto:Method: HEAD 2022-09-09T22:33:23.109 DEBUG:boto:Path: /jkcgbj-13/myobj 2022-09-09T22:33:23.109 DEBUG:boto:Data: 2022-09-09T22:33:23.110 DEBUG:boto:Headers: {} 2022-09-09T22:33:23.116 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:23.117 DEBUG:boto:Port: 8000 2022-09-09T22:33:23.119 DEBUG:boto:Params: {} 2022-09-09T22:33:23.122 DEBUG:boto:Token: None 2022-09-09T22:33:23.123 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:23 GMT /jkcgbj-13/myobj 2022-09-09T22:33:23.124 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UVj2zeqCriXnGAWf41VD3MxWOF0= 2022-09-09T22:33:23.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UVj2zeqCriXnGAWf41VD3MxWOF0=', 'Content-Length': '0'} 2022-09-09T22:33:23.130 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec6b450ebb5c2654-00631bbf33-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:23.132 DEBUG:boto:path=/jkcgbj-13/myobj 2022-09-09T22:33:23.133 DEBUG:boto:auth_path=/jkcgbj-13/myobj 2022-09-09T22:33:23.134 DEBUG:boto:Method: HEAD 2022-09-09T22:33:23.136 DEBUG:boto:Path: /jkcgbj-13/myobj 2022-09-09T22:33:23.136 DEBUG:boto:Data: 2022-09-09T22:33:23.137 DEBUG:boto:Headers: {} 2022-09-09T22:33:23.137 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:23.138 DEBUG:boto:Port: 8000 2022-09-09T22:33:23.138 DEBUG:boto:Params: {} 2022-09-09T22:33:23.139 DEBUG:boto:Token: None 2022-09-09T22:33:23.139 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:23 GMT /jkcgbj-13/myobj 2022-09-09T22:33:23.140 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UVj2zeqCriXnGAWf41VD3MxWOF0= 2022-09-09T22:33:23.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UVj2zeqCriXnGAWf41VD3MxWOF0=', 'Content-Length': '0'} 2022-09-09T22:33:23.146 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004abe753f910eb61e-00631bbf33-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:23.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:33:23.148 DEBUG:boto:path=/jkcgbj-13/myobj 2022-09-09T22:33:23.148 DEBUG:boto:auth_path=/jkcgbj-13/myobj 2022-09-09T22:33:23.149 DEBUG:boto:Method: GET 2022-09-09T22:33:23.150 DEBUG:boto:Path: /jkcgbj-13/myobj 2022-09-09T22:33:23.151 DEBUG:boto:Data: 2022-09-09T22:33:23.152 DEBUG:boto:Headers: {} 2022-09-09T22:33:23.153 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:23.154 DEBUG:boto:Port: 8000 2022-09-09T22:33:23.155 DEBUG:boto:Params: {} 2022-09-09T22:33:23.157 DEBUG:boto:Token: None 2022-09-09T22:33:23.158 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:23 GMT /jkcgbj-13/myobj 2022-09-09T22:33:23.159 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Nqft5Def8FDjaG034DnU87EP1Vw= 2022-09-09T22:33:23.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Nqft5Def8FDjaG034DnU87EP1Vw=', 'Content-Length': '0'} 2022-09-09T22:33:23.168 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe7ef58a885cf25d-00631bbf33-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:23.169 DEBUG:boto:path=/jkcgbj-13/myobj 2022-09-09T22:33:23.170 DEBUG:boto:auth_path=/jkcgbj-13/myobj 2022-09-09T22:33:23.171 DEBUG:boto:Method: GET 2022-09-09T22:33:23.171 DEBUG:boto:Path: /jkcgbj-13/myobj 2022-09-09T22:33:23.172 DEBUG:boto:Data: 2022-09-09T22:33:23.175 DEBUG:boto:Headers: {} 2022-09-09T22:33:23.177 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:23.178 DEBUG:boto:Port: 8000 2022-09-09T22:33:23.179 DEBUG:boto:Params: {} 2022-09-09T22:33:23.180 DEBUG:boto:Token: None 2022-09-09T22:33:23.180 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:23 GMT /jkcgbj-13/myobj 2022-09-09T22:33:23.181 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Nqft5Def8FDjaG034DnU87EP1Vw= 2022-09-09T22:33:23.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Nqft5Def8FDjaG034DnU87EP1Vw=', 'Content-Length': '0'} 2022-09-09T22:33:23.186 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db10fb0f8364f22c-00631bbf33-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:23.187 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:33:23.189 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:33:23.189 DEBUG:boto:path=/jkcgbj-13/?prefix=myobj 2022-09-09T22:33:23.190 DEBUG:boto:auth_path=/jkcgbj-13/?prefix=myobj 2022-09-09T22:33:23.190 DEBUG:boto:Method: GET 2022-09-09T22:33:23.191 DEBUG:boto:Path: /jkcgbj-13/?prefix=myobj 2022-09-09T22:33:23.192 DEBUG:boto:Data: 2022-09-09T22:33:23.192 DEBUG:boto:Headers: {} 2022-09-09T22:33:23.193 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:23.194 DEBUG:boto:Port: 8000 2022-09-09T22:33:23.194 DEBUG:boto:Params: {} 2022-09-09T22:33:23.195 DEBUG:boto:Token: None 2022-09-09T22:33:23.196 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:23 GMT /jkcgbj-13/ 2022-09-09T22:33:23.197 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:AqJ7BZYi5G4+bKHWmDxay6HdUIs= 2022-09-09T22:33:23.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:AqJ7BZYi5G4+bKHWmDxay6HdUIs=', 'Content-Length': '0'} 2022-09-09T22:33:23.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb1057596f6fe008-00631bbf33-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:23.254 DEBUG:boto:b'jkcgbj-13myobj1000falsemyobj2022-09-09T22:33:06.761Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:23.256 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:33:23.257 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:33:23.257 DEBUG:boto:path=/jkcgbj-13/?prefix=myobj 2022-09-09T22:33:23.258 DEBUG:boto:auth_path=/jkcgbj-13/?prefix=myobj 2022-09-09T22:33:23.259 DEBUG:boto:Method: GET 2022-09-09T22:33:23.259 DEBUG:boto:Path: /jkcgbj-13/?prefix=myobj 2022-09-09T22:33:23.264 DEBUG:boto:Data: 2022-09-09T22:33:23.265 DEBUG:boto:Headers: {} 2022-09-09T22:33:23.265 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:23.266 DEBUG:boto:Port: 8000 2022-09-09T22:33:23.267 DEBUG:boto:Params: {} 2022-09-09T22:33:23.267 DEBUG:boto:Token: None 2022-09-09T22:33:23.268 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:23 GMT /jkcgbj-13/ 2022-09-09T22:33:23.268 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:AqJ7BZYi5G4+bKHWmDxay6HdUIs= 2022-09-09T22:33:23.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:AqJ7BZYi5G4+bKHWmDxay6HdUIs=', 'Content-Length': '0'} 2022-09-09T22:33:23.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ac81b2b43bf8a9b-00631bbf33-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:23.277 DEBUG:boto:b'jkcgbj-13myobj1000falsemyobj2022-09-09T22:33:06.761Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:23.280 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-13 zones={test-zone2, test-zone1} 2022-09-09T22:33:23.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:33:23.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:33:23.284 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:33:23.341 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:33:23.341 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:33:23.413 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:33:23.402+0000 7fa4ba9b6dc0 1 bucket sync caught up with source: 2022-09-09T22:33:23.413 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000001.4.5, , , , , , , , , ] 2022-09-09T22:33:23.414 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000001.4.5, , , , , , , , , ] 2022-09-09T22:33:23.415 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:33:23.402+0000 7fa4ba9b6dc0 0 bucket checkpoint complete 2022-09-09T22:33:23.435 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-13 zones={test-zone3, test-zone1} 2022-09-09T22:33:23.437 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:33:23.438 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:33:23.439 DEBUG:boto:Method: HEAD 2022-09-09T22:33:23.440 DEBUG:boto:Path: /jkcgbj-13/ 2022-09-09T22:33:23.441 DEBUG:boto:Data: 2022-09-09T22:33:23.442 DEBUG:boto:Headers: {} 2022-09-09T22:33:23.444 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:23.445 DEBUG:boto:Port: 8001 2022-09-09T22:33:23.446 DEBUG:boto:Params: {} 2022-09-09T22:33:23.447 DEBUG:boto:Token: None 2022-09-09T22:33:23.448 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:23 GMT /jkcgbj-13/ 2022-09-09T22:33:23.450 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:41Ac+XYoTccMflYtMJYLiv+RF+w= 2022-09-09T22:33:23.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:41Ac+XYoTccMflYtMJYLiv+RF+w=', 'Content-Length': '0'} 2022-09-09T22:33:23.457 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e98ea2620056d808-00631bbf33-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:33:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:23.458 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:33:23.458 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:33:23.459 DEBUG:boto:Method: HEAD 2022-09-09T22:33:23.460 DEBUG:boto:Path: /jkcgbj-13/ 2022-09-09T22:33:23.460 DEBUG:boto:Data: 2022-09-09T22:33:23.461 DEBUG:boto:Headers: {} 2022-09-09T22:33:23.461 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:23.462 DEBUG:boto:Port: 8000 2022-09-09T22:33:23.462 DEBUG:boto:Params: {} 2022-09-09T22:33:23.463 DEBUG:boto:Token: None 2022-09-09T22:33:23.464 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:23 GMT /jkcgbj-13/ 2022-09-09T22:33:23.465 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:41Ac+XYoTccMflYtMJYLiv+RF+w= 2022-09-09T22:33:23.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:41Ac+XYoTccMflYtMJYLiv+RF+w=', 'Content-Length': '0'} 2022-09-09T22:33:23.470 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000059083fea7518db34-00631bbf33-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:33:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:23.471 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:33:23.472 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:33:23.472 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:33:23.473 DEBUG:boto:path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:33:23.473 DEBUG:boto:auth_path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:33:23.474 DEBUG:boto:Method: GET 2022-09-09T22:33:23.482 DEBUG:boto:Path: /jkcgbj-13/?versions&max-keys=999 2022-09-09T22:33:23.483 DEBUG:boto:Data: 2022-09-09T22:33:23.484 DEBUG:boto:Headers: {} 2022-09-09T22:33:23.486 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:23.487 DEBUG:boto:Port: 8001 2022-09-09T22:33:23.488 DEBUG:boto:Params: {} 2022-09-09T22:33:23.490 DEBUG:boto:Token: None 2022-09-09T22:33:23.492 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:23 GMT /jkcgbj-13/?versions 2022-09-09T22:33:23.493 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:aGpcCI2jkrbIgW1z2q1lqCYCFNc= 2022-09-09T22:33:23.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:aGpcCI2jkrbIgW1z2q1lqCYCFNc=', 'Content-Length': '0'} 2022-09-09T22:33:23.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020af935b4b28ce20-00631bbf33-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:23.502 DEBUG:boto:b'jkcgbj-13999falsemyobjnulltrue2022-09-09T22:33:06.761Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:23.504 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-09T22:33:23.505 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:33:23.506 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:33:23.507 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:33:23.508 DEBUG:boto:path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:33:23.510 DEBUG:boto:auth_path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:33:23.511 DEBUG:boto:Method: GET 2022-09-09T22:33:23.512 DEBUG:boto:Path: /jkcgbj-13/?versions&max-keys=999 2022-09-09T22:33:23.513 DEBUG:boto:Data: 2022-09-09T22:33:23.514 DEBUG:boto:Headers: {} 2022-09-09T22:33:23.515 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:23.516 DEBUG:boto:Port: 8000 2022-09-09T22:33:23.517 DEBUG:boto:Params: {} 2022-09-09T22:33:23.518 DEBUG:boto:Token: None 2022-09-09T22:33:23.519 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:23 GMT /jkcgbj-13/?versions 2022-09-09T22:33:23.521 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:aGpcCI2jkrbIgW1z2q1lqCYCFNc= 2022-09-09T22:33:23.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:aGpcCI2jkrbIgW1z2q1lqCYCFNc=', 'Content-Length': '0'} 2022-09-09T22:33:23.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031068b66053a3311-00631bbf33-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:23.530 DEBUG:boto:b'jkcgbj-13999falsemyobjnulltrue2022-09-09T22:33:06.761Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:23.531 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-09T22:33:23.532 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:33:23.532 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:33:23.533 DEBUG:boto:path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:33:23.533 DEBUG:boto:auth_path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:33:23.534 DEBUG:boto:Method: GET 2022-09-09T22:33:23.534 DEBUG:boto:Path: /jkcgbj-13/?versions&max-keys=999 2022-09-09T22:33:23.536 DEBUG:boto:Data: 2022-09-09T22:33:23.536 DEBUG:boto:Headers: {} 2022-09-09T22:33:23.539 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:23.542 DEBUG:boto:Port: 8001 2022-09-09T22:33:23.543 DEBUG:boto:Params: {} 2022-09-09T22:33:23.544 DEBUG:boto:Token: None 2022-09-09T22:33:23.544 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:23 GMT /jkcgbj-13/?versions 2022-09-09T22:33:23.545 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:aGpcCI2jkrbIgW1z2q1lqCYCFNc= 2022-09-09T22:33:23.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:aGpcCI2jkrbIgW1z2q1lqCYCFNc=', 'Content-Length': '0'} 2022-09-09T22:33:23.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e7574b6e8adc2ca-00631bbf33-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:23.562 DEBUG:boto:b'jkcgbj-13999falsemyobjnulltrue2022-09-09T22:33:06.761Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:23.565 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:33:23.567 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:33:23.568 DEBUG:boto:path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:33:23.569 DEBUG:boto:auth_path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:33:23.569 DEBUG:boto:Method: GET 2022-09-09T22:33:23.570 DEBUG:boto:Path: /jkcgbj-13/?versions&max-keys=999 2022-09-09T22:33:23.571 DEBUG:boto:Data: 2022-09-09T22:33:23.571 DEBUG:boto:Headers: {} 2022-09-09T22:33:23.572 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:23.573 DEBUG:boto:Port: 8000 2022-09-09T22:33:23.573 DEBUG:boto:Params: {} 2022-09-09T22:33:23.574 DEBUG:boto:Token: None 2022-09-09T22:33:23.575 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:23 GMT /jkcgbj-13/?versions 2022-09-09T22:33:23.576 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:aGpcCI2jkrbIgW1z2q1lqCYCFNc= 2022-09-09T22:33:23.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:aGpcCI2jkrbIgW1z2q1lqCYCFNc=', 'Content-Length': '0'} 2022-09-09T22:33:23.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e47b781b3a45b5a-00631bbf33-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:23.584 DEBUG:boto:b'jkcgbj-13999falsemyobjnulltrue2022-09-09T22:33:06.761Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:23.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:33:23.586 DEBUG:boto:path=/jkcgbj-13/myobj 2022-09-09T22:33:23.587 DEBUG:boto:auth_path=/jkcgbj-13/myobj 2022-09-09T22:33:23.588 DEBUG:boto:path=/jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:23.589 DEBUG:boto:auth_path=/jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:23.591 DEBUG:boto:Method: GET 2022-09-09T22:33:23.592 DEBUG:boto:Path: /jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:23.593 DEBUG:boto:Data: 2022-09-09T22:33:23.594 DEBUG:boto:Headers: {} 2022-09-09T22:33:23.594 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:23.595 DEBUG:boto:Port: 8001 2022-09-09T22:33:23.595 DEBUG:boto:Params: {} 2022-09-09T22:33:23.596 DEBUG:boto:Token: None 2022-09-09T22:33:23.597 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:23 GMT /jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:23.598 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2Yp8IawPGfvomWS2uMOaUHHpJNQ= 2022-09-09T22:33:23.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2Yp8IawPGfvomWS2uMOaUHHpJNQ=', 'Content-Length': '0'} 2022-09-09T22:33:23.602 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000808892f70ab12fc8-00631bbf33-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:23.603 DEBUG:boto:path=/jkcgbj-13/myobj 2022-09-09T22:33:23.604 DEBUG:boto:auth_path=/jkcgbj-13/myobj 2022-09-09T22:33:23.604 DEBUG:boto:path=/jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:23.605 DEBUG:boto:auth_path=/jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:23.605 DEBUG:boto:Method: GET 2022-09-09T22:33:23.606 DEBUG:boto:Path: /jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:23.606 DEBUG:boto:Data: 2022-09-09T22:33:23.607 DEBUG:boto:Headers: {} 2022-09-09T22:33:23.608 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:23.608 DEBUG:boto:Port: 8000 2022-09-09T22:33:23.609 DEBUG:boto:Params: {} 2022-09-09T22:33:23.616 DEBUG:boto:Token: None 2022-09-09T22:33:23.618 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:23 GMT /jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:23.618 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2Yp8IawPGfvomWS2uMOaUHHpJNQ= 2022-09-09T22:33:23.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2Yp8IawPGfvomWS2uMOaUHHpJNQ=', 'Content-Length': '0'} 2022-09-09T22:33:23.628 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b902ccb9e9962d5-00631bbf33-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:23.629 DEBUG:boto:path=/jkcgbj-13/myobj 2022-09-09T22:33:23.629 DEBUG:boto:auth_path=/jkcgbj-13/myobj 2022-09-09T22:33:23.630 DEBUG:boto:path=/jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:23.631 DEBUG:boto:auth_path=/jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:23.631 DEBUG:boto:Method: HEAD 2022-09-09T22:33:23.632 DEBUG:boto:Path: /jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:23.632 DEBUG:boto:Data: 2022-09-09T22:33:23.633 DEBUG:boto:Headers: {} 2022-09-09T22:33:23.634 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:23.635 DEBUG:boto:Port: 8001 2022-09-09T22:33:23.635 DEBUG:boto:Params: {} 2022-09-09T22:33:23.636 DEBUG:boto:Token: None 2022-09-09T22:33:23.637 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:23 GMT /jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:23.638 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Qc1PvJhEJ52hBaVb1t/1FuYUd4E= 2022-09-09T22:33:23.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Qc1PvJhEJ52hBaVb1t/1FuYUd4E=', 'Content-Length': '0'} 2022-09-09T22:33:23.643 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005921b77884a3145a-00631bbf33-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:23.645 DEBUG:boto:path=/jkcgbj-13/myobj 2022-09-09T22:33:23.647 DEBUG:boto:auth_path=/jkcgbj-13/myobj 2022-09-09T22:33:23.648 DEBUG:boto:path=/jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:23.649 DEBUG:boto:auth_path=/jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:23.650 DEBUG:boto:Method: HEAD 2022-09-09T22:33:23.651 DEBUG:boto:Path: /jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:23.652 DEBUG:boto:Data: 2022-09-09T22:33:23.653 DEBUG:boto:Headers: {} 2022-09-09T22:33:23.654 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:23.655 DEBUG:boto:Port: 8000 2022-09-09T22:33:23.656 DEBUG:boto:Params: {} 2022-09-09T22:33:23.657 DEBUG:boto:Token: None 2022-09-09T22:33:23.658 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:23 GMT /jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:23.659 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Qc1PvJhEJ52hBaVb1t/1FuYUd4E= 2022-09-09T22:33:23.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Qc1PvJhEJ52hBaVb1t/1FuYUd4E=', 'Content-Length': '0'} 2022-09-09T22:33:23.666 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064356f654f749139-00631bbf33-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:23.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:33:23.668 DEBUG:boto:path=/jkcgbj-13/myobj 2022-09-09T22:33:23.669 DEBUG:boto:auth_path=/jkcgbj-13/myobj 2022-09-09T22:33:23.670 DEBUG:boto:path=/jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:23.672 DEBUG:boto:auth_path=/jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:23.673 DEBUG:boto:Method: GET 2022-09-09T22:33:23.674 DEBUG:boto:Path: /jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:23.675 DEBUG:boto:Data: 2022-09-09T22:33:23.676 DEBUG:boto:Headers: {} 2022-09-09T22:33:23.677 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:23.678 DEBUG:boto:Port: 8001 2022-09-09T22:33:23.678 DEBUG:boto:Params: {} 2022-09-09T22:33:23.679 DEBUG:boto:Token: None 2022-09-09T22:33:23.680 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:23 GMT /jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:23.680 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2Yp8IawPGfvomWS2uMOaUHHpJNQ= 2022-09-09T22:33:23.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2Yp8IawPGfvomWS2uMOaUHHpJNQ=', 'Content-Length': '0'} 2022-09-09T22:33:23.684 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f2f04a1f2f671549-00631bbf33-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:23.686 DEBUG:boto:path=/jkcgbj-13/myobj 2022-09-09T22:33:23.687 DEBUG:boto:auth_path=/jkcgbj-13/myobj 2022-09-09T22:33:23.688 DEBUG:boto:path=/jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:23.689 DEBUG:boto:auth_path=/jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:23.690 DEBUG:boto:Method: GET 2022-09-09T22:33:23.691 DEBUG:boto:Path: /jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:23.693 DEBUG:boto:Data: 2022-09-09T22:33:23.694 DEBUG:boto:Headers: {} 2022-09-09T22:33:23.695 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:23.696 DEBUG:boto:Port: 8000 2022-09-09T22:33:23.697 DEBUG:boto:Params: {} 2022-09-09T22:33:23.698 DEBUG:boto:Token: None 2022-09-09T22:33:23.699 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:23 GMT /jkcgbj-13/myobj?versionId=null 2022-09-09T22:33:23.700 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2Yp8IawPGfvomWS2uMOaUHHpJNQ= 2022-09-09T22:33:23.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2Yp8IawPGfvomWS2uMOaUHHpJNQ=', 'Content-Length': '0'} 2022-09-09T22:33:23.705 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000528691583747a854-00631bbf33-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:23.747 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:33:23.748 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:33:23.749 DEBUG:boto:path=/jkcgbj-13/?prefix=myobj 2022-09-09T22:33:23.750 DEBUG:boto:auth_path=/jkcgbj-13/?prefix=myobj 2022-09-09T22:33:23.751 DEBUG:boto:Method: GET 2022-09-09T22:33:23.752 DEBUG:boto:Path: /jkcgbj-13/?prefix=myobj 2022-09-09T22:33:23.753 DEBUG:boto:Data: 2022-09-09T22:33:23.754 DEBUG:boto:Headers: {} 2022-09-09T22:33:23.755 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:23.756 DEBUG:boto:Port: 8001 2022-09-09T22:33:23.757 DEBUG:boto:Params: {} 2022-09-09T22:33:23.758 DEBUG:boto:Token: None 2022-09-09T22:33:23.759 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:23 GMT /jkcgbj-13/ 2022-09-09T22:33:23.761 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:AqJ7BZYi5G4+bKHWmDxay6HdUIs= 2022-09-09T22:33:23.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:AqJ7BZYi5G4+bKHWmDxay6HdUIs=', 'Content-Length': '0'} 2022-09-09T22:33:23.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e786c087e17cbd49-00631bbf33-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:23.770 DEBUG:boto:b'jkcgbj-13myobj1000falsemyobj2022-09-09T22:33:06.761Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:23.784 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:33:23.785 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:33:23.786 DEBUG:boto:path=/jkcgbj-13/?prefix=myobj 2022-09-09T22:33:23.787 DEBUG:boto:auth_path=/jkcgbj-13/?prefix=myobj 2022-09-09T22:33:23.788 DEBUG:boto:Method: GET 2022-09-09T22:33:23.789 DEBUG:boto:Path: /jkcgbj-13/?prefix=myobj 2022-09-09T22:33:23.790 DEBUG:boto:Data: 2022-09-09T22:33:23.792 DEBUG:boto:Headers: {} 2022-09-09T22:33:23.793 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:23.794 DEBUG:boto:Port: 8000 2022-09-09T22:33:23.795 DEBUG:boto:Params: {} 2022-09-09T22:33:23.796 DEBUG:boto:Token: None 2022-09-09T22:33:23.797 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:23 GMT /jkcgbj-13/ 2022-09-09T22:33:23.798 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:AqJ7BZYi5G4+bKHWmDxay6HdUIs= 2022-09-09T22:33:23.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:AqJ7BZYi5G4+bKHWmDxay6HdUIs=', 'Content-Length': '0'} 2022-09-09T22:33:23.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000810dad40cd9e660f-00631bbf33-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:23.808 DEBUG:boto:b'jkcgbj-13myobj1000falsemyobj2022-09-09T22:33:06.761Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:23.810 DEBUG:boto:path=/jkcgbj-13/myobj 2022-09-09T22:33:23.811 DEBUG:boto:auth_path=/jkcgbj-13/myobj 2022-09-09T22:33:23.812 DEBUG:boto:Method: HEAD 2022-09-09T22:33:23.813 DEBUG:boto:Path: /jkcgbj-13/myobj 2022-09-09T22:33:23.814 DEBUG:boto:Data: 2022-09-09T22:33:23.815 DEBUG:boto:Headers: {} 2022-09-09T22:33:23.816 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:23.817 DEBUG:boto:Port: 8001 2022-09-09T22:33:23.818 DEBUG:boto:Params: {} 2022-09-09T22:33:23.819 DEBUG:boto:Token: None 2022-09-09T22:33:23.820 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:23 GMT /jkcgbj-13/myobj 2022-09-09T22:33:23.821 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UVj2zeqCriXnGAWf41VD3MxWOF0= 2022-09-09T22:33:23.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UVj2zeqCriXnGAWf41VD3MxWOF0=', 'Content-Length': '0'} 2022-09-09T22:33:23.825 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e501e288298346b-00631bbf33-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:23.826 DEBUG:boto:path=/jkcgbj-13/myobj 2022-09-09T22:33:23.826 DEBUG:boto:auth_path=/jkcgbj-13/myobj 2022-09-09T22:33:23.827 DEBUG:boto:Method: HEAD 2022-09-09T22:33:23.828 DEBUG:boto:Path: /jkcgbj-13/myobj 2022-09-09T22:33:23.828 DEBUG:boto:Data: 2022-09-09T22:33:23.829 DEBUG:boto:Headers: {} 2022-09-09T22:33:23.829 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:23.830 DEBUG:boto:Port: 8000 2022-09-09T22:33:23.831 DEBUG:boto:Params: {} 2022-09-09T22:33:23.832 DEBUG:boto:Token: None 2022-09-09T22:33:23.833 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:23 GMT /jkcgbj-13/myobj 2022-09-09T22:33:23.834 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UVj2zeqCriXnGAWf41VD3MxWOF0= 2022-09-09T22:33:23.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UVj2zeqCriXnGAWf41VD3MxWOF0=', 'Content-Length': '0'} 2022-09-09T22:33:23.838 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fcfb03c3137ff6a4-00631bbf33-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:23.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:33:23.839 DEBUG:boto:path=/jkcgbj-13/myobj 2022-09-09T22:33:23.840 DEBUG:boto:auth_path=/jkcgbj-13/myobj 2022-09-09T22:33:23.841 DEBUG:boto:Method: GET 2022-09-09T22:33:23.841 DEBUG:boto:Path: /jkcgbj-13/myobj 2022-09-09T22:33:23.842 DEBUG:boto:Data: 2022-09-09T22:33:23.842 DEBUG:boto:Headers: {} 2022-09-09T22:33:23.843 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:23.844 DEBUG:boto:Port: 8001 2022-09-09T22:33:23.844 DEBUG:boto:Params: {} 2022-09-09T22:33:23.845 DEBUG:boto:Token: None 2022-09-09T22:33:23.846 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:23 GMT /jkcgbj-13/myobj 2022-09-09T22:33:23.846 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Nqft5Def8FDjaG034DnU87EP1Vw= 2022-09-09T22:33:23.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Nqft5Def8FDjaG034DnU87EP1Vw=', 'Content-Length': '0'} 2022-09-09T22:33:23.850 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001151a06146aa9419-00631bbf33-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:23.851 DEBUG:boto:path=/jkcgbj-13/myobj 2022-09-09T22:33:23.852 DEBUG:boto:auth_path=/jkcgbj-13/myobj 2022-09-09T22:33:23.852 DEBUG:boto:Method: GET 2022-09-09T22:33:23.853 DEBUG:boto:Path: /jkcgbj-13/myobj 2022-09-09T22:33:23.854 DEBUG:boto:Data: 2022-09-09T22:33:23.854 DEBUG:boto:Headers: {} 2022-09-09T22:33:23.855 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:23.855 DEBUG:boto:Port: 8000 2022-09-09T22:33:23.856 DEBUG:boto:Params: {} 2022-09-09T22:33:23.856 DEBUG:boto:Token: None 2022-09-09T22:33:23.857 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:23 GMT /jkcgbj-13/myobj 2022-09-09T22:33:23.858 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Nqft5Def8FDjaG034DnU87EP1Vw= 2022-09-09T22:33:23.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Nqft5Def8FDjaG034DnU87EP1Vw=', 'Content-Length': '0'} 2022-09-09T22:33:23.862 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091d78bf874f10078-00631bbf33-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:23.863 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:33:23.863 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:33:23.864 DEBUG:boto:path=/jkcgbj-13/?prefix=myobj 2022-09-09T22:33:23.865 DEBUG:boto:auth_path=/jkcgbj-13/?prefix=myobj 2022-09-09T22:33:23.865 DEBUG:boto:Method: GET 2022-09-09T22:33:23.866 DEBUG:boto:Path: /jkcgbj-13/?prefix=myobj 2022-09-09T22:33:23.867 DEBUG:boto:Data: 2022-09-09T22:33:23.867 DEBUG:boto:Headers: {} 2022-09-09T22:33:23.868 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:23.868 DEBUG:boto:Port: 8001 2022-09-09T22:33:23.869 DEBUG:boto:Params: {} 2022-09-09T22:33:23.869 DEBUG:boto:Token: None 2022-09-09T22:33:23.870 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:23 GMT /jkcgbj-13/ 2022-09-09T22:33:23.871 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:AqJ7BZYi5G4+bKHWmDxay6HdUIs= 2022-09-09T22:33:23.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:AqJ7BZYi5G4+bKHWmDxay6HdUIs=', 'Content-Length': '0'} 2022-09-09T22:33:23.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029627b435b8bd9c2-00631bbf33-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:23.918 DEBUG:boto:b'jkcgbj-13myobj1000falsemyobj2022-09-09T22:33:06.761Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:23.920 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:33:23.920 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:33:23.921 DEBUG:boto:path=/jkcgbj-13/?prefix=myobj 2022-09-09T22:33:23.922 DEBUG:boto:auth_path=/jkcgbj-13/?prefix=myobj 2022-09-09T22:33:23.923 DEBUG:boto:Method: GET 2022-09-09T22:33:23.923 DEBUG:boto:Path: /jkcgbj-13/?prefix=myobj 2022-09-09T22:33:23.924 DEBUG:boto:Data: 2022-09-09T22:33:23.925 DEBUG:boto:Headers: {} 2022-09-09T22:33:23.925 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:23.926 DEBUG:boto:Port: 8000 2022-09-09T22:33:23.927 DEBUG:boto:Params: {} 2022-09-09T22:33:23.928 DEBUG:boto:Token: None 2022-09-09T22:33:23.929 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:23 GMT /jkcgbj-13/ 2022-09-09T22:33:23.930 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:AqJ7BZYi5G4+bKHWmDxay6HdUIs= 2022-09-09T22:33:23.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:AqJ7BZYi5G4+bKHWmDxay6HdUIs=', 'Content-Length': '0'} 2022-09-09T22:33:23.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000419626c4cb3f0c2b-00631bbf33-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:23.940 DEBUG:boto:b'jkcgbj-13myobj1000falsemyobj2022-09-09T22:33:06.761Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:23.942 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-13 zones={test-zone3, test-zone1} 2022-09-09T22:33:23.943 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:33:23.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:33:23.945 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:33:24.000 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:33:24.000 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:33:24.072 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:33:24.067+0000 7f78e9587dc0 1 waiting to reach incremental sync.. 2022-09-09T22:33:29.075 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:33:29.070+0000 7f78e9587dc0 1 bucket sync caught up with source: 2022-09-09T22:33:29.076 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000001.4.5, , , , , , , , , ] 2022-09-09T22:33:29.076 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000001.4.5, , , , , , , , , ] 2022-09-09T22:33:29.076 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:33:29.070+0000 7f78e9587dc0 0 bucket checkpoint complete 2022-09-09T22:33:29.102 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:33:29.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:33:29.105 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:33:29.161 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:33:29.161 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:33:29.228 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:33:29.222+0000 7fb94bc7edc0 1 bucket sync caught up with source: 2022-09-09T22:33:29.229 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000001.11.5, , , , , , , , , ] 2022-09-09T22:33:29.229 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000001.11.5, , , , , , , , , ] 2022-09-09T22:33:29.229 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:33:29.222+0000 7fb94bc7edc0 0 bucket checkpoint complete 2022-09-09T22:33:29.256 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-14 zones={test-zone1, test-zone2} 2022-09-09T22:33:29.258 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:33:29.258 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:33:29.259 DEBUG:boto:Method: HEAD 2022-09-09T22:33:29.260 DEBUG:boto:Path: /jkcgbj-14/ 2022-09-09T22:33:29.260 DEBUG:boto:Data: 2022-09-09T22:33:29.261 DEBUG:boto:Headers: {} 2022-09-09T22:33:29.261 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:29.262 DEBUG:boto:Port: 8000 2022-09-09T22:33:29.262 DEBUG:boto:Params: {} 2022-09-09T22:33:29.263 DEBUG:boto:Token: None 2022-09-09T22:33:29.264 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:29 GMT /jkcgbj-14/ 2022-09-09T22:33:29.264 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:t/8yb6/UtvwgqFoQxAxSjZhXp+w= 2022-09-09T22:33:29.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:t/8yb6/UtvwgqFoQxAxSjZhXp+w=', 'Content-Length': '0'} 2022-09-09T22:33:29.269 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b84bd71614ff3896-00631bbf39-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:33:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:29.270 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:33:29.271 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:33:29.271 DEBUG:boto:Method: HEAD 2022-09-09T22:33:29.272 DEBUG:boto:Path: /jkcgbj-14/ 2022-09-09T22:33:29.272 DEBUG:boto:Data: 2022-09-09T22:33:29.273 DEBUG:boto:Headers: {} 2022-09-09T22:33:29.273 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:29.274 DEBUG:boto:Port: 8000 2022-09-09T22:33:29.275 DEBUG:boto:Params: {} 2022-09-09T22:33:29.276 DEBUG:boto:Token: None 2022-09-09T22:33:29.276 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:29 GMT /jkcgbj-14/ 2022-09-09T22:33:29.277 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:t/8yb6/UtvwgqFoQxAxSjZhXp+w= 2022-09-09T22:33:29.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:t/8yb6/UtvwgqFoQxAxSjZhXp+w=', 'Content-Length': '0'} 2022-09-09T22:33:29.282 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002c1607e68469eca5-00631bbf39-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:33:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:29.283 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:33:29.283 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:33:29.284 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:33:29.284 DEBUG:boto:path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:33:29.285 DEBUG:boto:auth_path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:33:29.286 DEBUG:boto:Method: GET 2022-09-09T22:33:29.286 DEBUG:boto:Path: /jkcgbj-14/?versions&max-keys=999 2022-09-09T22:33:29.287 DEBUG:boto:Data: 2022-09-09T22:33:29.288 DEBUG:boto:Headers: {} 2022-09-09T22:33:29.288 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:29.289 DEBUG:boto:Port: 8000 2022-09-09T22:33:29.289 DEBUG:boto:Params: {} 2022-09-09T22:33:29.290 DEBUG:boto:Token: None 2022-09-09T22:33:29.291 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:29 GMT /jkcgbj-14/?versions 2022-09-09T22:33:29.291 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:98V15E7J+/YJrZpiK94lBvM9ACg= 2022-09-09T22:33:29.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:98V15E7J+/YJrZpiK94lBvM9ACg=', 'Content-Length': '0'} 2022-09-09T22:33:29.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5f78b5a85e1039f-00631bbf39-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:29.298 DEBUG:boto:b'jkcgbj-14999falsemyobjnulltrue2022-09-09T22:33:06.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:29.300 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-09T22:33:29.301 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:33:29.302 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:33:29.303 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:33:29.304 DEBUG:boto:path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:33:29.305 DEBUG:boto:auth_path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:33:29.306 DEBUG:boto:Method: GET 2022-09-09T22:33:29.307 DEBUG:boto:Path: /jkcgbj-14/?versions&max-keys=999 2022-09-09T22:33:29.308 DEBUG:boto:Data: 2022-09-09T22:33:29.309 DEBUG:boto:Headers: {} 2022-09-09T22:33:29.310 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:29.311 DEBUG:boto:Port: 8000 2022-09-09T22:33:29.312 DEBUG:boto:Params: {} 2022-09-09T22:33:29.313 DEBUG:boto:Token: None 2022-09-09T22:33:29.314 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:29 GMT /jkcgbj-14/?versions 2022-09-09T22:33:29.315 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:98V15E7J+/YJrZpiK94lBvM9ACg= 2022-09-09T22:33:29.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:98V15E7J+/YJrZpiK94lBvM9ACg=', 'Content-Length': '0'} 2022-09-09T22:33:29.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a779e38948ad9993-00631bbf39-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:29.325 DEBUG:boto:b'jkcgbj-14999falsemyobjnulltrue2022-09-09T22:33:06.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:29.326 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-09T22:33:29.327 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:33:29.328 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:33:29.329 DEBUG:boto:path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:33:29.330 DEBUG:boto:auth_path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:33:29.331 DEBUG:boto:Method: GET 2022-09-09T22:33:29.332 DEBUG:boto:Path: /jkcgbj-14/?versions&max-keys=999 2022-09-09T22:33:29.333 DEBUG:boto:Data: 2022-09-09T22:33:29.333 DEBUG:boto:Headers: {} 2022-09-09T22:33:29.334 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:29.335 DEBUG:boto:Port: 8000 2022-09-09T22:33:29.336 DEBUG:boto:Params: {} 2022-09-09T22:33:29.337 DEBUG:boto:Token: None 2022-09-09T22:33:29.338 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:29 GMT /jkcgbj-14/?versions 2022-09-09T22:33:29.339 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:98V15E7J+/YJrZpiK94lBvM9ACg= 2022-09-09T22:33:29.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:98V15E7J+/YJrZpiK94lBvM9ACg=', 'Content-Length': '0'} 2022-09-09T22:33:29.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0725db827079513-00631bbf39-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:29.347 DEBUG:boto:b'jkcgbj-14999falsemyobjnulltrue2022-09-09T22:33:06.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:29.348 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:33:29.349 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:33:29.350 DEBUG:boto:path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:33:29.351 DEBUG:boto:auth_path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:33:29.352 DEBUG:boto:Method: GET 2022-09-09T22:33:29.353 DEBUG:boto:Path: /jkcgbj-14/?versions&max-keys=999 2022-09-09T22:33:29.354 DEBUG:boto:Data: 2022-09-09T22:33:29.355 DEBUG:boto:Headers: {} 2022-09-09T22:33:29.356 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:29.356 DEBUG:boto:Port: 8000 2022-09-09T22:33:29.357 DEBUG:boto:Params: {} 2022-09-09T22:33:29.357 DEBUG:boto:Token: None 2022-09-09T22:33:29.358 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:29 GMT /jkcgbj-14/?versions 2022-09-09T22:33:29.358 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:98V15E7J+/YJrZpiK94lBvM9ACg= 2022-09-09T22:33:29.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:98V15E7J+/YJrZpiK94lBvM9ACg=', 'Content-Length': '0'} 2022-09-09T22:33:29.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037599c20daccc09c-00631bbf39-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:29.365 DEBUG:boto:b'jkcgbj-14999falsemyobjnulltrue2022-09-09T22:33:06.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:29.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:33:29.368 DEBUG:boto:path=/jkcgbj-14/myobj 2022-09-09T22:33:29.369 DEBUG:boto:auth_path=/jkcgbj-14/myobj 2022-09-09T22:33:29.370 DEBUG:boto:path=/jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:29.371 DEBUG:boto:auth_path=/jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:29.372 DEBUG:boto:Method: GET 2022-09-09T22:33:29.373 DEBUG:boto:Path: /jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:29.373 DEBUG:boto:Data: 2022-09-09T22:33:29.374 DEBUG:boto:Headers: {} 2022-09-09T22:33:29.375 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:29.375 DEBUG:boto:Port: 8000 2022-09-09T22:33:29.376 DEBUG:boto:Params: {} 2022-09-09T22:33:29.376 DEBUG:boto:Token: None 2022-09-09T22:33:29.377 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:29 GMT /jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:29.377 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sVwJoYPYLERAsfG7aEc3ecs+KpE= 2022-09-09T22:33:29.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sVwJoYPYLERAsfG7aEc3ecs+KpE=', 'Content-Length': '0'} 2022-09-09T22:33:29.382 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a8c72bd03500d46f-00631bbf39-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:29.422 DEBUG:boto:path=/jkcgbj-14/myobj 2022-09-09T22:33:29.423 DEBUG:boto:auth_path=/jkcgbj-14/myobj 2022-09-09T22:33:29.424 DEBUG:boto:path=/jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:29.425 DEBUG:boto:auth_path=/jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:29.425 DEBUG:boto:Method: GET 2022-09-09T22:33:29.426 DEBUG:boto:Path: /jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:29.427 DEBUG:boto:Data: 2022-09-09T22:33:29.427 DEBUG:boto:Headers: {} 2022-09-09T22:33:29.428 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:29.430 DEBUG:boto:Port: 8000 2022-09-09T22:33:29.430 DEBUG:boto:Params: {} 2022-09-09T22:33:29.432 DEBUG:boto:Token: None 2022-09-09T22:33:29.433 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:29 GMT /jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:29.434 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sVwJoYPYLERAsfG7aEc3ecs+KpE= 2022-09-09T22:33:29.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sVwJoYPYLERAsfG7aEc3ecs+KpE=', 'Content-Length': '0'} 2022-09-09T22:33:29.438 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8ff9431e2b9cb1d-00631bbf39-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:29.440 DEBUG:boto:path=/jkcgbj-14/myobj 2022-09-09T22:33:29.440 DEBUG:boto:auth_path=/jkcgbj-14/myobj 2022-09-09T22:33:29.441 DEBUG:boto:path=/jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:29.441 DEBUG:boto:auth_path=/jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:29.442 DEBUG:boto:Method: HEAD 2022-09-09T22:33:29.443 DEBUG:boto:Path: /jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:29.443 DEBUG:boto:Data: 2022-09-09T22:33:29.444 DEBUG:boto:Headers: {} 2022-09-09T22:33:29.445 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:29.445 DEBUG:boto:Port: 8000 2022-09-09T22:33:29.446 DEBUG:boto:Params: {} 2022-09-09T22:33:29.446 DEBUG:boto:Token: None 2022-09-09T22:33:29.447 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:29 GMT /jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:29.448 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0U/mYINlyMzFpo4eHHWSQI9T3p8= 2022-09-09T22:33:29.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0U/mYINlyMzFpo4eHHWSQI9T3p8=', 'Content-Length': '0'} 2022-09-09T22:33:29.452 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f8590c2d5afc9e87-00631bbf39-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:29.453 DEBUG:boto:path=/jkcgbj-14/myobj 2022-09-09T22:33:29.453 DEBUG:boto:auth_path=/jkcgbj-14/myobj 2022-09-09T22:33:29.454 DEBUG:boto:path=/jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:29.454 DEBUG:boto:auth_path=/jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:29.455 DEBUG:boto:Method: HEAD 2022-09-09T22:33:29.455 DEBUG:boto:Path: /jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:29.456 DEBUG:boto:Data: 2022-09-09T22:33:29.456 DEBUG:boto:Headers: {} 2022-09-09T22:33:29.457 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:29.457 DEBUG:boto:Port: 8000 2022-09-09T22:33:29.458 DEBUG:boto:Params: {} 2022-09-09T22:33:29.459 DEBUG:boto:Token: None 2022-09-09T22:33:29.459 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:29 GMT /jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:29.460 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0U/mYINlyMzFpo4eHHWSQI9T3p8= 2022-09-09T22:33:29.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0U/mYINlyMzFpo4eHHWSQI9T3p8=', 'Content-Length': '0'} 2022-09-09T22:33:29.466 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000019f0dd7940f58de1-00631bbf39-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:29.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:33:29.467 DEBUG:boto:path=/jkcgbj-14/myobj 2022-09-09T22:33:29.468 DEBUG:boto:auth_path=/jkcgbj-14/myobj 2022-09-09T22:33:29.469 DEBUG:boto:path=/jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:29.469 DEBUG:boto:auth_path=/jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:29.470 DEBUG:boto:Method: GET 2022-09-09T22:33:29.470 DEBUG:boto:Path: /jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:29.471 DEBUG:boto:Data: 2022-09-09T22:33:29.471 DEBUG:boto:Headers: {} 2022-09-09T22:33:29.472 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:29.472 DEBUG:boto:Port: 8000 2022-09-09T22:33:29.473 DEBUG:boto:Params: {} 2022-09-09T22:33:29.474 DEBUG:boto:Token: None 2022-09-09T22:33:29.474 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:29 GMT /jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:29.475 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sVwJoYPYLERAsfG7aEc3ecs+KpE= 2022-09-09T22:33:29.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sVwJoYPYLERAsfG7aEc3ecs+KpE=', 'Content-Length': '0'} 2022-09-09T22:33:29.479 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f47f217dbb0b1dc-00631bbf39-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:29.480 DEBUG:boto:path=/jkcgbj-14/myobj 2022-09-09T22:33:29.481 DEBUG:boto:auth_path=/jkcgbj-14/myobj 2022-09-09T22:33:29.482 DEBUG:boto:path=/jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:29.483 DEBUG:boto:auth_path=/jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:29.484 DEBUG:boto:Method: GET 2022-09-09T22:33:29.484 DEBUG:boto:Path: /jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:29.485 DEBUG:boto:Data: 2022-09-09T22:33:29.485 DEBUG:boto:Headers: {} 2022-09-09T22:33:29.486 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:29.488 DEBUG:boto:Port: 8000 2022-09-09T22:33:29.489 DEBUG:boto:Params: {} 2022-09-09T22:33:29.489 DEBUG:boto:Token: None 2022-09-09T22:33:29.490 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:29 GMT /jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:29.491 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sVwJoYPYLERAsfG7aEc3ecs+KpE= 2022-09-09T22:33:29.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sVwJoYPYLERAsfG7aEc3ecs+KpE=', 'Content-Length': '0'} 2022-09-09T22:33:29.495 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000fa4dc29e25173d6-00631bbf39-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:29.542 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:33:29.543 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:33:29.549 DEBUG:boto:path=/jkcgbj-14/?prefix=myobj 2022-09-09T22:33:29.549 DEBUG:boto:auth_path=/jkcgbj-14/?prefix=myobj 2022-09-09T22:33:29.550 DEBUG:boto:Method: GET 2022-09-09T22:33:29.551 DEBUG:boto:Path: /jkcgbj-14/?prefix=myobj 2022-09-09T22:33:29.551 DEBUG:boto:Data: 2022-09-09T22:33:29.552 DEBUG:boto:Headers: {} 2022-09-09T22:33:29.552 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:29.553 DEBUG:boto:Port: 8000 2022-09-09T22:33:29.553 DEBUG:boto:Params: {} 2022-09-09T22:33:29.554 DEBUG:boto:Token: None 2022-09-09T22:33:29.555 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:29 GMT /jkcgbj-14/ 2022-09-09T22:33:29.556 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:QGF5P59ahZdVf8MKHZy54nWeqvo= 2022-09-09T22:33:29.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:QGF5P59ahZdVf8MKHZy54nWeqvo=', 'Content-Length': '0'} 2022-09-09T22:33:29.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f25c8faf6c53d36-00631bbf39-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:29.564 DEBUG:boto:b'jkcgbj-14myobj1000falsemyobj2022-09-09T22:33:06.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:29.566 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:33:29.567 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:33:29.568 DEBUG:boto:path=/jkcgbj-14/?prefix=myobj 2022-09-09T22:33:29.569 DEBUG:boto:auth_path=/jkcgbj-14/?prefix=myobj 2022-09-09T22:33:29.570 DEBUG:boto:Method: GET 2022-09-09T22:33:29.571 DEBUG:boto:Path: /jkcgbj-14/?prefix=myobj 2022-09-09T22:33:29.572 DEBUG:boto:Data: 2022-09-09T22:33:29.573 DEBUG:boto:Headers: {} 2022-09-09T22:33:29.574 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:29.576 DEBUG:boto:Port: 8000 2022-09-09T22:33:29.577 DEBUG:boto:Params: {} 2022-09-09T22:33:29.578 DEBUG:boto:Token: None 2022-09-09T22:33:29.579 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:29 GMT /jkcgbj-14/ 2022-09-09T22:33:29.580 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:QGF5P59ahZdVf8MKHZy54nWeqvo= 2022-09-09T22:33:29.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:QGF5P59ahZdVf8MKHZy54nWeqvo=', 'Content-Length': '0'} 2022-09-09T22:33:29.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9dccd1fb6cd7716-00631bbf39-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:29.593 DEBUG:boto:b'jkcgbj-14myobj1000falsemyobj2022-09-09T22:33:06.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:29.594 DEBUG:boto:path=/jkcgbj-14/myobj 2022-09-09T22:33:29.595 DEBUG:boto:auth_path=/jkcgbj-14/myobj 2022-09-09T22:33:29.595 DEBUG:boto:Method: HEAD 2022-09-09T22:33:29.596 DEBUG:boto:Path: /jkcgbj-14/myobj 2022-09-09T22:33:29.596 DEBUG:boto:Data: 2022-09-09T22:33:29.597 DEBUG:boto:Headers: {} 2022-09-09T22:33:29.598 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:29.599 DEBUG:boto:Port: 8000 2022-09-09T22:33:29.601 DEBUG:boto:Params: {} 2022-09-09T22:33:29.602 DEBUG:boto:Token: None 2022-09-09T22:33:29.603 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:29 GMT /jkcgbj-14/myobj 2022-09-09T22:33:29.604 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:diD432/gyqBneouw6XsuevuWZSo= 2022-09-09T22:33:29.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:diD432/gyqBneouw6XsuevuWZSo=', 'Content-Length': '0'} 2022-09-09T22:33:29.609 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084cfc683b6a70048-00631bbf39-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:29.611 DEBUG:boto:path=/jkcgbj-14/myobj 2022-09-09T22:33:29.612 DEBUG:boto:auth_path=/jkcgbj-14/myobj 2022-09-09T22:33:29.613 DEBUG:boto:Method: HEAD 2022-09-09T22:33:29.613 DEBUG:boto:Path: /jkcgbj-14/myobj 2022-09-09T22:33:29.614 DEBUG:boto:Data: 2022-09-09T22:33:29.615 DEBUG:boto:Headers: {} 2022-09-09T22:33:29.616 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:29.617 DEBUG:boto:Port: 8000 2022-09-09T22:33:29.618 DEBUG:boto:Params: {} 2022-09-09T22:33:29.619 DEBUG:boto:Token: None 2022-09-09T22:33:29.620 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:29 GMT /jkcgbj-14/myobj 2022-09-09T22:33:29.621 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:diD432/gyqBneouw6XsuevuWZSo= 2022-09-09T22:33:29.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:diD432/gyqBneouw6XsuevuWZSo=', 'Content-Length': '0'} 2022-09-09T22:33:29.625 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a139f8f2c528c029-00631bbf39-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:29.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:33:29.627 DEBUG:boto:path=/jkcgbj-14/myobj 2022-09-09T22:33:29.627 DEBUG:boto:auth_path=/jkcgbj-14/myobj 2022-09-09T22:33:29.628 DEBUG:boto:Method: GET 2022-09-09T22:33:29.629 DEBUG:boto:Path: /jkcgbj-14/myobj 2022-09-09T22:33:29.629 DEBUG:boto:Data: 2022-09-09T22:33:29.630 DEBUG:boto:Headers: {} 2022-09-09T22:33:29.630 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:29.632 DEBUG:boto:Port: 8000 2022-09-09T22:33:29.633 DEBUG:boto:Params: {} 2022-09-09T22:33:29.634 DEBUG:boto:Token: None 2022-09-09T22:33:29.635 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:29 GMT /jkcgbj-14/myobj 2022-09-09T22:33:29.636 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Muzaln0LreFQpwsFciGUsUlst9E= 2022-09-09T22:33:29.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Muzaln0LreFQpwsFciGUsUlst9E=', 'Content-Length': '0'} 2022-09-09T22:33:29.643 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001290fa678e0bad1c-00631bbf39-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:29.645 DEBUG:boto:path=/jkcgbj-14/myobj 2022-09-09T22:33:29.646 DEBUG:boto:auth_path=/jkcgbj-14/myobj 2022-09-09T22:33:29.646 DEBUG:boto:Method: GET 2022-09-09T22:33:29.647 DEBUG:boto:Path: /jkcgbj-14/myobj 2022-09-09T22:33:29.648 DEBUG:boto:Data: 2022-09-09T22:33:29.648 DEBUG:boto:Headers: {} 2022-09-09T22:33:29.649 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:29.650 DEBUG:boto:Port: 8000 2022-09-09T22:33:29.651 DEBUG:boto:Params: {} 2022-09-09T22:33:29.652 DEBUG:boto:Token: None 2022-09-09T22:33:29.653 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:29 GMT /jkcgbj-14/myobj 2022-09-09T22:33:29.654 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Muzaln0LreFQpwsFciGUsUlst9E= 2022-09-09T22:33:29.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Muzaln0LreFQpwsFciGUsUlst9E=', 'Content-Length': '0'} 2022-09-09T22:33:29.658 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e7dd46d7c13a80c2-00631bbf39-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:29.662 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:33:29.663 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:33:29.664 DEBUG:boto:path=/jkcgbj-14/?prefix=myobj 2022-09-09T22:33:29.665 DEBUG:boto:auth_path=/jkcgbj-14/?prefix=myobj 2022-09-09T22:33:29.666 DEBUG:boto:Method: GET 2022-09-09T22:33:29.666 DEBUG:boto:Path: /jkcgbj-14/?prefix=myobj 2022-09-09T22:33:29.667 DEBUG:boto:Data: 2022-09-09T22:33:29.669 DEBUG:boto:Headers: {} 2022-09-09T22:33:29.669 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:29.670 DEBUG:boto:Port: 8000 2022-09-09T22:33:29.670 DEBUG:boto:Params: {} 2022-09-09T22:33:29.671 DEBUG:boto:Token: None 2022-09-09T22:33:29.671 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:29 GMT /jkcgbj-14/ 2022-09-09T22:33:29.672 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:QGF5P59ahZdVf8MKHZy54nWeqvo= 2022-09-09T22:33:29.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:QGF5P59ahZdVf8MKHZy54nWeqvo=', 'Content-Length': '0'} 2022-09-09T22:33:29.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e88fe55f5bfbab24-00631bbf39-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:29.722 DEBUG:boto:b'jkcgbj-14myobj1000falsemyobj2022-09-09T22:33:06.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:29.723 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:33:29.724 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:33:29.724 DEBUG:boto:path=/jkcgbj-14/?prefix=myobj 2022-09-09T22:33:29.725 DEBUG:boto:auth_path=/jkcgbj-14/?prefix=myobj 2022-09-09T22:33:29.727 DEBUG:boto:Method: GET 2022-09-09T22:33:29.727 DEBUG:boto:Path: /jkcgbj-14/?prefix=myobj 2022-09-09T22:33:29.728 DEBUG:boto:Data: 2022-09-09T22:33:29.729 DEBUG:boto:Headers: {} 2022-09-09T22:33:29.729 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:29.730 DEBUG:boto:Port: 8000 2022-09-09T22:33:29.730 DEBUG:boto:Params: {} 2022-09-09T22:33:29.731 DEBUG:boto:Token: None 2022-09-09T22:33:29.732 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:29 GMT /jkcgbj-14/ 2022-09-09T22:33:29.733 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:QGF5P59ahZdVf8MKHZy54nWeqvo= 2022-09-09T22:33:29.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:QGF5P59ahZdVf8MKHZy54nWeqvo=', 'Content-Length': '0'} 2022-09-09T22:33:29.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad24bbf358ceadad-00631bbf39-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:29.741 DEBUG:boto:b'jkcgbj-14myobj1000falsemyobj2022-09-09T22:33:06.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:29.743 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-14 zones={test-zone1, test-zone2} 2022-09-09T22:33:29.744 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:33:29.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:33:29.747 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:33:29.804 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:33:29.804 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:33:29.877 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:33:29.869+0000 7f860a75bdc0 1 bucket sync caught up with source: 2022-09-09T22:33:29.878 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000001.11.5, , , , , , , , , ] 2022-09-09T22:33:29.878 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000001.11.5, , , , , , , , , ] 2022-09-09T22:33:29.879 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:33:29.869+0000 7f860a75bdc0 0 bucket checkpoint complete 2022-09-09T22:33:29.905 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-14 zones={test-zone3, test-zone2} 2022-09-09T22:33:29.906 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:33:29.907 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:33:29.908 DEBUG:boto:Method: HEAD 2022-09-09T22:33:29.908 DEBUG:boto:Path: /jkcgbj-14/ 2022-09-09T22:33:29.909 DEBUG:boto:Data: 2022-09-09T22:33:29.909 DEBUG:boto:Headers: {} 2022-09-09T22:33:29.910 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:29.910 DEBUG:boto:Port: 8001 2022-09-09T22:33:29.911 DEBUG:boto:Params: {} 2022-09-09T22:33:29.912 DEBUG:boto:Token: None 2022-09-09T22:33:29.912 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:29 GMT /jkcgbj-14/ 2022-09-09T22:33:29.913 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:t/8yb6/UtvwgqFoQxAxSjZhXp+w= 2022-09-09T22:33:29.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:t/8yb6/UtvwgqFoQxAxSjZhXp+w=', 'Content-Length': '0'} 2022-09-09T22:33:29.920 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000003c86c69a7699886-00631bbf39-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:33:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:29.921 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:33:29.921 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:33:29.922 DEBUG:boto:Method: HEAD 2022-09-09T22:33:29.923 DEBUG:boto:Path: /jkcgbj-14/ 2022-09-09T22:33:29.924 DEBUG:boto:Data: 2022-09-09T22:33:29.924 DEBUG:boto:Headers: {} 2022-09-09T22:33:29.925 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:29.927 DEBUG:boto:Port: 8000 2022-09-09T22:33:29.928 DEBUG:boto:Params: {} 2022-09-09T22:33:29.929 DEBUG:boto:Token: None 2022-09-09T22:33:29.930 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:29 GMT /jkcgbj-14/ 2022-09-09T22:33:29.931 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:t/8yb6/UtvwgqFoQxAxSjZhXp+w= 2022-09-09T22:33:29.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:t/8yb6/UtvwgqFoQxAxSjZhXp+w=', 'Content-Length': '0'} 2022-09-09T22:33:29.938 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000086b28e8dce91f31b-00631bbf39-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:33:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:29.939 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:33:29.940 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:33:29.941 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:33:29.942 DEBUG:boto:path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:33:29.943 DEBUG:boto:auth_path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:33:29.944 DEBUG:boto:Method: GET 2022-09-09T22:33:29.945 DEBUG:boto:Path: /jkcgbj-14/?versions&max-keys=999 2022-09-09T22:33:29.945 DEBUG:boto:Data: 2022-09-09T22:33:29.946 DEBUG:boto:Headers: {} 2022-09-09T22:33:29.947 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:29.948 DEBUG:boto:Port: 8001 2022-09-09T22:33:29.949 DEBUG:boto:Params: {} 2022-09-09T22:33:29.950 DEBUG:boto:Token: None 2022-09-09T22:33:29.950 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:29 GMT /jkcgbj-14/?versions 2022-09-09T22:33:29.951 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:98V15E7J+/YJrZpiK94lBvM9ACg= 2022-09-09T22:33:29.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:98V15E7J+/YJrZpiK94lBvM9ACg=', 'Content-Length': '0'} 2022-09-09T22:33:29.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfc5da42ede4d663-00631bbf39-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:29.959 DEBUG:boto:b'jkcgbj-14999falsemyobjnulltrue2022-09-09T22:33:06.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:29.961 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-09T22:33:29.963 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:33:29.964 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:33:29.965 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:33:29.966 DEBUG:boto:path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:33:29.966 DEBUG:boto:auth_path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:33:29.967 DEBUG:boto:Method: GET 2022-09-09T22:33:29.968 DEBUG:boto:Path: /jkcgbj-14/?versions&max-keys=999 2022-09-09T22:33:29.968 DEBUG:boto:Data: 2022-09-09T22:33:29.969 DEBUG:boto:Headers: {} 2022-09-09T22:33:29.970 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:29.971 DEBUG:boto:Port: 8000 2022-09-09T22:33:29.971 DEBUG:boto:Params: {} 2022-09-09T22:33:29.972 DEBUG:boto:Token: None 2022-09-09T22:33:29.973 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:29 GMT /jkcgbj-14/?versions 2022-09-09T22:33:29.974 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:98V15E7J+/YJrZpiK94lBvM9ACg= 2022-09-09T22:33:29.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:98V15E7J+/YJrZpiK94lBvM9ACg=', 'Content-Length': '0'} 2022-09-09T22:33:29.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aab1ef2464d18a54-00631bbf39-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:29.981 DEBUG:boto:b'jkcgbj-14999falsemyobjnulltrue2022-09-09T22:33:06.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:29.982 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-09T22:33:29.983 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:33:29.984 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:33:29.985 DEBUG:boto:path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:33:29.986 DEBUG:boto:auth_path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:33:29.987 DEBUG:boto:Method: GET 2022-09-09T22:33:29.988 DEBUG:boto:Path: /jkcgbj-14/?versions&max-keys=999 2022-09-09T22:33:29.990 DEBUG:boto:Data: 2022-09-09T22:33:29.990 DEBUG:boto:Headers: {} 2022-09-09T22:33:29.991 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:29.991 DEBUG:boto:Port: 8001 2022-09-09T22:33:29.992 DEBUG:boto:Params: {} 2022-09-09T22:33:29.993 DEBUG:boto:Token: None 2022-09-09T22:33:29.993 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:29 GMT /jkcgbj-14/?versions 2022-09-09T22:33:29.994 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:98V15E7J+/YJrZpiK94lBvM9ACg= 2022-09-09T22:33:29.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:98V15E7J+/YJrZpiK94lBvM9ACg=', 'Content-Length': '0'} 2022-09-09T22:33:30.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f4fac323227bafe-00631bbf39-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.002 DEBUG:boto:b'jkcgbj-14999falsemyobjnulltrue2022-09-09T22:33:06.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:30.003 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:33:30.003 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:33:30.004 DEBUG:boto:path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:33:30.004 DEBUG:boto:auth_path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:33:30.005 DEBUG:boto:Method: GET 2022-09-09T22:33:30.005 DEBUG:boto:Path: /jkcgbj-14/?versions&max-keys=999 2022-09-09T22:33:30.006 DEBUG:boto:Data: 2022-09-09T22:33:30.006 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.007 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:30.008 DEBUG:boto:Port: 8000 2022-09-09T22:33:30.008 DEBUG:boto:Params: {} 2022-09-09T22:33:30.009 DEBUG:boto:Token: None 2022-09-09T22:33:30.009 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-14/?versions 2022-09-09T22:33:30.010 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:yBLpQsUODhfK7QJy8OD378QRnHs= 2022-09-09T22:33:30.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:yBLpQsUODhfK7QJy8OD378QRnHs=', 'Content-Length': '0'} 2022-09-09T22:33:30.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030cbc7a7e21363e9-00631bbf3a-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.023 DEBUG:boto:b'jkcgbj-14999falsemyobjnulltrue2022-09-09T22:33:06.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:30.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:33:30.025 DEBUG:boto:path=/jkcgbj-14/myobj 2022-09-09T22:33:30.025 DEBUG:boto:auth_path=/jkcgbj-14/myobj 2022-09-09T22:33:30.026 DEBUG:boto:path=/jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:30.026 DEBUG:boto:auth_path=/jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:30.027 DEBUG:boto:Method: GET 2022-09-09T22:33:30.028 DEBUG:boto:Path: /jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:30.028 DEBUG:boto:Data: 2022-09-09T22:33:30.029 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.030 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:30.030 DEBUG:boto:Port: 8001 2022-09-09T22:33:30.031 DEBUG:boto:Params: {} 2022-09-09T22:33:30.031 DEBUG:boto:Token: None 2022-09-09T22:33:30.032 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:30.033 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:5xsOTyFZe5Ny6g+oXr1cY6pXBoo= 2022-09-09T22:33:30.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:5xsOTyFZe5Ny6g+oXr1cY6pXBoo=', 'Content-Length': '0'} 2022-09-09T22:33:30.046 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7c874e233c47aad-00631bbf3a-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.082 DEBUG:boto:path=/jkcgbj-14/myobj 2022-09-09T22:33:30.083 DEBUG:boto:auth_path=/jkcgbj-14/myobj 2022-09-09T22:33:30.085 DEBUG:boto:path=/jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:30.086 DEBUG:boto:auth_path=/jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:30.088 DEBUG:boto:Method: GET 2022-09-09T22:33:30.088 DEBUG:boto:Path: /jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:30.089 DEBUG:boto:Data: 2022-09-09T22:33:30.090 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.090 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:30.091 DEBUG:boto:Port: 8000 2022-09-09T22:33:30.092 DEBUG:boto:Params: {} 2022-09-09T22:33:30.093 DEBUG:boto:Token: None 2022-09-09T22:33:30.094 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:30.096 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:5xsOTyFZe5Ny6g+oXr1cY6pXBoo= 2022-09-09T22:33:30.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:5xsOTyFZe5Ny6g+oXr1cY6pXBoo=', 'Content-Length': '0'} 2022-09-09T22:33:30.104 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000125bb4263bb21b1c-00631bbf3a-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.108 DEBUG:boto:path=/jkcgbj-14/myobj 2022-09-09T22:33:30.110 DEBUG:boto:auth_path=/jkcgbj-14/myobj 2022-09-09T22:33:30.111 DEBUG:boto:path=/jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:30.113 DEBUG:boto:auth_path=/jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:30.114 DEBUG:boto:Method: HEAD 2022-09-09T22:33:30.115 DEBUG:boto:Path: /jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:30.116 DEBUG:boto:Data: 2022-09-09T22:33:30.117 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.119 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:30.121 DEBUG:boto:Port: 8001 2022-09-09T22:33:30.122 DEBUG:boto:Params: {} 2022-09-09T22:33:30.123 DEBUG:boto:Token: None 2022-09-09T22:33:30.124 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:30.125 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bErXLxrQz+ovA7X2q2GCihDXJ2s= 2022-09-09T22:33:30.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bErXLxrQz+ovA7X2q2GCihDXJ2s=', 'Content-Length': '0'} 2022-09-09T22:33:30.131 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a81dc674d52bb4ce-00631bbf3a-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.133 DEBUG:boto:path=/jkcgbj-14/myobj 2022-09-09T22:33:30.134 DEBUG:boto:auth_path=/jkcgbj-14/myobj 2022-09-09T22:33:30.135 DEBUG:boto:path=/jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:30.136 DEBUG:boto:auth_path=/jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:30.137 DEBUG:boto:Method: HEAD 2022-09-09T22:33:30.138 DEBUG:boto:Path: /jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:30.139 DEBUG:boto:Data: 2022-09-09T22:33:30.140 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.142 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:30.143 DEBUG:boto:Port: 8000 2022-09-09T22:33:30.144 DEBUG:boto:Params: {} 2022-09-09T22:33:30.145 DEBUG:boto:Token: None 2022-09-09T22:33:30.147 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:30.148 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bErXLxrQz+ovA7X2q2GCihDXJ2s= 2022-09-09T22:33:30.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bErXLxrQz+ovA7X2q2GCihDXJ2s=', 'Content-Length': '0'} 2022-09-09T22:33:30.164 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bfd951ec29c085d7-00631bbf3a-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:33:30.168 DEBUG:boto:path=/jkcgbj-14/myobj 2022-09-09T22:33:30.169 DEBUG:boto:auth_path=/jkcgbj-14/myobj 2022-09-09T22:33:30.170 DEBUG:boto:path=/jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:30.171 DEBUG:boto:auth_path=/jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:30.172 DEBUG:boto:Method: GET 2022-09-09T22:33:30.173 DEBUG:boto:Path: /jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:30.174 DEBUG:boto:Data: 2022-09-09T22:33:30.175 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.176 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:30.177 DEBUG:boto:Port: 8001 2022-09-09T22:33:30.178 DEBUG:boto:Params: {} 2022-09-09T22:33:30.179 DEBUG:boto:Token: None 2022-09-09T22:33:30.180 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:30.181 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:5xsOTyFZe5Ny6g+oXr1cY6pXBoo= 2022-09-09T22:33:30.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:5xsOTyFZe5Ny6g+oXr1cY6pXBoo=', 'Content-Length': '0'} 2022-09-09T22:33:30.190 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000596ffc6da88ca422-00631bbf3a-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.192 DEBUG:boto:path=/jkcgbj-14/myobj 2022-09-09T22:33:30.193 DEBUG:boto:auth_path=/jkcgbj-14/myobj 2022-09-09T22:33:30.193 DEBUG:boto:path=/jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:30.194 DEBUG:boto:auth_path=/jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:30.195 DEBUG:boto:Method: GET 2022-09-09T22:33:30.195 DEBUG:boto:Path: /jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:30.196 DEBUG:boto:Data: 2022-09-09T22:33:30.197 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.197 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:30.198 DEBUG:boto:Port: 8000 2022-09-09T22:33:30.199 DEBUG:boto:Params: {} 2022-09-09T22:33:30.199 DEBUG:boto:Token: None 2022-09-09T22:33:30.200 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-14/myobj?versionId=null 2022-09-09T22:33:30.201 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:5xsOTyFZe5Ny6g+oXr1cY6pXBoo= 2022-09-09T22:33:30.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:5xsOTyFZe5Ny6g+oXr1cY6pXBoo=', 'Content-Length': '0'} 2022-09-09T22:33:30.205 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ea7dbcfc20261d0-00631bbf3a-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.207 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:33:30.208 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:33:30.209 DEBUG:boto:path=/jkcgbj-14/?prefix=myobj 2022-09-09T22:33:30.210 DEBUG:boto:auth_path=/jkcgbj-14/?prefix=myobj 2022-09-09T22:33:30.211 DEBUG:boto:Method: GET 2022-09-09T22:33:30.212 DEBUG:boto:Path: /jkcgbj-14/?prefix=myobj 2022-09-09T22:33:30.213 DEBUG:boto:Data: 2022-09-09T22:33:30.214 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.215 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:30.216 DEBUG:boto:Port: 8001 2022-09-09T22:33:30.217 DEBUG:boto:Params: {} 2022-09-09T22:33:30.217 DEBUG:boto:Token: None 2022-09-09T22:33:30.218 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-14/ 2022-09-09T22:33:30.219 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:flSaN2uE9+bREsVxrArzDzaTiSk= 2022-09-09T22:33:30.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:flSaN2uE9+bREsVxrArzDzaTiSk=', 'Content-Length': '0'} 2022-09-09T22:33:30.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000161c4ac3de7040fc-00631bbf3a-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.226 DEBUG:boto:b'jkcgbj-14myobj1000falsemyobj2022-09-09T22:33:06.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:30.226 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:33:30.227 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:33:30.228 DEBUG:boto:path=/jkcgbj-14/?prefix=myobj 2022-09-09T22:33:30.228 DEBUG:boto:auth_path=/jkcgbj-14/?prefix=myobj 2022-09-09T22:33:30.229 DEBUG:boto:Method: GET 2022-09-09T22:33:30.229 DEBUG:boto:Path: /jkcgbj-14/?prefix=myobj 2022-09-09T22:33:30.230 DEBUG:boto:Data: 2022-09-09T22:33:30.230 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.231 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:30.231 DEBUG:boto:Port: 8000 2022-09-09T22:33:30.232 DEBUG:boto:Params: {} 2022-09-09T22:33:30.232 DEBUG:boto:Token: None 2022-09-09T22:33:30.233 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-14/ 2022-09-09T22:33:30.234 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:flSaN2uE9+bREsVxrArzDzaTiSk= 2022-09-09T22:33:30.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:flSaN2uE9+bREsVxrArzDzaTiSk=', 'Content-Length': '0'} 2022-09-09T22:33:30.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055ee638f53dc94b8-00631bbf3a-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.240 DEBUG:boto:b'jkcgbj-14myobj1000falsemyobj2022-09-09T22:33:06.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:30.241 DEBUG:boto:path=/jkcgbj-14/myobj 2022-09-09T22:33:30.242 DEBUG:boto:auth_path=/jkcgbj-14/myobj 2022-09-09T22:33:30.242 DEBUG:boto:Method: HEAD 2022-09-09T22:33:30.243 DEBUG:boto:Path: /jkcgbj-14/myobj 2022-09-09T22:33:30.243 DEBUG:boto:Data: 2022-09-09T22:33:30.244 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.245 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:30.246 DEBUG:boto:Port: 8001 2022-09-09T22:33:30.247 DEBUG:boto:Params: {} 2022-09-09T22:33:30.248 DEBUG:boto:Token: None 2022-09-09T22:33:30.248 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-14/myobj 2022-09-09T22:33:30.250 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rePZMeFLim+VPELGSbPHb3iPxzo= 2022-09-09T22:33:30.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rePZMeFLim+VPELGSbPHb3iPxzo=', 'Content-Length': '0'} 2022-09-09T22:33:30.255 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000659f0563bfbb29bd-00631bbf3a-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.255 DEBUG:boto:path=/jkcgbj-14/myobj 2022-09-09T22:33:30.256 DEBUG:boto:auth_path=/jkcgbj-14/myobj 2022-09-09T22:33:30.257 DEBUG:boto:Method: HEAD 2022-09-09T22:33:30.257 DEBUG:boto:Path: /jkcgbj-14/myobj 2022-09-09T22:33:30.258 DEBUG:boto:Data: 2022-09-09T22:33:30.258 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.259 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:30.259 DEBUG:boto:Port: 8000 2022-09-09T22:33:30.260 DEBUG:boto:Params: {} 2022-09-09T22:33:30.260 DEBUG:boto:Token: None 2022-09-09T22:33:30.261 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-14/myobj 2022-09-09T22:33:30.262 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rePZMeFLim+VPELGSbPHb3iPxzo= 2022-09-09T22:33:30.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rePZMeFLim+VPELGSbPHb3iPxzo=', 'Content-Length': '0'} 2022-09-09T22:33:30.265 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8e21631c2b91d59-00631bbf3a-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:33:30.267 DEBUG:boto:path=/jkcgbj-14/myobj 2022-09-09T22:33:30.267 DEBUG:boto:auth_path=/jkcgbj-14/myobj 2022-09-09T22:33:30.268 DEBUG:boto:Method: GET 2022-09-09T22:33:30.269 DEBUG:boto:Path: /jkcgbj-14/myobj 2022-09-09T22:33:30.269 DEBUG:boto:Data: 2022-09-09T22:33:30.270 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.270 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:30.271 DEBUG:boto:Port: 8001 2022-09-09T22:33:30.271 DEBUG:boto:Params: {} 2022-09-09T22:33:30.272 DEBUG:boto:Token: None 2022-09-09T22:33:30.272 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-14/myobj 2022-09-09T22:33:30.273 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:j9EGfs0x89jzcc+tZX0CgjVer0M= 2022-09-09T22:33:30.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:j9EGfs0x89jzcc+tZX0CgjVer0M=', 'Content-Length': '0'} 2022-09-09T22:33:30.278 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000092156ad52fde41f4-00631bbf3a-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.318 DEBUG:boto:path=/jkcgbj-14/myobj 2022-09-09T22:33:30.319 DEBUG:boto:auth_path=/jkcgbj-14/myobj 2022-09-09T22:33:30.320 DEBUG:boto:Method: GET 2022-09-09T22:33:30.320 DEBUG:boto:Path: /jkcgbj-14/myobj 2022-09-09T22:33:30.321 DEBUG:boto:Data: 2022-09-09T22:33:30.322 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.322 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:30.323 DEBUG:boto:Port: 8000 2022-09-09T22:33:30.323 DEBUG:boto:Params: {} 2022-09-09T22:33:30.324 DEBUG:boto:Token: None 2022-09-09T22:33:30.325 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-14/myobj 2022-09-09T22:33:30.325 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:j9EGfs0x89jzcc+tZX0CgjVer0M= 2022-09-09T22:33:30.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:j9EGfs0x89jzcc+tZX0CgjVer0M=', 'Content-Length': '0'} 2022-09-09T22:33:30.330 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ad173c0eec9a322-00631bbf3a-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.331 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:33:30.331 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:33:30.333 DEBUG:boto:path=/jkcgbj-14/?prefix=myobj 2022-09-09T22:33:30.333 DEBUG:boto:auth_path=/jkcgbj-14/?prefix=myobj 2022-09-09T22:33:30.334 DEBUG:boto:Method: GET 2022-09-09T22:33:30.335 DEBUG:boto:Path: /jkcgbj-14/?prefix=myobj 2022-09-09T22:33:30.335 DEBUG:boto:Data: 2022-09-09T22:33:30.336 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.336 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:30.337 DEBUG:boto:Port: 8001 2022-09-09T22:33:30.337 DEBUG:boto:Params: {} 2022-09-09T22:33:30.338 DEBUG:boto:Token: None 2022-09-09T22:33:30.339 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-14/ 2022-09-09T22:33:30.339 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:flSaN2uE9+bREsVxrArzDzaTiSk= 2022-09-09T22:33:30.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:flSaN2uE9+bREsVxrArzDzaTiSk=', 'Content-Length': '0'} 2022-09-09T22:33:30.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007439a68b18e504a3-00631bbf3a-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.347 DEBUG:boto:b'jkcgbj-14myobj1000falsemyobj2022-09-09T22:33:06.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:30.348 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:33:30.348 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:33:30.349 DEBUG:boto:path=/jkcgbj-14/?prefix=myobj 2022-09-09T22:33:30.349 DEBUG:boto:auth_path=/jkcgbj-14/?prefix=myobj 2022-09-09T22:33:30.350 DEBUG:boto:Method: GET 2022-09-09T22:33:30.351 DEBUG:boto:Path: /jkcgbj-14/?prefix=myobj 2022-09-09T22:33:30.351 DEBUG:boto:Data: 2022-09-09T22:33:30.352 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.352 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:30.353 DEBUG:boto:Port: 8000 2022-09-09T22:33:30.353 DEBUG:boto:Params: {} 2022-09-09T22:33:30.354 DEBUG:boto:Token: None 2022-09-09T22:33:30.355 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-14/ 2022-09-09T22:33:30.356 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:flSaN2uE9+bREsVxrArzDzaTiSk= 2022-09-09T22:33:30.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:flSaN2uE9+bREsVxrArzDzaTiSk=', 'Content-Length': '0'} 2022-09-09T22:33:30.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095b0762c117d11cd-00631bbf3a-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.364 DEBUG:boto:b'jkcgbj-14myobj1000falsemyobj2022-09-09T22:33:06.800Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:30.365 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-14 zones={test-zone3, test-zone2} 2022-09-09T22:33:30.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:33:30.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:33:30.368 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:33:30.431 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:33:30.431 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:33:30.509 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:33:30.489+0000 7f8dde59bdc0 1 bucket sync caught up with source: 2022-09-09T22:33:30.509 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000001.14.5, , , , , , , , , ] 2022-09-09T22:33:30.510 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000001.14.5, , , , , , , , , ] 2022-09-09T22:33:30.511 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:33:30.489+0000 7f8dde59bdc0 0 bucket checkpoint complete 2022-09-09T22:33:30.524 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-15 zones={test-zone1, test-zone3} 2022-09-09T22:33:30.524 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:33:30.525 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:33:30.525 DEBUG:boto:Method: HEAD 2022-09-09T22:33:30.526 DEBUG:boto:Path: /jkcgbj-15/ 2022-09-09T22:33:30.527 DEBUG:boto:Data: 2022-09-09T22:33:30.527 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.528 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:30.528 DEBUG:boto:Port: 8000 2022-09-09T22:33:30.529 DEBUG:boto:Params: {} 2022-09-09T22:33:30.529 DEBUG:boto:Token: None 2022-09-09T22:33:30.530 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-15/ 2022-09-09T22:33:30.532 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Id1OQBk3+IR5u7AWXwQmpMEOOtM= 2022-09-09T22:33:30.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Id1OQBk3+IR5u7AWXwQmpMEOOtM=', 'Content-Length': '0'} 2022-09-09T22:33:30.540 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000017f3744e80e42468-00631bbf3a-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.542 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:33:30.543 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:33:30.544 DEBUG:boto:Method: HEAD 2022-09-09T22:33:30.545 DEBUG:boto:Path: /jkcgbj-15/ 2022-09-09T22:33:30.545 DEBUG:boto:Data: 2022-09-09T22:33:30.547 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.547 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:30.548 DEBUG:boto:Port: 8001 2022-09-09T22:33:30.548 DEBUG:boto:Params: {} 2022-09-09T22:33:30.549 DEBUG:boto:Token: None 2022-09-09T22:33:30.549 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-15/ 2022-09-09T22:33:30.550 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Id1OQBk3+IR5u7AWXwQmpMEOOtM= 2022-09-09T22:33:30.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Id1OQBk3+IR5u7AWXwQmpMEOOtM=', 'Content-Length': '0'} 2022-09-09T22:33:30.560 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000afc677575e730748-00631bbf3a-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.562 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:33:30.563 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:33:30.567 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:33:30.568 DEBUG:boto:path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:33:30.569 DEBUG:boto:auth_path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:33:30.570 DEBUG:boto:Method: GET 2022-09-09T22:33:30.571 DEBUG:boto:Path: /jkcgbj-15/?versions&max-keys=999 2022-09-09T22:33:30.572 DEBUG:boto:Data: 2022-09-09T22:33:30.573 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.573 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:30.575 DEBUG:boto:Port: 8000 2022-09-09T22:33:30.576 DEBUG:boto:Params: {} 2022-09-09T22:33:30.577 DEBUG:boto:Token: None 2022-09-09T22:33:30.578 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-15/?versions 2022-09-09T22:33:30.579 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8V92OAg4d5KxwBgrgoM3Asg2SMQ= 2022-09-09T22:33:30.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8V92OAg4d5KxwBgrgoM3Asg2SMQ=', 'Content-Length': '0'} 2022-09-09T22:33:30.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df1d600106a1fa48-00631bbf3a-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.591 DEBUG:boto:b'jkcgbj-15999falsemyobjnulltrue2022-09-09T22:33:06.844Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:30.592 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-09T22:33:30.593 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:33:30.594 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:33:30.595 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:33:30.596 DEBUG:boto:path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:33:30.596 DEBUG:boto:auth_path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:33:30.597 DEBUG:boto:Method: GET 2022-09-09T22:33:30.598 DEBUG:boto:Path: /jkcgbj-15/?versions&max-keys=999 2022-09-09T22:33:30.599 DEBUG:boto:Data: 2022-09-09T22:33:30.601 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.602 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:30.603 DEBUG:boto:Port: 8001 2022-09-09T22:33:30.612 DEBUG:boto:Params: {} 2022-09-09T22:33:30.613 DEBUG:boto:Token: None 2022-09-09T22:33:30.614 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-15/?versions 2022-09-09T22:33:30.615 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8V92OAg4d5KxwBgrgoM3Asg2SMQ= 2022-09-09T22:33:30.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8V92OAg4d5KxwBgrgoM3Asg2SMQ=', 'Content-Length': '0'} 2022-09-09T22:33:30.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007af16f1d15a7fd61-00631bbf3a-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.624 DEBUG:boto:b'jkcgbj-15999falsemyobjnulltrue2022-09-09T22:33:06.844Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:30.625 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-09T22:33:30.626 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:33:30.626 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:33:30.627 DEBUG:boto:path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:33:30.628 DEBUG:boto:auth_path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:33:30.628 DEBUG:boto:Method: GET 2022-09-09T22:33:30.629 DEBUG:boto:Path: /jkcgbj-15/?versions&max-keys=999 2022-09-09T22:33:30.629 DEBUG:boto:Data: 2022-09-09T22:33:30.631 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.632 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:30.633 DEBUG:boto:Port: 8000 2022-09-09T22:33:30.635 DEBUG:boto:Params: {} 2022-09-09T22:33:30.636 DEBUG:boto:Token: None 2022-09-09T22:33:30.638 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-15/?versions 2022-09-09T22:33:30.639 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8V92OAg4d5KxwBgrgoM3Asg2SMQ= 2022-09-09T22:33:30.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8V92OAg4d5KxwBgrgoM3Asg2SMQ=', 'Content-Length': '0'} 2022-09-09T22:33:30.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8af1920dc7b9f1b-00631bbf3a-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.647 DEBUG:boto:b'jkcgbj-15999falsemyobjnulltrue2022-09-09T22:33:06.844Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:30.648 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:33:30.649 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:33:30.649 DEBUG:boto:path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:33:30.650 DEBUG:boto:auth_path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:33:30.650 DEBUG:boto:Method: GET 2022-09-09T22:33:30.651 DEBUG:boto:Path: /jkcgbj-15/?versions&max-keys=999 2022-09-09T22:33:30.652 DEBUG:boto:Data: 2022-09-09T22:33:30.653 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.653 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:30.654 DEBUG:boto:Port: 8001 2022-09-09T22:33:30.654 DEBUG:boto:Params: {} 2022-09-09T22:33:30.655 DEBUG:boto:Token: None 2022-09-09T22:33:30.656 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-15/?versions 2022-09-09T22:33:30.657 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8V92OAg4d5KxwBgrgoM3Asg2SMQ= 2022-09-09T22:33:30.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8V92OAg4d5KxwBgrgoM3Asg2SMQ=', 'Content-Length': '0'} 2022-09-09T22:33:30.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ed3ea9f7a532a87-00631bbf3a-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.664 DEBUG:boto:b'jkcgbj-15999falsemyobjnulltrue2022-09-09T22:33:06.844Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:30.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:33:30.666 DEBUG:boto:path=/jkcgbj-15/myobj 2022-09-09T22:33:30.666 DEBUG:boto:auth_path=/jkcgbj-15/myobj 2022-09-09T22:33:30.667 DEBUG:boto:path=/jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:30.668 DEBUG:boto:auth_path=/jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:30.669 DEBUG:boto:Method: GET 2022-09-09T22:33:30.669 DEBUG:boto:Path: /jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:30.670 DEBUG:boto:Data: 2022-09-09T22:33:30.670 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.671 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:30.671 DEBUG:boto:Port: 8000 2022-09-09T22:33:30.672 DEBUG:boto:Params: {} 2022-09-09T22:33:30.673 DEBUG:boto:Token: None 2022-09-09T22:33:30.674 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:30.674 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6/KtatajFH6HXw0yxIUrE6uwoFU= 2022-09-09T22:33:30.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6/KtatajFH6HXw0yxIUrE6uwoFU=', 'Content-Length': '0'} 2022-09-09T22:33:30.678 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d722f6491425f943-00631bbf3a-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.679 DEBUG:boto:path=/jkcgbj-15/myobj 2022-09-09T22:33:30.680 DEBUG:boto:auth_path=/jkcgbj-15/myobj 2022-09-09T22:33:30.681 DEBUG:boto:path=/jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:30.681 DEBUG:boto:auth_path=/jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:30.682 DEBUG:boto:Method: GET 2022-09-09T22:33:30.682 DEBUG:boto:Path: /jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:30.683 DEBUG:boto:Data: 2022-09-09T22:33:30.684 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.684 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:30.685 DEBUG:boto:Port: 8001 2022-09-09T22:33:30.685 DEBUG:boto:Params: {} 2022-09-09T22:33:30.686 DEBUG:boto:Token: None 2022-09-09T22:33:30.687 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:30.687 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6/KtatajFH6HXw0yxIUrE6uwoFU= 2022-09-09T22:33:30.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6/KtatajFH6HXw0yxIUrE6uwoFU=', 'Content-Length': '0'} 2022-09-09T22:33:30.691 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000035482b6bd5158b2b-00631bbf3a-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.734 DEBUG:boto:path=/jkcgbj-15/myobj 2022-09-09T22:33:30.735 DEBUG:boto:auth_path=/jkcgbj-15/myobj 2022-09-09T22:33:30.736 DEBUG:boto:path=/jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:30.736 DEBUG:boto:auth_path=/jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:30.737 DEBUG:boto:Method: HEAD 2022-09-09T22:33:30.737 DEBUG:boto:Path: /jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:30.738 DEBUG:boto:Data: 2022-09-09T22:33:30.738 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.739 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:30.739 DEBUG:boto:Port: 8000 2022-09-09T22:33:30.740 DEBUG:boto:Params: {} 2022-09-09T22:33:30.741 DEBUG:boto:Token: None 2022-09-09T22:33:30.741 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:30.742 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:xpHpKqjUAZ5rlKym53IlO3N+2Dw= 2022-09-09T22:33:30.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:xpHpKqjUAZ5rlKym53IlO3N+2Dw=', 'Content-Length': '0'} 2022-09-09T22:33:30.745 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e8468768bbb15e18-00631bbf3a-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.746 DEBUG:boto:path=/jkcgbj-15/myobj 2022-09-09T22:33:30.747 DEBUG:boto:auth_path=/jkcgbj-15/myobj 2022-09-09T22:33:30.747 DEBUG:boto:path=/jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:30.748 DEBUG:boto:auth_path=/jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:30.748 DEBUG:boto:Method: HEAD 2022-09-09T22:33:30.749 DEBUG:boto:Path: /jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:30.749 DEBUG:boto:Data: 2022-09-09T22:33:30.750 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.750 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:30.751 DEBUG:boto:Port: 8001 2022-09-09T22:33:30.751 DEBUG:boto:Params: {} 2022-09-09T22:33:30.752 DEBUG:boto:Token: None 2022-09-09T22:33:30.753 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:30.753 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:xpHpKqjUAZ5rlKym53IlO3N+2Dw= 2022-09-09T22:33:30.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:xpHpKqjUAZ5rlKym53IlO3N+2Dw=', 'Content-Length': '0'} 2022-09-09T22:33:30.757 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000395097384eae5deb-00631bbf3a-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:33:30.759 DEBUG:boto:path=/jkcgbj-15/myobj 2022-09-09T22:33:30.760 DEBUG:boto:auth_path=/jkcgbj-15/myobj 2022-09-09T22:33:30.761 DEBUG:boto:path=/jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:30.762 DEBUG:boto:auth_path=/jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:30.763 DEBUG:boto:Method: GET 2022-09-09T22:33:30.764 DEBUG:boto:Path: /jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:30.765 DEBUG:boto:Data: 2022-09-09T22:33:30.766 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.767 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:30.768 DEBUG:boto:Port: 8000 2022-09-09T22:33:30.769 DEBUG:boto:Params: {} 2022-09-09T22:33:30.770 DEBUG:boto:Token: None 2022-09-09T22:33:30.771 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:30.773 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6/KtatajFH6HXw0yxIUrE6uwoFU= 2022-09-09T22:33:30.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6/KtatajFH6HXw0yxIUrE6uwoFU=', 'Content-Length': '0'} 2022-09-09T22:33:30.777 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6bb72aa9ddf1cff-00631bbf3a-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.778 DEBUG:boto:path=/jkcgbj-15/myobj 2022-09-09T22:33:30.779 DEBUG:boto:auth_path=/jkcgbj-15/myobj 2022-09-09T22:33:30.779 DEBUG:boto:path=/jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:30.780 DEBUG:boto:auth_path=/jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:30.780 DEBUG:boto:Method: GET 2022-09-09T22:33:30.781 DEBUG:boto:Path: /jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:30.781 DEBUG:boto:Data: 2022-09-09T22:33:30.782 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.782 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:30.783 DEBUG:boto:Port: 8001 2022-09-09T22:33:30.783 DEBUG:boto:Params: {} 2022-09-09T22:33:30.784 DEBUG:boto:Token: None 2022-09-09T22:33:30.785 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:30.786 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6/KtatajFH6HXw0yxIUrE6uwoFU= 2022-09-09T22:33:30.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6/KtatajFH6HXw0yxIUrE6uwoFU=', 'Content-Length': '0'} 2022-09-09T22:33:30.789 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a33bc0d6c30971c-00631bbf3a-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.791 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:33:30.792 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:33:30.793 DEBUG:boto:path=/jkcgbj-15/?prefix=myobj 2022-09-09T22:33:30.794 DEBUG:boto:auth_path=/jkcgbj-15/?prefix=myobj 2022-09-09T22:33:30.795 DEBUG:boto:Method: GET 2022-09-09T22:33:30.796 DEBUG:boto:Path: /jkcgbj-15/?prefix=myobj 2022-09-09T22:33:30.797 DEBUG:boto:Data: 2022-09-09T22:33:30.798 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.799 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:30.800 DEBUG:boto:Port: 8000 2022-09-09T22:33:30.801 DEBUG:boto:Params: {} 2022-09-09T22:33:30.802 DEBUG:boto:Token: None 2022-09-09T22:33:30.803 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-15/ 2022-09-09T22:33:30.804 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UMl7m/mBBbwxFxXeDNfZ8auCKLA= 2022-09-09T22:33:30.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UMl7m/mBBbwxFxXeDNfZ8auCKLA=', 'Content-Length': '0'} 2022-09-09T22:33:30.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbd180ba7f7279d8-00631bbf3a-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.814 DEBUG:boto:b'jkcgbj-15myobj1000falsemyobj2022-09-09T22:33:06.844Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:30.815 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:33:30.816 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:33:30.816 DEBUG:boto:path=/jkcgbj-15/?prefix=myobj 2022-09-09T22:33:30.817 DEBUG:boto:auth_path=/jkcgbj-15/?prefix=myobj 2022-09-09T22:33:30.818 DEBUG:boto:Method: GET 2022-09-09T22:33:30.818 DEBUG:boto:Path: /jkcgbj-15/?prefix=myobj 2022-09-09T22:33:30.819 DEBUG:boto:Data: 2022-09-09T22:33:30.819 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.820 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:30.820 DEBUG:boto:Port: 8001 2022-09-09T22:33:30.821 DEBUG:boto:Params: {} 2022-09-09T22:33:30.822 DEBUG:boto:Token: None 2022-09-09T22:33:30.822 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-15/ 2022-09-09T22:33:30.823 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UMl7m/mBBbwxFxXeDNfZ8auCKLA= 2022-09-09T22:33:30.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UMl7m/mBBbwxFxXeDNfZ8auCKLA=', 'Content-Length': '0'} 2022-09-09T22:33:30.829 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbdee1801bd562aa-00631bbf3a-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.830 DEBUG:boto:b'jkcgbj-15myobj1000falsemyobj2022-09-09T22:33:06.844Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:30.831 DEBUG:boto:path=/jkcgbj-15/myobj 2022-09-09T22:33:30.831 DEBUG:boto:auth_path=/jkcgbj-15/myobj 2022-09-09T22:33:30.832 DEBUG:boto:Method: HEAD 2022-09-09T22:33:30.832 DEBUG:boto:Path: /jkcgbj-15/myobj 2022-09-09T22:33:30.833 DEBUG:boto:Data: 2022-09-09T22:33:30.833 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.834 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:30.834 DEBUG:boto:Port: 8000 2022-09-09T22:33:30.835 DEBUG:boto:Params: {} 2022-09-09T22:33:30.836 DEBUG:boto:Token: None 2022-09-09T22:33:30.836 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-15/myobj 2022-09-09T22:33:30.837 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nNisYbdchN55WJLTOiZ3icydu7E= 2022-09-09T22:33:30.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nNisYbdchN55WJLTOiZ3icydu7E=', 'Content-Length': '0'} 2022-09-09T22:33:30.840 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e0c5e79a3ec2e26-00631bbf3a-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.841 DEBUG:boto:path=/jkcgbj-15/myobj 2022-09-09T22:33:30.842 DEBUG:boto:auth_path=/jkcgbj-15/myobj 2022-09-09T22:33:30.842 DEBUG:boto:Method: HEAD 2022-09-09T22:33:30.843 DEBUG:boto:Path: /jkcgbj-15/myobj 2022-09-09T22:33:30.843 DEBUG:boto:Data: 2022-09-09T22:33:30.844 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.844 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:30.845 DEBUG:boto:Port: 8001 2022-09-09T22:33:30.845 DEBUG:boto:Params: {} 2022-09-09T22:33:30.846 DEBUG:boto:Token: None 2022-09-09T22:33:30.847 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-15/myobj 2022-09-09T22:33:30.847 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nNisYbdchN55WJLTOiZ3icydu7E= 2022-09-09T22:33:30.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nNisYbdchN55WJLTOiZ3icydu7E=', 'Content-Length': '0'} 2022-09-09T22:33:30.851 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001da09d7240ff7aa3-00631bbf3a-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:33:30.853 DEBUG:boto:path=/jkcgbj-15/myobj 2022-09-09T22:33:30.853 DEBUG:boto:auth_path=/jkcgbj-15/myobj 2022-09-09T22:33:30.854 DEBUG:boto:Method: GET 2022-09-09T22:33:30.854 DEBUG:boto:Path: /jkcgbj-15/myobj 2022-09-09T22:33:30.855 DEBUG:boto:Data: 2022-09-09T22:33:30.855 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.856 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:30.856 DEBUG:boto:Port: 8000 2022-09-09T22:33:30.857 DEBUG:boto:Params: {} 2022-09-09T22:33:30.857 DEBUG:boto:Token: None 2022-09-09T22:33:30.858 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-15/myobj 2022-09-09T22:33:30.859 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TmMHiijbb7A3uGJ0VgWXaTRumNE= 2022-09-09T22:33:30.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TmMHiijbb7A3uGJ0VgWXaTRumNE=', 'Content-Length': '0'} 2022-09-09T22:33:30.863 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c398fe49d472c64-00631bbf3a-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.864 DEBUG:boto:path=/jkcgbj-15/myobj 2022-09-09T22:33:30.864 DEBUG:boto:auth_path=/jkcgbj-15/myobj 2022-09-09T22:33:30.865 DEBUG:boto:Method: GET 2022-09-09T22:33:30.865 DEBUG:boto:Path: /jkcgbj-15/myobj 2022-09-09T22:33:30.866 DEBUG:boto:Data: 2022-09-09T22:33:30.866 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.867 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:30.868 DEBUG:boto:Port: 8001 2022-09-09T22:33:30.869 DEBUG:boto:Params: {} 2022-09-09T22:33:30.869 DEBUG:boto:Token: None 2022-09-09T22:33:30.870 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-15/myobj 2022-09-09T22:33:30.870 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TmMHiijbb7A3uGJ0VgWXaTRumNE= 2022-09-09T22:33:30.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TmMHiijbb7A3uGJ0VgWXaTRumNE=', 'Content-Length': '0'} 2022-09-09T22:33:30.874 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000801de500d4d5c299-00631bbf3a-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.919 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:33:30.920 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:33:30.920 DEBUG:boto:path=/jkcgbj-15/?prefix=myobj 2022-09-09T22:33:30.921 DEBUG:boto:auth_path=/jkcgbj-15/?prefix=myobj 2022-09-09T22:33:30.922 DEBUG:boto:Method: GET 2022-09-09T22:33:30.922 DEBUG:boto:Path: /jkcgbj-15/?prefix=myobj 2022-09-09T22:33:30.923 DEBUG:boto:Data: 2022-09-09T22:33:30.924 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.925 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:30.926 DEBUG:boto:Port: 8000 2022-09-09T22:33:30.927 DEBUG:boto:Params: {} 2022-09-09T22:33:30.930 DEBUG:boto:Token: None 2022-09-09T22:33:30.931 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-15/ 2022-09-09T22:33:30.932 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UMl7m/mBBbwxFxXeDNfZ8auCKLA= 2022-09-09T22:33:30.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UMl7m/mBBbwxFxXeDNfZ8auCKLA=', 'Content-Length': '0'} 2022-09-09T22:33:30.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003092fdd2c2610d1d-00631bbf3a-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.941 DEBUG:boto:b'jkcgbj-15myobj1000falsemyobj2022-09-09T22:33:06.844Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:30.943 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:33:30.944 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:33:30.944 DEBUG:boto:path=/jkcgbj-15/?prefix=myobj 2022-09-09T22:33:30.946 DEBUG:boto:auth_path=/jkcgbj-15/?prefix=myobj 2022-09-09T22:33:30.947 DEBUG:boto:Method: GET 2022-09-09T22:33:30.948 DEBUG:boto:Path: /jkcgbj-15/?prefix=myobj 2022-09-09T22:33:30.949 DEBUG:boto:Data: 2022-09-09T22:33:30.950 DEBUG:boto:Headers: {} 2022-09-09T22:33:30.952 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:30.953 DEBUG:boto:Port: 8001 2022-09-09T22:33:30.955 DEBUG:boto:Params: {} 2022-09-09T22:33:30.957 DEBUG:boto:Token: None 2022-09-09T22:33:30.959 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:30 GMT /jkcgbj-15/ 2022-09-09T22:33:30.960 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UMl7m/mBBbwxFxXeDNfZ8auCKLA= 2022-09-09T22:33:30.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UMl7m/mBBbwxFxXeDNfZ8auCKLA=', 'Content-Length': '0'} 2022-09-09T22:33:30.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051158b3076756749-00631bbf3a-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:30.968 DEBUG:boto:b'jkcgbj-15myobj1000falsemyobj2022-09-09T22:33:06.844Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:30.969 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-15 zones={test-zone1, test-zone3} 2022-09-09T22:33:30.970 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:33:30.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:33:30.973 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:33:31.029 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:33:31.030 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:33:31.100 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:33:31.094+0000 7fbc84565dc0 1 bucket sync caught up with source: 2022-09-09T22:33:31.100 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000001.14.5, , , , , , , , , ] 2022-09-09T22:33:31.101 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000001.14.5, , , , , , , , , ] 2022-09-09T22:33:31.101 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:33:31.094+0000 7fbc84565dc0 0 bucket checkpoint complete 2022-09-09T22:33:31.136 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-15 zones={test-zone2, test-zone3} 2022-09-09T22:33:31.137 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:33:31.137 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:33:31.138 DEBUG:boto:Method: HEAD 2022-09-09T22:33:31.138 DEBUG:boto:Path: /jkcgbj-15/ 2022-09-09T22:33:31.139 DEBUG:boto:Data: 2022-09-09T22:33:31.139 DEBUG:boto:Headers: {} 2022-09-09T22:33:31.140 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:31.141 DEBUG:boto:Port: 8000 2022-09-09T22:33:31.141 DEBUG:boto:Params: {} 2022-09-09T22:33:31.143 DEBUG:boto:Token: None 2022-09-09T22:33:31.144 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:31 GMT /jkcgbj-15/ 2022-09-09T22:33:31.145 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:NsiBWTzAQgQXvhB92Mw+CEi0bIA= 2022-09-09T22:33:31.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:NsiBWTzAQgQXvhB92Mw+CEi0bIA=', 'Content-Length': '0'} 2022-09-09T22:33:31.150 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085510cd583014a81-00631bbf3b-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:33:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:31.151 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:33:31.152 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:33:31.153 DEBUG:boto:Method: HEAD 2022-09-09T22:33:31.154 DEBUG:boto:Path: /jkcgbj-15/ 2022-09-09T22:33:31.155 DEBUG:boto:Data: 2022-09-09T22:33:31.156 DEBUG:boto:Headers: {} 2022-09-09T22:33:31.156 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:31.157 DEBUG:boto:Port: 8001 2022-09-09T22:33:31.158 DEBUG:boto:Params: {} 2022-09-09T22:33:31.159 DEBUG:boto:Token: None 2022-09-09T22:33:31.160 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:31 GMT /jkcgbj-15/ 2022-09-09T22:33:31.161 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:NsiBWTzAQgQXvhB92Mw+CEi0bIA= 2022-09-09T22:33:31.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:NsiBWTzAQgQXvhB92Mw+CEi0bIA=', 'Content-Length': '0'} 2022-09-09T22:33:31.169 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c9093ce2d6d6dfab-00631bbf3b-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:33:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:31.170 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:33:31.170 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:33:31.171 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:33:31.171 DEBUG:boto:path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:33:31.172 DEBUG:boto:auth_path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:33:31.173 DEBUG:boto:Method: GET 2022-09-09T22:33:31.174 DEBUG:boto:Path: /jkcgbj-15/?versions&max-keys=999 2022-09-09T22:33:31.175 DEBUG:boto:Data: 2022-09-09T22:33:31.176 DEBUG:boto:Headers: {} 2022-09-09T22:33:31.176 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:31.178 DEBUG:boto:Port: 8000 2022-09-09T22:33:31.179 DEBUG:boto:Params: {} 2022-09-09T22:33:31.180 DEBUG:boto:Token: None 2022-09-09T22:33:31.181 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:31 GMT /jkcgbj-15/?versions 2022-09-09T22:33:31.181 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:QHivyNsq9xZxd5tXaBHWQ2Zquq8= 2022-09-09T22:33:31.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:QHivyNsq9xZxd5tXaBHWQ2Zquq8=', 'Content-Length': '0'} 2022-09-09T22:33:31.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c993efd4f880b1d3-00631bbf3b-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:31.189 DEBUG:boto:b'jkcgbj-15999falsemyobjnulltrue2022-09-09T22:33:06.844Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:31.190 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-09T22:33:31.191 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:33:31.192 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:33:31.192 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:33:31.193 DEBUG:boto:path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:33:31.193 DEBUG:boto:auth_path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:33:31.194 DEBUG:boto:Method: GET 2022-09-09T22:33:31.194 DEBUG:boto:Path: /jkcgbj-15/?versions&max-keys=999 2022-09-09T22:33:31.198 DEBUG:boto:Data: 2022-09-09T22:33:31.199 DEBUG:boto:Headers: {} 2022-09-09T22:33:31.199 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:31.200 DEBUG:boto:Port: 8001 2022-09-09T22:33:31.200 DEBUG:boto:Params: {} 2022-09-09T22:33:31.201 DEBUG:boto:Token: None 2022-09-09T22:33:31.202 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:31 GMT /jkcgbj-15/?versions 2022-09-09T22:33:31.202 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:QHivyNsq9xZxd5tXaBHWQ2Zquq8= 2022-09-09T22:33:31.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:QHivyNsq9xZxd5tXaBHWQ2Zquq8=', 'Content-Length': '0'} 2022-09-09T22:33:31.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000901087949a2a8dbd-00631bbf3b-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:31.210 DEBUG:boto:b'jkcgbj-15999falsemyobjnulltrue2022-09-09T22:33:06.844Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:31.211 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-09T22:33:31.212 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:33:31.212 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:33:31.213 DEBUG:boto:path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:33:31.213 DEBUG:boto:auth_path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:33:31.214 DEBUG:boto:Method: GET 2022-09-09T22:33:31.215 DEBUG:boto:Path: /jkcgbj-15/?versions&max-keys=999 2022-09-09T22:33:31.215 DEBUG:boto:Data: 2022-09-09T22:33:31.216 DEBUG:boto:Headers: {} 2022-09-09T22:33:31.216 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:31.217 DEBUG:boto:Port: 8000 2022-09-09T22:33:31.218 DEBUG:boto:Params: {} 2022-09-09T22:33:31.220 DEBUG:boto:Token: None 2022-09-09T22:33:31.221 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:31 GMT /jkcgbj-15/?versions 2022-09-09T22:33:31.222 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:QHivyNsq9xZxd5tXaBHWQ2Zquq8= 2022-09-09T22:33:31.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:QHivyNsq9xZxd5tXaBHWQ2Zquq8=', 'Content-Length': '0'} 2022-09-09T22:33:31.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b13ed584e36f8a71-00631bbf3b-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:31.233 DEBUG:boto:b'jkcgbj-15999falsemyobjnulltrue2022-09-09T22:33:06.844Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:31.234 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:33:31.235 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:33:31.235 DEBUG:boto:path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:33:31.236 DEBUG:boto:auth_path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:33:31.237 DEBUG:boto:Method: GET 2022-09-09T22:33:31.238 DEBUG:boto:Path: /jkcgbj-15/?versions&max-keys=999 2022-09-09T22:33:31.239 DEBUG:boto:Data: 2022-09-09T22:33:31.240 DEBUG:boto:Headers: {} 2022-09-09T22:33:31.241 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:31.242 DEBUG:boto:Port: 8001 2022-09-09T22:33:31.243 DEBUG:boto:Params: {} 2022-09-09T22:33:31.244 DEBUG:boto:Token: None 2022-09-09T22:33:31.245 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:31 GMT /jkcgbj-15/?versions 2022-09-09T22:33:31.247 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:QHivyNsq9xZxd5tXaBHWQ2Zquq8= 2022-09-09T22:33:31.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:QHivyNsq9xZxd5tXaBHWQ2Zquq8=', 'Content-Length': '0'} 2022-09-09T22:33:31.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a135f9d0b8729a51-00631bbf3b-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:31.261 DEBUG:boto:b'jkcgbj-15999falsemyobjnulltrue2022-09-09T22:33:06.844Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:31.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:33:31.263 DEBUG:boto:path=/jkcgbj-15/myobj 2022-09-09T22:33:31.263 DEBUG:boto:auth_path=/jkcgbj-15/myobj 2022-09-09T22:33:31.264 DEBUG:boto:path=/jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:31.264 DEBUG:boto:auth_path=/jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:31.265 DEBUG:boto:Method: GET 2022-09-09T22:33:31.266 DEBUG:boto:Path: /jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:31.267 DEBUG:boto:Data: 2022-09-09T22:33:31.268 DEBUG:boto:Headers: {} 2022-09-09T22:33:31.268 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:31.269 DEBUG:boto:Port: 8000 2022-09-09T22:33:31.270 DEBUG:boto:Params: {} 2022-09-09T22:33:31.272 DEBUG:boto:Token: None 2022-09-09T22:33:31.273 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:31 GMT /jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:31.274 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:r4sDC5jp30VZKl8Lg4//LO7ScOo= 2022-09-09T22:33:31.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:r4sDC5jp30VZKl8Lg4//LO7ScOo=', 'Content-Length': '0'} 2022-09-09T22:33:31.279 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000049dbbfb44c9fd862-00631bbf3b-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:31.280 DEBUG:boto:path=/jkcgbj-15/myobj 2022-09-09T22:33:31.281 DEBUG:boto:auth_path=/jkcgbj-15/myobj 2022-09-09T22:33:31.283 DEBUG:boto:path=/jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:31.283 DEBUG:boto:auth_path=/jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:31.284 DEBUG:boto:Method: GET 2022-09-09T22:33:31.285 DEBUG:boto:Path: /jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:31.286 DEBUG:boto:Data: 2022-09-09T22:33:31.287 DEBUG:boto:Headers: {} 2022-09-09T22:33:31.288 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:31.289 DEBUG:boto:Port: 8001 2022-09-09T22:33:31.289 DEBUG:boto:Params: {} 2022-09-09T22:33:31.290 DEBUG:boto:Token: None 2022-09-09T22:33:31.291 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:31 GMT /jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:31.292 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:r4sDC5jp30VZKl8Lg4//LO7ScOo= 2022-09-09T22:33:31.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:r4sDC5jp30VZKl8Lg4//LO7ScOo=', 'Content-Length': '0'} 2022-09-09T22:33:31.298 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6d937ade24a98fb-00631bbf3b-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:31.300 DEBUG:boto:path=/jkcgbj-15/myobj 2022-09-09T22:33:31.301 DEBUG:boto:auth_path=/jkcgbj-15/myobj 2022-09-09T22:33:31.303 DEBUG:boto:path=/jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:31.304 DEBUG:boto:auth_path=/jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:31.305 DEBUG:boto:Method: HEAD 2022-09-09T22:33:31.305 DEBUG:boto:Path: /jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:31.306 DEBUG:boto:Data: 2022-09-09T22:33:31.307 DEBUG:boto:Headers: {} 2022-09-09T22:33:31.307 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:31.308 DEBUG:boto:Port: 8000 2022-09-09T22:33:31.308 DEBUG:boto:Params: {} 2022-09-09T22:33:31.309 DEBUG:boto:Token: None 2022-09-09T22:33:31.309 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:31 GMT /jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:31.310 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:eWK34oa1lR9nngEBm+bVk7nDZL0= 2022-09-09T22:33:31.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:eWK34oa1lR9nngEBm+bVk7nDZL0=', 'Content-Length': '0'} 2022-09-09T22:33:31.314 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004930fe16ba3047ac-00631bbf3b-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:31.316 DEBUG:boto:path=/jkcgbj-15/myobj 2022-09-09T22:33:31.317 DEBUG:boto:auth_path=/jkcgbj-15/myobj 2022-09-09T22:33:31.318 DEBUG:boto:path=/jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:31.319 DEBUG:boto:auth_path=/jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:31.321 DEBUG:boto:Method: HEAD 2022-09-09T22:33:31.322 DEBUG:boto:Path: /jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:31.323 DEBUG:boto:Data: 2022-09-09T22:33:31.325 DEBUG:boto:Headers: {} 2022-09-09T22:33:31.327 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:31.328 DEBUG:boto:Port: 8001 2022-09-09T22:33:31.329 DEBUG:boto:Params: {} 2022-09-09T22:33:31.330 DEBUG:boto:Token: None 2022-09-09T22:33:31.331 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:31 GMT /jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:31.332 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:eWK34oa1lR9nngEBm+bVk7nDZL0= 2022-09-09T22:33:31.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:eWK34oa1lR9nngEBm+bVk7nDZL0=', 'Content-Length': '0'} 2022-09-09T22:33:31.340 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f31a0baa998eb2c-00631bbf3b-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:31.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:33:31.343 DEBUG:boto:path=/jkcgbj-15/myobj 2022-09-09T22:33:31.344 DEBUG:boto:auth_path=/jkcgbj-15/myobj 2022-09-09T22:33:31.345 DEBUG:boto:path=/jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:31.347 DEBUG:boto:auth_path=/jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:31.348 DEBUG:boto:Method: GET 2022-09-09T22:33:31.349 DEBUG:boto:Path: /jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:31.350 DEBUG:boto:Data: 2022-09-09T22:33:31.351 DEBUG:boto:Headers: {} 2022-09-09T22:33:31.352 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:31.353 DEBUG:boto:Port: 8000 2022-09-09T22:33:31.353 DEBUG:boto:Params: {} 2022-09-09T22:33:31.354 DEBUG:boto:Token: None 2022-09-09T22:33:31.355 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:31 GMT /jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:31.355 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:r4sDC5jp30VZKl8Lg4//LO7ScOo= 2022-09-09T22:33:31.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:r4sDC5jp30VZKl8Lg4//LO7ScOo=', 'Content-Length': '0'} 2022-09-09T22:33:31.359 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c2139a5a21afa7d-00631bbf3b-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:31.360 DEBUG:boto:path=/jkcgbj-15/myobj 2022-09-09T22:33:31.361 DEBUG:boto:auth_path=/jkcgbj-15/myobj 2022-09-09T22:33:31.361 DEBUG:boto:path=/jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:31.362 DEBUG:boto:auth_path=/jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:31.362 DEBUG:boto:Method: GET 2022-09-09T22:33:31.363 DEBUG:boto:Path: /jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:31.363 DEBUG:boto:Data: 2022-09-09T22:33:31.364 DEBUG:boto:Headers: {} 2022-09-09T22:33:31.365 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:31.365 DEBUG:boto:Port: 8001 2022-09-09T22:33:31.366 DEBUG:boto:Params: {} 2022-09-09T22:33:31.366 DEBUG:boto:Token: None 2022-09-09T22:33:31.367 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:31 GMT /jkcgbj-15/myobj?versionId=null 2022-09-09T22:33:31.368 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:r4sDC5jp30VZKl8Lg4//LO7ScOo= 2022-09-09T22:33:31.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:r4sDC5jp30VZKl8Lg4//LO7ScOo=', 'Content-Length': '0'} 2022-09-09T22:33:31.372 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c4922feca0cb39fb-00631bbf3b-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:31.414 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:33:31.415 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:33:31.416 DEBUG:boto:path=/jkcgbj-15/?prefix=myobj 2022-09-09T22:33:31.416 DEBUG:boto:auth_path=/jkcgbj-15/?prefix=myobj 2022-09-09T22:33:31.417 DEBUG:boto:Method: GET 2022-09-09T22:33:31.417 DEBUG:boto:Path: /jkcgbj-15/?prefix=myobj 2022-09-09T22:33:31.418 DEBUG:boto:Data: 2022-09-09T22:33:31.419 DEBUG:boto:Headers: {} 2022-09-09T22:33:31.419 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:31.420 DEBUG:boto:Port: 8000 2022-09-09T22:33:31.421 DEBUG:boto:Params: {} 2022-09-09T22:33:31.422 DEBUG:boto:Token: None 2022-09-09T22:33:31.422 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:31 GMT /jkcgbj-15/ 2022-09-09T22:33:31.423 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4/cPXeemJuDA1OAMoNii5G11s+o= 2022-09-09T22:33:31.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4/cPXeemJuDA1OAMoNii5G11s+o=', 'Content-Length': '0'} 2022-09-09T22:33:31.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c64799fff4a63cc-00631bbf3b-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:31.432 DEBUG:boto:b'jkcgbj-15myobj1000falsemyobj2022-09-09T22:33:06.844Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:31.433 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:33:31.580 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:33:31.581 DEBUG:boto:path=/jkcgbj-15/?prefix=myobj 2022-09-09T22:33:31.584 DEBUG:boto:auth_path=/jkcgbj-15/?prefix=myobj 2022-09-09T22:33:31.585 DEBUG:boto:Method: GET 2022-09-09T22:33:31.586 DEBUG:boto:Path: /jkcgbj-15/?prefix=myobj 2022-09-09T22:33:31.587 DEBUG:boto:Data: 2022-09-09T22:33:31.588 DEBUG:boto:Headers: {} 2022-09-09T22:33:31.589 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:31.590 DEBUG:boto:Port: 8001 2022-09-09T22:33:31.590 DEBUG:boto:Params: {} 2022-09-09T22:33:31.592 DEBUG:boto:Token: None 2022-09-09T22:33:31.593 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:31 GMT /jkcgbj-15/ 2022-09-09T22:33:31.595 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4/cPXeemJuDA1OAMoNii5G11s+o= 2022-09-09T22:33:31.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4/cPXeemJuDA1OAMoNii5G11s+o=', 'Content-Length': '0'} 2022-09-09T22:33:31.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000287c0b15ad87e189-00631bbf3b-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:31.611 DEBUG:boto:b'jkcgbj-15myobj1000falsemyobj2022-09-09T22:33:06.844Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:31.612 DEBUG:boto:path=/jkcgbj-15/myobj 2022-09-09T22:33:31.613 DEBUG:boto:auth_path=/jkcgbj-15/myobj 2022-09-09T22:33:31.614 DEBUG:boto:Method: HEAD 2022-09-09T22:33:31.615 DEBUG:boto:Path: /jkcgbj-15/myobj 2022-09-09T22:33:31.616 DEBUG:boto:Data: 2022-09-09T22:33:31.617 DEBUG:boto:Headers: {} 2022-09-09T22:33:31.617 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:31.618 DEBUG:boto:Port: 8000 2022-09-09T22:33:31.619 DEBUG:boto:Params: {} 2022-09-09T22:33:31.619 DEBUG:boto:Token: None 2022-09-09T22:33:31.620 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:31 GMT /jkcgbj-15/myobj 2022-09-09T22:33:31.620 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:tackmTfeWTCQdb+xaTTqqLy1m5M= 2022-09-09T22:33:31.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:tackmTfeWTCQdb+xaTTqqLy1m5M=', 'Content-Length': '0'} 2022-09-09T22:33:31.624 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ab2ea9925c56e71-00631bbf3b-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:31.624 DEBUG:boto:path=/jkcgbj-15/myobj 2022-09-09T22:33:31.625 DEBUG:boto:auth_path=/jkcgbj-15/myobj 2022-09-09T22:33:31.626 DEBUG:boto:Method: HEAD 2022-09-09T22:33:31.627 DEBUG:boto:Path: /jkcgbj-15/myobj 2022-09-09T22:33:31.628 DEBUG:boto:Data: 2022-09-09T22:33:31.631 DEBUG:boto:Headers: {} 2022-09-09T22:33:31.632 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:31.632 DEBUG:boto:Port: 8001 2022-09-09T22:33:31.633 DEBUG:boto:Params: {} 2022-09-09T22:33:31.634 DEBUG:boto:Token: None 2022-09-09T22:33:31.636 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:31 GMT /jkcgbj-15/myobj 2022-09-09T22:33:31.638 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:tackmTfeWTCQdb+xaTTqqLy1m5M= 2022-09-09T22:33:31.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:tackmTfeWTCQdb+xaTTqqLy1m5M=', 'Content-Length': '0'} 2022-09-09T22:33:31.649 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef536e63df9edc87-00631bbf3b-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:31.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-09T22:33:31.654 DEBUG:boto:path=/jkcgbj-15/myobj 2022-09-09T22:33:31.656 DEBUG:boto:auth_path=/jkcgbj-15/myobj 2022-09-09T22:33:31.658 DEBUG:boto:Method: GET 2022-09-09T22:33:31.659 DEBUG:boto:Path: /jkcgbj-15/myobj 2022-09-09T22:33:31.660 DEBUG:boto:Data: 2022-09-09T22:33:31.661 DEBUG:boto:Headers: {} 2022-09-09T22:33:31.662 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:31.662 DEBUG:boto:Port: 8000 2022-09-09T22:33:31.663 DEBUG:boto:Params: {} 2022-09-09T22:33:31.664 DEBUG:boto:Token: None 2022-09-09T22:33:31.664 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:31 GMT /jkcgbj-15/myobj 2022-09-09T22:33:31.665 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nC+5s7aYF2GBFzSSjYNgHxr02cI= 2022-09-09T22:33:31.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nC+5s7aYF2GBFzSSjYNgHxr02cI=', 'Content-Length': '0'} 2022-09-09T22:33:31.679 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f7dae050a62970c-00631bbf3b-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:31.680 DEBUG:boto:path=/jkcgbj-15/myobj 2022-09-09T22:33:31.681 DEBUG:boto:auth_path=/jkcgbj-15/myobj 2022-09-09T22:33:31.681 DEBUG:boto:Method: GET 2022-09-09T22:33:31.682 DEBUG:boto:Path: /jkcgbj-15/myobj 2022-09-09T22:33:31.683 DEBUG:boto:Data: 2022-09-09T22:33:31.683 DEBUG:boto:Headers: {} 2022-09-09T22:33:31.684 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:31.685 DEBUG:boto:Port: 8001 2022-09-09T22:33:31.686 DEBUG:boto:Params: {} 2022-09-09T22:33:31.686 DEBUG:boto:Token: None 2022-09-09T22:33:31.688 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:31 GMT /jkcgbj-15/myobj 2022-09-09T22:33:31.689 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nC+5s7aYF2GBFzSSjYNgHxr02cI= 2022-09-09T22:33:31.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nC+5s7aYF2GBFzSSjYNgHxr02cI=', 'Content-Length': '0'} 2022-09-09T22:33:31.700 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005847de43cb846cce-00631bbf3b-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:31.706 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:33:31.707 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:33:31.708 DEBUG:boto:path=/jkcgbj-15/?prefix=myobj 2022-09-09T22:33:31.711 DEBUG:boto:auth_path=/jkcgbj-15/?prefix=myobj 2022-09-09T22:33:31.713 DEBUG:boto:Method: GET 2022-09-09T22:33:31.715 DEBUG:boto:Path: /jkcgbj-15/?prefix=myobj 2022-09-09T22:33:31.716 DEBUG:boto:Data: 2022-09-09T22:33:31.717 DEBUG:boto:Headers: {} 2022-09-09T22:33:31.718 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:33:31.720 DEBUG:boto:Port: 8000 2022-09-09T22:33:31.721 DEBUG:boto:Params: {} 2022-09-09T22:33:31.723 DEBUG:boto:Token: None 2022-09-09T22:33:31.723 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:31 GMT /jkcgbj-15/ 2022-09-09T22:33:31.724 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4/cPXeemJuDA1OAMoNii5G11s+o= 2022-09-09T22:33:31.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4/cPXeemJuDA1OAMoNii5G11s+o=', 'Content-Length': '0'} 2022-09-09T22:33:31.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a76708fa1d633f2a-00631bbf3b-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:31.737 DEBUG:boto:b'jkcgbj-15myobj1000falsemyobj2022-09-09T22:33:06.844Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:31.738 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:33:31.739 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:33:31.739 DEBUG:boto:path=/jkcgbj-15/?prefix=myobj 2022-09-09T22:33:31.740 DEBUG:boto:auth_path=/jkcgbj-15/?prefix=myobj 2022-09-09T22:33:31.741 DEBUG:boto:Method: GET 2022-09-09T22:33:31.742 DEBUG:boto:Path: /jkcgbj-15/?prefix=myobj 2022-09-09T22:33:31.742 DEBUG:boto:Data: 2022-09-09T22:33:31.743 DEBUG:boto:Headers: {} 2022-09-09T22:33:31.744 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:33:31.745 DEBUG:boto:Port: 8001 2022-09-09T22:33:31.745 DEBUG:boto:Params: {} 2022-09-09T22:33:31.746 DEBUG:boto:Token: None 2022-09-09T22:33:31.747 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:33:31 GMT /jkcgbj-15/ 2022-09-09T22:33:31.748 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4/cPXeemJuDA1OAMoNii5G11s+o= 2022-09-09T22:33:31.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4/cPXeemJuDA1OAMoNii5G11s+o=', 'Content-Length': '0'} 2022-09-09T22:33:31.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006dbaa22138eb7955-00631bbf3b-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:33:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:31.760 DEBUG:boto:b'jkcgbj-15myobj1000falsemyobj2022-09-09T22:33:06.844Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:33:31.761 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-15 zones={test-zone2, test-zone3} 2022-09-09T22:33:31.762 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:33:31.763 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:33:31.764 DEBUG:boto:Method: HEAD 2022-09-09T22:33:31.764 DEBUG:boto:Path: /jkcgbj-13/ 2022-09-09T22:33:31.765 DEBUG:boto:Data: 2022-09-09T22:33:31.765 DEBUG:boto:Headers: {} 2022-09-09T22:33:31.766 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:31.767 DEBUG:boto:Port: 8000 2022-09-09T22:33:31.768 DEBUG:boto:Params: {} 2022-09-09T22:33:31.769 DEBUG:boto:Token: None 2022-09-09T22:33:31.771 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:31 GMT /jkcgbj-13/ 2022-09-09T22:33:31.772 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dvv5V68Dz+eXdvBvp0JHjQ8g+ms= 2022-09-09T22:33:31.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dvv5V68Dz+eXdvBvp0JHjQ8g+ms=', 'Content-Length': '0'} 2022-09-09T22:33:31.780 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b28c456be415198d-00631bbf3b-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:33:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:31.782 DEBUG:boto:path=/jkcgbj-13/myobj 2022-09-09T22:33:31.783 DEBUG:boto:auth_path=/jkcgbj-13/myobj 2022-09-09T22:33:31.785 DEBUG:boto:Method: HEAD 2022-09-09T22:33:31.787 DEBUG:boto:Path: /jkcgbj-13/myobj 2022-09-09T22:33:31.788 DEBUG:boto:Data: 2022-09-09T22:33:31.788 DEBUG:boto:Headers: {} 2022-09-09T22:33:31.789 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:31.790 DEBUG:boto:Port: 8000 2022-09-09T22:33:31.791 DEBUG:boto:Params: {} 2022-09-09T22:33:31.795 DEBUG:boto:Token: None 2022-09-09T22:33:31.795 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:33:31 GMT /jkcgbj-13/myobj 2022-09-09T22:33:31.796 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:BAmQWs/tJCY4GAkJw1tdzX8QTKI= 2022-09-09T22:33:31.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:BAmQWs/tJCY4GAkJw1tdzX8QTKI=', 'Content-Length': '0'} 2022-09-09T22:33:31.801 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb75afaa08ec2653-00631bbf3b-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:33:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:31.802 DEBUG:boto:path=/jkcgbj-13/myobj 2022-09-09T22:33:31.803 DEBUG:boto:auth_path=/jkcgbj-13/myobj 2022-09-09T22:33:31.804 DEBUG:boto:Method: DELETE 2022-09-09T22:33:31.804 DEBUG:boto:Path: /jkcgbj-13/myobj 2022-09-09T22:33:31.805 DEBUG:boto:Data: 2022-09-09T22:33:31.806 DEBUG:boto:Headers: {} 2022-09-09T22:33:31.807 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:33:31.808 DEBUG:boto:Port: 8000 2022-09-09T22:33:31.810 DEBUG:boto:Params: {} 2022-09-09T22:33:31.811 DEBUG:boto:Token: None 2022-09-09T22:33:31.812 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:33:31 GMT /jkcgbj-13/myobj 2022-09-09T22:33:31.814 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7/nqKXykeehv6sjV9VV8U1NLf8k= 2022-09-09T22:33:31.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:33:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7/nqKXykeehv6sjV9VV8U1NLf8k=', 'Content-Length': '0'} 2022-09-09T22:33:31.825 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c39d25aa75b67bec-00631bbf3b-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:33:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:33:31.827 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:33:31.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:33:31.829 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:33:31.887 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:33:31.888 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:33:31.971 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:33:31.966+0000 7f3a2e866dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:33:31.972 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000001.4.5, , , , , , , , , ] 2022-09-09T22:33:31.972 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000002.8.5, , , , , , , , , ] 2022-09-09T22:33:36.974 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:33:36.970+0000 7f3a2e866dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:33:36.975 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000001.4.5, , , , , , , , , ] 2022-09-09T22:33:36.976 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000002.8.5, , , , , , , , , ] 2022-09-09T22:33:41.976 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:33:41.970+0000 7f3a2e866dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:33:41.976 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000001.4.5, , , , , , , , , ] 2022-09-09T22:33:41.977 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000002.8.5, , , , , , , , , ] 2022-09-09T22:33:46.979 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:33:46.973+0000 7f3a2e866dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:33:46.980 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000001.4.5, , , , , , , , , ] 2022-09-09T22:33:46.981 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000002.8.5, , , , , , , , , ] 2022-09-09T22:33:51.982 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:33:51.977+0000 7f3a2e866dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:33:51.983 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000001.4.5, , , , , , , , , ] 2022-09-09T22:33:51.983 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000002.8.5, , , , , , , , , ] 2022-09-09T22:33:56.985 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:33:56.981+0000 7f3a2e866dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:33:56.987 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000001.4.5, , , , , , , , , ] 2022-09-09T22:33:56.987 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000002.8.5, , , , , , , , , ] 2022-09-09T22:34:01.988 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:34:01.981+0000 7f3a2e866dc0 1 bucket sync caught up with source: 2022-09-09T22:34:01.989 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000002.8.5, , , , , , , , , ] 2022-09-09T22:34:01.989 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000002.8.5, , , , , , , , , ] 2022-09-09T22:34:01.990 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:34:01.981+0000 7f3a2e866dc0 0 bucket checkpoint complete 2022-09-09T22:34:02.018 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-13 zones={test-zone2, test-zone1} 2022-09-09T22:34:02.018 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:34:02.019 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:34:02.020 DEBUG:boto:Method: HEAD 2022-09-09T22:34:02.021 DEBUG:boto:Path: /jkcgbj-13/ 2022-09-09T22:34:02.022 DEBUG:boto:Data: 2022-09-09T22:34:02.023 DEBUG:boto:Headers: {} 2022-09-09T22:34:02.024 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:34:02.024 DEBUG:boto:Port: 8000 2022-09-09T22:34:02.025 DEBUG:boto:Params: {} 2022-09-09T22:34:02.026 DEBUG:boto:Token: None 2022-09-09T22:34:02.027 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:34:02 GMT /jkcgbj-13/ 2022-09-09T22:34:02.027 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:GvvC1RJ/JknEl2t+4oBzFTi/mQ8= 2022-09-09T22:34:02.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:GvvC1RJ/JknEl2t+4oBzFTi/mQ8=', 'Content-Length': '0'} 2022-09-09T22:34:02.034 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003bbb29008a15dc35-00631bbf5a-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:02.035 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:34:02.036 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:34:02.036 DEBUG:boto:Method: HEAD 2022-09-09T22:34:02.037 DEBUG:boto:Path: /jkcgbj-13/ 2022-09-09T22:34:02.038 DEBUG:boto:Data: 2022-09-09T22:34:02.038 DEBUG:boto:Headers: {} 2022-09-09T22:34:02.039 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:34:02.039 DEBUG:boto:Port: 8000 2022-09-09T22:34:02.040 DEBUG:boto:Params: {} 2022-09-09T22:34:02.040 DEBUG:boto:Token: None 2022-09-09T22:34:02.041 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:34:02 GMT /jkcgbj-13/ 2022-09-09T22:34:02.042 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:GvvC1RJ/JknEl2t+4oBzFTi/mQ8= 2022-09-09T22:34:02.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:GvvC1RJ/JknEl2t+4oBzFTi/mQ8=', 'Content-Length': '0'} 2022-09-09T22:34:02.048 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db6a8f9af44d455b-00631bbf5a-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:02.049 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:34:02.049 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:34:02.050 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:34:02.050 DEBUG:boto:path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:34:02.051 DEBUG:boto:auth_path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:34:02.052 DEBUG:boto:Method: GET 2022-09-09T22:34:02.053 DEBUG:boto:Path: /jkcgbj-13/?versions&max-keys=999 2022-09-09T22:34:02.053 DEBUG:boto:Data: 2022-09-09T22:34:02.054 DEBUG:boto:Headers: {} 2022-09-09T22:34:02.055 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:34:02.055 DEBUG:boto:Port: 8000 2022-09-09T22:34:02.056 DEBUG:boto:Params: {} 2022-09-09T22:34:02.057 DEBUG:boto:Token: None 2022-09-09T22:34:02.058 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:34:02 GMT /jkcgbj-13/?versions 2022-09-09T22:34:02.058 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TP06F7+fTibMHSBFWxZKb6Og9yA= 2022-09-09T22:34:02.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TP06F7+fTibMHSBFWxZKb6Og9yA=', 'Content-Length': '0'} 2022-09-09T22:34:02.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c606cbac38333f54-00631bbf5a-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:02.110 DEBUG:boto:b'jkcgbj-13999false' 2022-09-09T22:34:02.111 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:34:02.112 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:34:02.113 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:34:02.113 DEBUG:boto:path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:34:02.114 DEBUG:boto:auth_path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:34:02.115 DEBUG:boto:Method: GET 2022-09-09T22:34:02.116 DEBUG:boto:Path: /jkcgbj-13/?versions&max-keys=999 2022-09-09T22:34:02.117 DEBUG:boto:Data: 2022-09-09T22:34:02.118 DEBUG:boto:Headers: {} 2022-09-09T22:34:02.118 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:34:02.119 DEBUG:boto:Port: 8000 2022-09-09T22:34:02.120 DEBUG:boto:Params: {} 2022-09-09T22:34:02.121 DEBUG:boto:Token: None 2022-09-09T22:34:02.122 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:34:02 GMT /jkcgbj-13/?versions 2022-09-09T22:34:02.122 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TP06F7+fTibMHSBFWxZKb6Og9yA= 2022-09-09T22:34:02.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TP06F7+fTibMHSBFWxZKb6Og9yA=', 'Content-Length': '0'} 2022-09-09T22:34:02.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bda8214c5ba69557-00631bbf5a-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:02.130 DEBUG:boto:b'jkcgbj-13999false' 2022-09-09T22:34:02.131 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:34:02.133 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:34:02.134 DEBUG:boto:path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:34:02.135 DEBUG:boto:auth_path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:34:02.136 DEBUG:boto:Method: GET 2022-09-09T22:34:02.136 DEBUG:boto:Path: /jkcgbj-13/?versions&max-keys=999 2022-09-09T22:34:02.137 DEBUG:boto:Data: 2022-09-09T22:34:02.138 DEBUG:boto:Headers: {} 2022-09-09T22:34:02.138 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:34:02.139 DEBUG:boto:Port: 8000 2022-09-09T22:34:02.139 DEBUG:boto:Params: {} 2022-09-09T22:34:02.140 DEBUG:boto:Token: None 2022-09-09T22:34:02.141 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:34:02 GMT /jkcgbj-13/?versions 2022-09-09T22:34:02.141 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TP06F7+fTibMHSBFWxZKb6Og9yA= 2022-09-09T22:34:02.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TP06F7+fTibMHSBFWxZKb6Og9yA=', 'Content-Length': '0'} 2022-09-09T22:34:02.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001af9efff320f7bad-00631bbf5a-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:02.149 DEBUG:boto:b'jkcgbj-13999false' 2022-09-09T22:34:02.150 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:34:02.151 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:34:02.152 DEBUG:boto:path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:34:02.153 DEBUG:boto:auth_path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:34:02.154 DEBUG:boto:Method: GET 2022-09-09T22:34:02.155 DEBUG:boto:Path: /jkcgbj-13/?versions&max-keys=999 2022-09-09T22:34:02.156 DEBUG:boto:Data: 2022-09-09T22:34:02.157 DEBUG:boto:Headers: {} 2022-09-09T22:34:02.158 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:34:02.159 DEBUG:boto:Port: 8000 2022-09-09T22:34:02.159 DEBUG:boto:Params: {} 2022-09-09T22:34:02.160 DEBUG:boto:Token: None 2022-09-09T22:34:02.161 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:34:02 GMT /jkcgbj-13/?versions 2022-09-09T22:34:02.161 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TP06F7+fTibMHSBFWxZKb6Og9yA= 2022-09-09T22:34:02.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TP06F7+fTibMHSBFWxZKb6Og9yA=', 'Content-Length': '0'} 2022-09-09T22:34:02.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec85ffa9f093a9e3-00631bbf5a-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:02.170 DEBUG:boto:b'jkcgbj-13999false' 2022-09-09T22:34:02.171 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-13 zones={test-zone2, test-zone1} 2022-09-09T22:34:02.172 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:34:02.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:34:02.174 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:34:02.231 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:34:02.232 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:34:02.298 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:34:02.291+0000 7f1d62bccdc0 1 bucket sync caught up with source: 2022-09-09T22:34:02.298 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000002.8.5, , , , , , , , , ] 2022-09-09T22:34:02.299 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000002.8.5, , , , , , , , , ] 2022-09-09T22:34:02.299 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:34:02.291+0000 7f1d62bccdc0 0 bucket checkpoint complete 2022-09-09T22:34:02.323 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-13 zones={test-zone3, test-zone1} 2022-09-09T22:34:02.323 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:34:02.324 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:34:02.325 DEBUG:boto:Method: HEAD 2022-09-09T22:34:02.325 DEBUG:boto:Path: /jkcgbj-13/ 2022-09-09T22:34:02.326 DEBUG:boto:Data: 2022-09-09T22:34:02.326 DEBUG:boto:Headers: {} 2022-09-09T22:34:02.327 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:34:02.327 DEBUG:boto:Port: 8001 2022-09-09T22:34:02.328 DEBUG:boto:Params: {} 2022-09-09T22:34:02.329 DEBUG:boto:Token: None 2022-09-09T22:34:02.330 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:34:02 GMT /jkcgbj-13/ 2022-09-09T22:34:02.332 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:GvvC1RJ/JknEl2t+4oBzFTi/mQ8= 2022-09-09T22:34:02.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:GvvC1RJ/JknEl2t+4oBzFTi/mQ8=', 'Content-Length': '0'} 2022-09-09T22:34:02.338 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000076cfdabf3506fbf8-00631bbf5a-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:02.339 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:34:02.340 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:34:02.341 DEBUG:boto:Method: HEAD 2022-09-09T22:34:02.342 DEBUG:boto:Path: /jkcgbj-13/ 2022-09-09T22:34:02.343 DEBUG:boto:Data: 2022-09-09T22:34:02.343 DEBUG:boto:Headers: {} 2022-09-09T22:34:02.344 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:34:02.344 DEBUG:boto:Port: 8000 2022-09-09T22:34:02.345 DEBUG:boto:Params: {} 2022-09-09T22:34:02.345 DEBUG:boto:Token: None 2022-09-09T22:34:02.346 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:34:02 GMT /jkcgbj-13/ 2022-09-09T22:34:02.347 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:GvvC1RJ/JknEl2t+4oBzFTi/mQ8= 2022-09-09T22:34:02.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:GvvC1RJ/JknEl2t+4oBzFTi/mQ8=', 'Content-Length': '0'} 2022-09-09T22:34:02.352 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d58cdd284fe712be-00631bbf5a-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:02.353 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:34:02.354 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:34:02.354 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:34:02.355 DEBUG:boto:path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:34:02.355 DEBUG:boto:auth_path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:34:02.356 DEBUG:boto:Method: GET 2022-09-09T22:34:02.357 DEBUG:boto:Path: /jkcgbj-13/?versions&max-keys=999 2022-09-09T22:34:02.357 DEBUG:boto:Data: 2022-09-09T22:34:02.358 DEBUG:boto:Headers: {} 2022-09-09T22:34:02.360 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:34:02.360 DEBUG:boto:Port: 8001 2022-09-09T22:34:02.361 DEBUG:boto:Params: {} 2022-09-09T22:34:02.363 DEBUG:boto:Token: None 2022-09-09T22:34:02.364 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:34:02 GMT /jkcgbj-13/?versions 2022-09-09T22:34:02.365 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TP06F7+fTibMHSBFWxZKb6Og9yA= 2022-09-09T22:34:02.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TP06F7+fTibMHSBFWxZKb6Og9yA=', 'Content-Length': '0'} 2022-09-09T22:34:02.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007073c300be419ce1-00631bbf5a-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:02.374 DEBUG:boto:b'jkcgbj-13999false' 2022-09-09T22:34:02.375 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:34:02.376 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:34:02.377 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:34:02.379 DEBUG:boto:path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:34:02.380 DEBUG:boto:auth_path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:34:02.381 DEBUG:boto:Method: GET 2022-09-09T22:34:02.381 DEBUG:boto:Path: /jkcgbj-13/?versions&max-keys=999 2022-09-09T22:34:02.382 DEBUG:boto:Data: 2022-09-09T22:34:02.383 DEBUG:boto:Headers: {} 2022-09-09T22:34:02.384 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:34:02.385 DEBUG:boto:Port: 8000 2022-09-09T22:34:02.386 DEBUG:boto:Params: {} 2022-09-09T22:34:02.387 DEBUG:boto:Token: None 2022-09-09T22:34:02.388 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:34:02 GMT /jkcgbj-13/?versions 2022-09-09T22:34:02.390 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TP06F7+fTibMHSBFWxZKb6Og9yA= 2022-09-09T22:34:02.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TP06F7+fTibMHSBFWxZKb6Og9yA=', 'Content-Length': '0'} 2022-09-09T22:34:02.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b276be0dd996de4b-00631bbf5a-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:02.399 DEBUG:boto:b'jkcgbj-13999false' 2022-09-09T22:34:02.400 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:34:02.400 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:34:02.401 DEBUG:boto:path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:34:02.401 DEBUG:boto:auth_path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:34:02.402 DEBUG:boto:Method: GET 2022-09-09T22:34:02.402 DEBUG:boto:Path: /jkcgbj-13/?versions&max-keys=999 2022-09-09T22:34:02.403 DEBUG:boto:Data: 2022-09-09T22:34:02.403 DEBUG:boto:Headers: {} 2022-09-09T22:34:02.404 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:34:02.404 DEBUG:boto:Port: 8001 2022-09-09T22:34:02.405 DEBUG:boto:Params: {} 2022-09-09T22:34:02.406 DEBUG:boto:Token: None 2022-09-09T22:34:02.406 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:34:02 GMT /jkcgbj-13/?versions 2022-09-09T22:34:02.407 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TP06F7+fTibMHSBFWxZKb6Og9yA= 2022-09-09T22:34:02.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TP06F7+fTibMHSBFWxZKb6Og9yA=', 'Content-Length': '0'} 2022-09-09T22:34:02.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc2b1f54f583625d-00631bbf5a-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:02.414 DEBUG:boto:b'jkcgbj-13999false' 2022-09-09T22:34:02.415 DEBUG:boto:path=/jkcgbj-13/ 2022-09-09T22:34:02.416 DEBUG:boto:auth_path=/jkcgbj-13/ 2022-09-09T22:34:02.416 DEBUG:boto:path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:34:02.417 DEBUG:boto:auth_path=/jkcgbj-13/?versions&max-keys=999 2022-09-09T22:34:02.417 DEBUG:boto:Method: GET 2022-09-09T22:34:02.418 DEBUG:boto:Path: /jkcgbj-13/?versions&max-keys=999 2022-09-09T22:34:02.418 DEBUG:boto:Data: 2022-09-09T22:34:02.419 DEBUG:boto:Headers: {} 2022-09-09T22:34:02.419 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:34:02.420 DEBUG:boto:Port: 8000 2022-09-09T22:34:02.420 DEBUG:boto:Params: {} 2022-09-09T22:34:02.421 DEBUG:boto:Token: None 2022-09-09T22:34:02.422 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:34:02 GMT /jkcgbj-13/?versions 2022-09-09T22:34:02.422 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TP06F7+fTibMHSBFWxZKb6Og9yA= 2022-09-09T22:34:02.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TP06F7+fTibMHSBFWxZKb6Og9yA=', 'Content-Length': '0'} 2022-09-09T22:34:02.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ab2155eb86adc07-00631bbf5a-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:02.430 DEBUG:boto:b'jkcgbj-13999false' 2022-09-09T22:34:02.430 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-13 zones={test-zone3, test-zone1} 2022-09-09T22:34:02.431 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:34:02.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:34:02.432 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:34:02.489 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:34:02.489 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:34:02.565 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:34:02.561+0000 7ff5ec3a8dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:34:02.565 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000001.4.5, , , , , , , , , ] 2022-09-09T22:34:02.566 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000002.8.5, , , , , , , , , ] 2022-09-09T22:34:07.568 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:34:07.561+0000 7ff5ec3a8dc0 1 bucket sync caught up with source: 2022-09-09T22:34:07.568 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000002.8.5, , , , , , , , , ] 2022-09-09T22:34:07.569 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000002.8.5, , , , , , , , , ] 2022-09-09T22:34:07.570 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:34:07.561+0000 7ff5ec3a8dc0 0 bucket checkpoint complete 2022-09-09T22:34:07.595 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:34:07.596 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:34:07.597 DEBUG:boto:Method: HEAD 2022-09-09T22:34:07.597 DEBUG:boto:Path: /jkcgbj-14/ 2022-09-09T22:34:07.598 DEBUG:boto:Data: 2022-09-09T22:34:07.599 DEBUG:boto:Headers: {} 2022-09-09T22:34:07.601 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:34:07.602 DEBUG:boto:Port: 8000 2022-09-09T22:34:07.603 DEBUG:boto:Params: {} 2022-09-09T22:34:07.603 DEBUG:boto:Token: None 2022-09-09T22:34:07.604 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:34:07 GMT /jkcgbj-14/ 2022-09-09T22:34:07.604 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8WUDN/4I/DBaa0xKgufrx4x0g3E= 2022-09-09T22:34:07.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8WUDN/4I/DBaa0xKgufrx4x0g3E=', 'Content-Length': '0'} 2022-09-09T22:34:07.609 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010958c685214625e-00631bbf5f-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:34:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:07.610 DEBUG:boto:path=/jkcgbj-14/myobj 2022-09-09T22:34:07.611 DEBUG:boto:auth_path=/jkcgbj-14/myobj 2022-09-09T22:34:07.611 DEBUG:boto:Method: HEAD 2022-09-09T22:34:07.612 DEBUG:boto:Path: /jkcgbj-14/myobj 2022-09-09T22:34:07.613 DEBUG:boto:Data: 2022-09-09T22:34:07.613 DEBUG:boto:Headers: {} 2022-09-09T22:34:07.614 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:34:07.614 DEBUG:boto:Port: 8000 2022-09-09T22:34:07.615 DEBUG:boto:Params: {} 2022-09-09T22:34:07.616 DEBUG:boto:Token: None 2022-09-09T22:34:07.617 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:34:07 GMT /jkcgbj-14/myobj 2022-09-09T22:34:07.618 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+4kGu1q5QhuTcGkCdFKHoJotqmA= 2022-09-09T22:34:07.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+4kGu1q5QhuTcGkCdFKHoJotqmA=', 'Content-Length': '0'} 2022-09-09T22:34:07.623 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c373c227bbd3aee3-00631bbf5f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:34:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:07.624 DEBUG:boto:path=/jkcgbj-14/myobj 2022-09-09T22:34:07.625 DEBUG:boto:auth_path=/jkcgbj-14/myobj 2022-09-09T22:34:07.625 DEBUG:boto:Method: DELETE 2022-09-09T22:34:07.626 DEBUG:boto:Path: /jkcgbj-14/myobj 2022-09-09T22:34:07.627 DEBUG:boto:Data: 2022-09-09T22:34:07.627 DEBUG:boto:Headers: {} 2022-09-09T22:34:07.628 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:34:07.628 DEBUG:boto:Port: 8000 2022-09-09T22:34:07.629 DEBUG:boto:Params: {} 2022-09-09T22:34:07.630 DEBUG:boto:Token: None 2022-09-09T22:34:07.630 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:34:07 GMT /jkcgbj-14/myobj 2022-09-09T22:34:07.631 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ne9cm+DU3a5B8LRBBMQC2GkMchI= 2022-09-09T22:34:07.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ne9cm+DU3a5B8LRBBMQC2GkMchI=', 'Content-Length': '0'} 2022-09-09T22:34:07.646 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000064ee765b3381bf5-00631bbf5f-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:34:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:07.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:34:07.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:34:07.653 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:34:07.707 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:34:07.708 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:34:07.773 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:34:07.767+0000 7f8af42fadc0 1 waiting for incremental sync to catch up: 2022-09-09T22:34:07.774 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000001.11.5, , , , , , , , , ] 2022-09-09T22:34:07.774 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000002.17.5, , , , , , , , , ] 2022-09-09T22:34:12.776 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:34:12.767+0000 7f8af42fadc0 1 waiting for incremental sync to catch up: 2022-09-09T22:34:12.777 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000001.11.5, , , , , , , , , ] 2022-09-09T22:34:12.777 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000002.17.5, , , , , , , , , ] 2022-09-09T22:34:17.779 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:34:17.771+0000 7f8af42fadc0 1 waiting for incremental sync to catch up: 2022-09-09T22:34:17.779 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000001.11.5, , , , , , , , , ] 2022-09-09T22:34:17.780 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000002.17.5, , , , , , , , , ] 2022-09-09T22:34:22.782 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:34:22.774+0000 7f8af42fadc0 1 bucket sync caught up with source: 2022-09-09T22:34:22.782 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000002.17.5, , , , , , , , , ] 2022-09-09T22:34:22.783 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000002.17.5, , , , , , , , , ] 2022-09-09T22:34:22.783 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:34:22.774+0000 7f8af42fadc0 0 bucket checkpoint complete 2022-09-09T22:34:22.812 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-14 zones={test-zone1, test-zone2} 2022-09-09T22:34:22.813 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:34:22.814 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:34:22.815 DEBUG:boto:Method: HEAD 2022-09-09T22:34:22.816 DEBUG:boto:Path: /jkcgbj-14/ 2022-09-09T22:34:22.816 DEBUG:boto:Data: 2022-09-09T22:34:22.817 DEBUG:boto:Headers: {} 2022-09-09T22:34:22.817 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:34:22.818 DEBUG:boto:Port: 8000 2022-09-09T22:34:22.819 DEBUG:boto:Params: {} 2022-09-09T22:34:22.820 DEBUG:boto:Token: None 2022-09-09T22:34:22.821 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:34:22 GMT /jkcgbj-14/ 2022-09-09T22:34:22.821 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7RRumtXrAZ6uWKWhqcus8pvWnzA= 2022-09-09T22:34:22.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:22 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7RRumtXrAZ6uWKWhqcus8pvWnzA=', 'Content-Length': '0'} 2022-09-09T22:34:22.827 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a959c09ae183b8b4-00631bbf6e-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:34:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:22.828 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:34:22.828 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:34:22.829 DEBUG:boto:Method: HEAD 2022-09-09T22:34:22.829 DEBUG:boto:Path: /jkcgbj-14/ 2022-09-09T22:34:22.830 DEBUG:boto:Data: 2022-09-09T22:34:22.831 DEBUG:boto:Headers: {} 2022-09-09T22:34:22.832 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:34:22.832 DEBUG:boto:Port: 8000 2022-09-09T22:34:22.833 DEBUG:boto:Params: {} 2022-09-09T22:34:22.833 DEBUG:boto:Token: None 2022-09-09T22:34:22.834 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:34:22 GMT /jkcgbj-14/ 2022-09-09T22:34:22.835 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7RRumtXrAZ6uWKWhqcus8pvWnzA= 2022-09-09T22:34:22.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:22 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7RRumtXrAZ6uWKWhqcus8pvWnzA=', 'Content-Length': '0'} 2022-09-09T22:34:22.840 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d21535c494a88047-00631bbf6e-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:34:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:22.841 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:34:22.841 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:34:22.842 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:34:22.842 DEBUG:boto:path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:34:22.843 DEBUG:boto:auth_path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:34:22.843 DEBUG:boto:Method: GET 2022-09-09T22:34:22.844 DEBUG:boto:Path: /jkcgbj-14/?versions&max-keys=999 2022-09-09T22:34:22.844 DEBUG:boto:Data: 2022-09-09T22:34:22.845 DEBUG:boto:Headers: {} 2022-09-09T22:34:22.846 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:34:22.846 DEBUG:boto:Port: 8000 2022-09-09T22:34:22.847 DEBUG:boto:Params: {} 2022-09-09T22:34:22.848 DEBUG:boto:Token: None 2022-09-09T22:34:22.848 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:34:22 GMT /jkcgbj-14/?versions 2022-09-09T22:34:22.849 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Int6zaPZgw4Tf3bmsO+uoMzu6uM= 2022-09-09T22:34:22.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:22 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Int6zaPZgw4Tf3bmsO+uoMzu6uM=', 'Content-Length': '0'} 2022-09-09T22:34:22.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000634a1be89fb5b8cb-00631bbf6e-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:34:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:22.898 DEBUG:boto:b'jkcgbj-14999false' 2022-09-09T22:34:22.900 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:34:22.900 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:34:22.901 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:34:22.902 DEBUG:boto:path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:34:22.903 DEBUG:boto:auth_path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:34:22.903 DEBUG:boto:Method: GET 2022-09-09T22:34:22.904 DEBUG:boto:Path: /jkcgbj-14/?versions&max-keys=999 2022-09-09T22:34:22.905 DEBUG:boto:Data: 2022-09-09T22:34:22.906 DEBUG:boto:Headers: {} 2022-09-09T22:34:22.908 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:34:22.909 DEBUG:boto:Port: 8000 2022-09-09T22:34:22.910 DEBUG:boto:Params: {} 2022-09-09T22:34:22.911 DEBUG:boto:Token: None 2022-09-09T22:34:22.912 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:34:22 GMT /jkcgbj-14/?versions 2022-09-09T22:34:22.913 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Int6zaPZgw4Tf3bmsO+uoMzu6uM= 2022-09-09T22:34:22.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:22 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Int6zaPZgw4Tf3bmsO+uoMzu6uM=', 'Content-Length': '0'} 2022-09-09T22:34:22.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039035743b09fc36f-00631bbf6e-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:34:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:22.921 DEBUG:boto:b'jkcgbj-14999false' 2022-09-09T22:34:22.922 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:34:22.923 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:34:22.924 DEBUG:boto:path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:34:22.925 DEBUG:boto:auth_path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:34:22.925 DEBUG:boto:Method: GET 2022-09-09T22:34:22.926 DEBUG:boto:Path: /jkcgbj-14/?versions&max-keys=999 2022-09-09T22:34:22.927 DEBUG:boto:Data: 2022-09-09T22:34:22.927 DEBUG:boto:Headers: {} 2022-09-09T22:34:22.928 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:34:22.929 DEBUG:boto:Port: 8000 2022-09-09T22:34:22.929 DEBUG:boto:Params: {} 2022-09-09T22:34:22.930 DEBUG:boto:Token: None 2022-09-09T22:34:22.931 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:34:22 GMT /jkcgbj-14/?versions 2022-09-09T22:34:22.932 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Int6zaPZgw4Tf3bmsO+uoMzu6uM= 2022-09-09T22:34:22.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:22 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Int6zaPZgw4Tf3bmsO+uoMzu6uM=', 'Content-Length': '0'} 2022-09-09T22:34:22.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cab05308517878b4-00631bbf6e-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:34:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:22.946 DEBUG:boto:b'jkcgbj-14999false' 2022-09-09T22:34:22.947 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:34:22.947 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:34:22.948 DEBUG:boto:path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:34:22.949 DEBUG:boto:auth_path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:34:22.950 DEBUG:boto:Method: GET 2022-09-09T22:34:22.951 DEBUG:boto:Path: /jkcgbj-14/?versions&max-keys=999 2022-09-09T22:34:22.951 DEBUG:boto:Data: 2022-09-09T22:34:22.952 DEBUG:boto:Headers: {} 2022-09-09T22:34:22.953 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:34:22.953 DEBUG:boto:Port: 8000 2022-09-09T22:34:22.954 DEBUG:boto:Params: {} 2022-09-09T22:34:22.955 DEBUG:boto:Token: None 2022-09-09T22:34:22.956 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:34:22 GMT /jkcgbj-14/?versions 2022-09-09T22:34:22.957 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Int6zaPZgw4Tf3bmsO+uoMzu6uM= 2022-09-09T22:34:22.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:22 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Int6zaPZgw4Tf3bmsO+uoMzu6uM=', 'Content-Length': '0'} 2022-09-09T22:34:22.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e689ac74970f35c8-00631bbf6e-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:34:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:22.971 DEBUG:boto:b'jkcgbj-14999false' 2022-09-09T22:34:22.972 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-14 zones={test-zone1, test-zone2} 2022-09-09T22:34:22.972 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:34:22.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:34:22.974 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:34:23.035 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:34:23.035 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:34:23.103 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:34:23.094+0000 7ff1f7b60dc0 1 bucket sync caught up with source: 2022-09-09T22:34:23.104 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000002.17.5, , , , , , , , , ] 2022-09-09T22:34:23.105 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000002.17.5, , , , , , , , , ] 2022-09-09T22:34:23.105 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:34:23.094+0000 7ff1f7b60dc0 0 bucket checkpoint complete 2022-09-09T22:34:23.129 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-14 zones={test-zone3, test-zone2} 2022-09-09T22:34:23.130 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:34:23.131 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:34:23.132 DEBUG:boto:Method: HEAD 2022-09-09T22:34:23.133 DEBUG:boto:Path: /jkcgbj-14/ 2022-09-09T22:34:23.134 DEBUG:boto:Data: 2022-09-09T22:34:23.135 DEBUG:boto:Headers: {} 2022-09-09T22:34:23.136 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:34:23.136 DEBUG:boto:Port: 8001 2022-09-09T22:34:23.137 DEBUG:boto:Params: {} 2022-09-09T22:34:23.138 DEBUG:boto:Token: None 2022-09-09T22:34:23.138 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:34:23 GMT /jkcgbj-14/ 2022-09-09T22:34:23.139 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:F0wdlRZC45qbdn433+2+OPu1ayA= 2022-09-09T22:34:23.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:F0wdlRZC45qbdn433+2+OPu1ayA=', 'Content-Length': '0'} 2022-09-09T22:34:23.149 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d50fa67698dc553-00631bbf6f-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:34:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:23.151 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:34:23.152 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:34:23.153 DEBUG:boto:Method: HEAD 2022-09-09T22:34:23.154 DEBUG:boto:Path: /jkcgbj-14/ 2022-09-09T22:34:23.154 DEBUG:boto:Data: 2022-09-09T22:34:23.155 DEBUG:boto:Headers: {} 2022-09-09T22:34:23.156 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:34:23.156 DEBUG:boto:Port: 8000 2022-09-09T22:34:23.157 DEBUG:boto:Params: {} 2022-09-09T22:34:23.158 DEBUG:boto:Token: None 2022-09-09T22:34:23.159 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:34:23 GMT /jkcgbj-14/ 2022-09-09T22:34:23.160 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:F0wdlRZC45qbdn433+2+OPu1ayA= 2022-09-09T22:34:23.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:F0wdlRZC45qbdn433+2+OPu1ayA=', 'Content-Length': '0'} 2022-09-09T22:34:23.170 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068ab5b44f13e648d-00631bbf6f-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:34:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:23.172 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:34:23.173 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:34:23.174 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:34:23.175 DEBUG:boto:path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:34:23.176 DEBUG:boto:auth_path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:34:23.177 DEBUG:boto:Method: GET 2022-09-09T22:34:23.178 DEBUG:boto:Path: /jkcgbj-14/?versions&max-keys=999 2022-09-09T22:34:23.179 DEBUG:boto:Data: 2022-09-09T22:34:23.180 DEBUG:boto:Headers: {} 2022-09-09T22:34:23.181 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:34:23.182 DEBUG:boto:Port: 8001 2022-09-09T22:34:23.183 DEBUG:boto:Params: {} 2022-09-09T22:34:23.184 DEBUG:boto:Token: None 2022-09-09T22:34:23.185 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:34:23 GMT /jkcgbj-14/?versions 2022-09-09T22:34:23.186 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:XQ3jlFD2UwzWy6ZDMZzWtLuQSHo= 2022-09-09T22:34:23.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:XQ3jlFD2UwzWy6ZDMZzWtLuQSHo=', 'Content-Length': '0'} 2022-09-09T22:34:23.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b65b08f1cbe73431-00631bbf6f-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:34:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:23.243 DEBUG:boto:b'jkcgbj-14999false' 2022-09-09T22:34:23.244 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:34:23.244 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:34:23.245 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:34:23.246 DEBUG:boto:path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:34:23.246 DEBUG:boto:auth_path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:34:23.247 DEBUG:boto:Method: GET 2022-09-09T22:34:23.247 DEBUG:boto:Path: /jkcgbj-14/?versions&max-keys=999 2022-09-09T22:34:23.248 DEBUG:boto:Data: 2022-09-09T22:34:23.248 DEBUG:boto:Headers: {} 2022-09-09T22:34:23.249 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:34:23.249 DEBUG:boto:Port: 8000 2022-09-09T22:34:23.250 DEBUG:boto:Params: {} 2022-09-09T22:34:23.251 DEBUG:boto:Token: None 2022-09-09T22:34:23.251 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:34:23 GMT /jkcgbj-14/?versions 2022-09-09T22:34:23.252 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:XQ3jlFD2UwzWy6ZDMZzWtLuQSHo= 2022-09-09T22:34:23.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:XQ3jlFD2UwzWy6ZDMZzWtLuQSHo=', 'Content-Length': '0'} 2022-09-09T22:34:23.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000f46364d5816b2d-00631bbf6f-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:34:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:23.263 DEBUG:boto:b'jkcgbj-14999false' 2022-09-09T22:34:23.264 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:34:23.265 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:34:23.266 DEBUG:boto:path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:34:23.267 DEBUG:boto:auth_path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:34:23.267 DEBUG:boto:Method: GET 2022-09-09T22:34:23.268 DEBUG:boto:Path: /jkcgbj-14/?versions&max-keys=999 2022-09-09T22:34:23.269 DEBUG:boto:Data: 2022-09-09T22:34:23.270 DEBUG:boto:Headers: {} 2022-09-09T22:34:23.271 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:34:23.271 DEBUG:boto:Port: 8001 2022-09-09T22:34:23.272 DEBUG:boto:Params: {} 2022-09-09T22:34:23.273 DEBUG:boto:Token: None 2022-09-09T22:34:23.273 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:34:23 GMT /jkcgbj-14/?versions 2022-09-09T22:34:23.274 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:XQ3jlFD2UwzWy6ZDMZzWtLuQSHo= 2022-09-09T22:34:23.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:XQ3jlFD2UwzWy6ZDMZzWtLuQSHo=', 'Content-Length': '0'} 2022-09-09T22:34:23.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d119b58a99634aa-00631bbf6f-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:34:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:23.284 DEBUG:boto:b'jkcgbj-14999false' 2022-09-09T22:34:23.285 DEBUG:boto:path=/jkcgbj-14/ 2022-09-09T22:34:23.285 DEBUG:boto:auth_path=/jkcgbj-14/ 2022-09-09T22:34:23.286 DEBUG:boto:path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:34:23.286 DEBUG:boto:auth_path=/jkcgbj-14/?versions&max-keys=999 2022-09-09T22:34:23.287 DEBUG:boto:Method: GET 2022-09-09T22:34:23.287 DEBUG:boto:Path: /jkcgbj-14/?versions&max-keys=999 2022-09-09T22:34:23.288 DEBUG:boto:Data: 2022-09-09T22:34:23.288 DEBUG:boto:Headers: {} 2022-09-09T22:34:23.289 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:34:23.290 DEBUG:boto:Port: 8000 2022-09-09T22:34:23.291 DEBUG:boto:Params: {} 2022-09-09T22:34:23.291 DEBUG:boto:Token: None 2022-09-09T22:34:23.292 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:34:23 GMT /jkcgbj-14/?versions 2022-09-09T22:34:23.293 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:XQ3jlFD2UwzWy6ZDMZzWtLuQSHo= 2022-09-09T22:34:23.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:XQ3jlFD2UwzWy6ZDMZzWtLuQSHo=', 'Content-Length': '0'} 2022-09-09T22:34:23.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025c6af2e1713d757-00631bbf6f-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:34:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:23.346 DEBUG:boto:b'jkcgbj-14999false' 2022-09-09T22:34:23.348 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-14 zones={test-zone3, test-zone2} 2022-09-09T22:34:23.349 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:34:23.350 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:34:23.351 DEBUG:boto:Method: HEAD 2022-09-09T22:34:23.352 DEBUG:boto:Path: /jkcgbj-15/ 2022-09-09T22:34:23.353 DEBUG:boto:Data: 2022-09-09T22:34:23.353 DEBUG:boto:Headers: {} 2022-09-09T22:34:23.354 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:34:23.355 DEBUG:boto:Port: 8001 2022-09-09T22:34:23.355 DEBUG:boto:Params: {} 2022-09-09T22:34:23.356 DEBUG:boto:Token: None 2022-09-09T22:34:23.357 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:34:23 GMT /jkcgbj-15/ 2022-09-09T22:34:23.358 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JY4kdkTPw8/fRcuVkpcHXusTL74= 2022-09-09T22:34:23.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JY4kdkTPw8/fRcuVkpcHXusTL74=', 'Content-Length': '0'} 2022-09-09T22:34:23.364 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c24b1c1799197d13-00631bbf6f-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:34:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:23.365 DEBUG:boto:path=/jkcgbj-15/myobj 2022-09-09T22:34:23.366 DEBUG:boto:auth_path=/jkcgbj-15/myobj 2022-09-09T22:34:23.367 DEBUG:boto:Method: HEAD 2022-09-09T22:34:23.367 DEBUG:boto:Path: /jkcgbj-15/myobj 2022-09-09T22:34:23.368 DEBUG:boto:Data: 2022-09-09T22:34:23.369 DEBUG:boto:Headers: {} 2022-09-09T22:34:23.369 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:34:23.370 DEBUG:boto:Port: 8001 2022-09-09T22:34:23.371 DEBUG:boto:Params: {} 2022-09-09T22:34:23.371 DEBUG:boto:Token: None 2022-09-09T22:34:23.372 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:34:23 GMT /jkcgbj-15/myobj 2022-09-09T22:34:23.373 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:feyvTgRfWoHwp2N4Q6llON7qe/A= 2022-09-09T22:34:23.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:feyvTgRfWoHwp2N4Q6llON7qe/A=', 'Content-Length': '0'} 2022-09-09T22:34:23.378 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062e0a7da0ad6c0ae-00631bbf6f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:34:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:23.379 DEBUG:boto:path=/jkcgbj-15/myobj 2022-09-09T22:34:23.380 DEBUG:boto:auth_path=/jkcgbj-15/myobj 2022-09-09T22:34:23.380 DEBUG:boto:Method: DELETE 2022-09-09T22:34:23.381 DEBUG:boto:Path: /jkcgbj-15/myobj 2022-09-09T22:34:23.382 DEBUG:boto:Data: 2022-09-09T22:34:23.382 DEBUG:boto:Headers: {} 2022-09-09T22:34:23.383 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:34:23.384 DEBUG:boto:Port: 8001 2022-09-09T22:34:23.384 DEBUG:boto:Params: {} 2022-09-09T22:34:23.385 DEBUG:boto:Token: None 2022-09-09T22:34:23.386 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:34:23 GMT /jkcgbj-15/myobj 2022-09-09T22:34:23.387 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:BbNF/cXX38iCCC+/uHSkqJBHCvE= 2022-09-09T22:34:23.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:BbNF/cXX38iCCC+/uHSkqJBHCvE=', 'Content-Length': '0'} 2022-09-09T22:34:23.403 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000054788b599a28719c-00631bbf6f-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:34:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:23.404 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:34:23.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:34:23.405 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:34:23.459 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:34:23.460 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:34:23.529 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:34:23.518+0000 7fce29731dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:34:23.530 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000001.14.5, , , , , , , , , ] 2022-09-09T22:34:23.531 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000002.17.5, , , , , , , , , ] 2022-09-09T22:34:28.527 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:34:28.518+0000 7fce29731dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:34:28.528 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000001.14.5, , , , , , , , , ] 2022-09-09T22:34:28.528 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000002.17.5, , , , , , , , , ] 2022-09-09T22:34:33.530 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:34:33.522+0000 7fce29731dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:34:33.531 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000001.14.5, , , , , , , , , ] 2022-09-09T22:34:33.532 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000002.17.5, , , , , , , , , ] 2022-09-09T22:34:38.533 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:34:38.526+0000 7fce29731dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:34:38.533 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000001.14.5, , , , , , , , , ] 2022-09-09T22:34:38.534 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000002.17.5, , , , , , , , , ] 2022-09-09T22:34:43.537 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:34:43.526+0000 7fce29731dc0 1 bucket sync caught up with source: 2022-09-09T22:34:43.537 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000002.17.5, , , , , , , , , ] 2022-09-09T22:34:43.539 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000002.17.5, , , , , , , , , ] 2022-09-09T22:34:43.540 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:34:43.526+0000 7fce29731dc0 0 bucket checkpoint complete 2022-09-09T22:34:43.566 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-15 zones={test-zone1, test-zone3} 2022-09-09T22:34:43.567 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:34:43.568 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:34:43.568 DEBUG:boto:Method: HEAD 2022-09-09T22:34:43.569 DEBUG:boto:Path: /jkcgbj-15/ 2022-09-09T22:34:43.570 DEBUG:boto:Data: 2022-09-09T22:34:43.570 DEBUG:boto:Headers: {} 2022-09-09T22:34:43.571 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:34:43.572 DEBUG:boto:Port: 8000 2022-09-09T22:34:43.572 DEBUG:boto:Params: {} 2022-09-09T22:34:43.573 DEBUG:boto:Token: None 2022-09-09T22:34:43.574 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:34:43 GMT /jkcgbj-15/ 2022-09-09T22:34:43.574 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:m3zZQgIT10R+ogKnxXnvHAwUtE4= 2022-09-09T22:34:43.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:m3zZQgIT10R+ogKnxXnvHAwUtE4=', 'Content-Length': '0'} 2022-09-09T22:34:43.583 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000290213bf6ab56137-00631bbf83-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:34:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:43.584 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:34:43.585 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:34:43.586 DEBUG:boto:Method: HEAD 2022-09-09T22:34:43.587 DEBUG:boto:Path: /jkcgbj-15/ 2022-09-09T22:34:43.587 DEBUG:boto:Data: 2022-09-09T22:34:43.588 DEBUG:boto:Headers: {} 2022-09-09T22:34:43.589 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:34:43.589 DEBUG:boto:Port: 8001 2022-09-09T22:34:43.590 DEBUG:boto:Params: {} 2022-09-09T22:34:43.590 DEBUG:boto:Token: None 2022-09-09T22:34:43.591 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:34:43 GMT /jkcgbj-15/ 2022-09-09T22:34:43.592 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:m3zZQgIT10R+ogKnxXnvHAwUtE4= 2022-09-09T22:34:43.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:m3zZQgIT10R+ogKnxXnvHAwUtE4=', 'Content-Length': '0'} 2022-09-09T22:34:43.601 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003fd6083f7a17a5a1-00631bbf83-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:34:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:43.602 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:34:43.603 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:34:43.604 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:34:43.605 DEBUG:boto:path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:34:43.606 DEBUG:boto:auth_path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:34:43.607 DEBUG:boto:Method: GET 2022-09-09T22:34:43.608 DEBUG:boto:Path: /jkcgbj-15/?versions&max-keys=999 2022-09-09T22:34:43.609 DEBUG:boto:Data: 2022-09-09T22:34:43.609 DEBUG:boto:Headers: {} 2022-09-09T22:34:43.610 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:34:43.610 DEBUG:boto:Port: 8000 2022-09-09T22:34:43.611 DEBUG:boto:Params: {} 2022-09-09T22:34:43.611 DEBUG:boto:Token: None 2022-09-09T22:34:43.612 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:34:43 GMT /jkcgbj-15/?versions 2022-09-09T22:34:43.613 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:U3kXZnatFvudN0j9RORrdCVhT5U= 2022-09-09T22:34:43.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:U3kXZnatFvudN0j9RORrdCVhT5U=', 'Content-Length': '0'} 2022-09-09T22:34:43.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f3fc939ac6b47a8-00631bbf83-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:34:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:43.624 DEBUG:boto:b'jkcgbj-15999false' 2022-09-09T22:34:43.626 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:34:43.627 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:34:43.628 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:34:43.629 DEBUG:boto:path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:34:43.629 DEBUG:boto:auth_path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:34:43.630 DEBUG:boto:Method: GET 2022-09-09T22:34:43.631 DEBUG:boto:Path: /jkcgbj-15/?versions&max-keys=999 2022-09-09T22:34:43.632 DEBUG:boto:Data: 2022-09-09T22:34:43.633 DEBUG:boto:Headers: {} 2022-09-09T22:34:43.634 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:34:43.635 DEBUG:boto:Port: 8001 2022-09-09T22:34:43.636 DEBUG:boto:Params: {} 2022-09-09T22:34:43.637 DEBUG:boto:Token: None 2022-09-09T22:34:43.638 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:34:43 GMT /jkcgbj-15/?versions 2022-09-09T22:34:43.639 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:U3kXZnatFvudN0j9RORrdCVhT5U= 2022-09-09T22:34:43.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:U3kXZnatFvudN0j9RORrdCVhT5U=', 'Content-Length': '0'} 2022-09-09T22:34:43.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008090d451c6856379-00631bbf83-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:34:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:43.690 DEBUG:boto:b'jkcgbj-15999false' 2022-09-09T22:34:43.692 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:34:43.693 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:34:43.694 DEBUG:boto:path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:34:43.695 DEBUG:boto:auth_path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:34:43.696 DEBUG:boto:Method: GET 2022-09-09T22:34:43.697 DEBUG:boto:Path: /jkcgbj-15/?versions&max-keys=999 2022-09-09T22:34:43.698 DEBUG:boto:Data: 2022-09-09T22:34:43.699 DEBUG:boto:Headers: {} 2022-09-09T22:34:43.700 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:34:43.701 DEBUG:boto:Port: 8000 2022-09-09T22:34:43.702 DEBUG:boto:Params: {} 2022-09-09T22:34:43.703 DEBUG:boto:Token: None 2022-09-09T22:34:43.705 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:34:43 GMT /jkcgbj-15/?versions 2022-09-09T22:34:43.706 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:U3kXZnatFvudN0j9RORrdCVhT5U= 2022-09-09T22:34:43.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:U3kXZnatFvudN0j9RORrdCVhT5U=', 'Content-Length': '0'} 2022-09-09T22:34:43.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b6276624d0d0ae6-00631bbf83-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:34:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:43.715 DEBUG:boto:b'jkcgbj-15999false' 2022-09-09T22:34:43.716 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:34:43.717 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:34:43.718 DEBUG:boto:path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:34:43.718 DEBUG:boto:auth_path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:34:43.719 DEBUG:boto:Method: GET 2022-09-09T22:34:43.719 DEBUG:boto:Path: /jkcgbj-15/?versions&max-keys=999 2022-09-09T22:34:43.720 DEBUG:boto:Data: 2022-09-09T22:34:43.721 DEBUG:boto:Headers: {} 2022-09-09T22:34:43.721 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:34:43.722 DEBUG:boto:Port: 8001 2022-09-09T22:34:43.723 DEBUG:boto:Params: {} 2022-09-09T22:34:43.723 DEBUG:boto:Token: None 2022-09-09T22:34:43.724 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:34:43 GMT /jkcgbj-15/?versions 2022-09-09T22:34:43.725 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:U3kXZnatFvudN0j9RORrdCVhT5U= 2022-09-09T22:34:43.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:U3kXZnatFvudN0j9RORrdCVhT5U=', 'Content-Length': '0'} 2022-09-09T22:34:43.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4589f6edb365734-00631bbf83-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:34:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:43.735 DEBUG:boto:b'jkcgbj-15999false' 2022-09-09T22:34:43.736 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-15 zones={test-zone1, test-zone3} 2022-09-09T22:34:43.736 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:34:43.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:34:43.738 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:34:43.790 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:34:43.791 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:34:43.863 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:34:43.856+0000 7f219c0c0dc0 1 bucket sync caught up with source: 2022-09-09T22:34:43.864 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000002.17.5, , , , , , , , , ] 2022-09-09T22:34:43.865 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000002.17.5, , , , , , , , , ] 2022-09-09T22:34:43.865 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:34:43.856+0000 7f219c0c0dc0 0 bucket checkpoint complete 2022-09-09T22:34:43.890 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-15 zones={test-zone2, test-zone3} 2022-09-09T22:34:43.890 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:34:43.891 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:34:43.892 DEBUG:boto:Method: HEAD 2022-09-09T22:34:43.892 DEBUG:boto:Path: /jkcgbj-15/ 2022-09-09T22:34:43.893 DEBUG:boto:Data: 2022-09-09T22:34:43.893 DEBUG:boto:Headers: {} 2022-09-09T22:34:43.894 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:34:43.894 DEBUG:boto:Port: 8000 2022-09-09T22:34:43.895 DEBUG:boto:Params: {} 2022-09-09T22:34:43.895 DEBUG:boto:Token: None 2022-09-09T22:34:43.896 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:34:43 GMT /jkcgbj-15/ 2022-09-09T22:34:43.897 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:m3zZQgIT10R+ogKnxXnvHAwUtE4= 2022-09-09T22:34:43.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:m3zZQgIT10R+ogKnxXnvHAwUtE4=', 'Content-Length': '0'} 2022-09-09T22:34:43.901 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0b8634bd01f085e-00631bbf83-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:34:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:43.902 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:34:43.903 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:34:43.903 DEBUG:boto:Method: HEAD 2022-09-09T22:34:43.904 DEBUG:boto:Path: /jkcgbj-15/ 2022-09-09T22:34:43.904 DEBUG:boto:Data: 2022-09-09T22:34:43.905 DEBUG:boto:Headers: {} 2022-09-09T22:34:43.905 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:34:43.906 DEBUG:boto:Port: 8001 2022-09-09T22:34:43.906 DEBUG:boto:Params: {} 2022-09-09T22:34:43.907 DEBUG:boto:Token: None 2022-09-09T22:34:43.908 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:34:43 GMT /jkcgbj-15/ 2022-09-09T22:34:43.908 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:m3zZQgIT10R+ogKnxXnvHAwUtE4= 2022-09-09T22:34:43.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:m3zZQgIT10R+ogKnxXnvHAwUtE4=', 'Content-Length': '0'} 2022-09-09T22:34:43.914 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007dcb5e3a6e02f5af-00631bbf83-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:34:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:43.915 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:34:43.915 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:34:43.916 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:34:43.916 DEBUG:boto:path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:34:43.917 DEBUG:boto:auth_path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:34:43.917 DEBUG:boto:Method: GET 2022-09-09T22:34:43.918 DEBUG:boto:Path: /jkcgbj-15/?versions&max-keys=999 2022-09-09T22:34:43.918 DEBUG:boto:Data: 2022-09-09T22:34:43.919 DEBUG:boto:Headers: {} 2022-09-09T22:34:43.920 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:34:43.920 DEBUG:boto:Port: 8000 2022-09-09T22:34:43.921 DEBUG:boto:Params: {} 2022-09-09T22:34:43.921 DEBUG:boto:Token: None 2022-09-09T22:34:43.922 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:34:43 GMT /jkcgbj-15/?versions 2022-09-09T22:34:43.922 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:U3kXZnatFvudN0j9RORrdCVhT5U= 2022-09-09T22:34:43.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:U3kXZnatFvudN0j9RORrdCVhT5U=', 'Content-Length': '0'} 2022-09-09T22:34:43.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000436ab299163d36f9-00631bbf83-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:34:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:43.929 DEBUG:boto:b'jkcgbj-15999false' 2022-09-09T22:34:43.930 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:34:43.931 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:34:43.932 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:34:43.932 DEBUG:boto:path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:34:43.934 DEBUG:boto:auth_path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:34:43.934 DEBUG:boto:Method: GET 2022-09-09T22:34:43.935 DEBUG:boto:Path: /jkcgbj-15/?versions&max-keys=999 2022-09-09T22:34:43.935 DEBUG:boto:Data: 2022-09-09T22:34:43.936 DEBUG:boto:Headers: {} 2022-09-09T22:34:43.937 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:34:43.938 DEBUG:boto:Port: 8001 2022-09-09T22:34:43.939 DEBUG:boto:Params: {} 2022-09-09T22:34:43.940 DEBUG:boto:Token: None 2022-09-09T22:34:43.940 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:34:43 GMT /jkcgbj-15/?versions 2022-09-09T22:34:43.941 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:U3kXZnatFvudN0j9RORrdCVhT5U= 2022-09-09T22:34:43.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:U3kXZnatFvudN0j9RORrdCVhT5U=', 'Content-Length': '0'} 2022-09-09T22:34:43.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012c3ac178da01a30-00631bbf83-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:34:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:43.948 DEBUG:boto:b'jkcgbj-15999false' 2022-09-09T22:34:43.949 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:34:43.950 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:34:43.950 DEBUG:boto:path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:34:43.951 DEBUG:boto:auth_path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:34:43.951 DEBUG:boto:Method: GET 2022-09-09T22:34:43.952 DEBUG:boto:Path: /jkcgbj-15/?versions&max-keys=999 2022-09-09T22:34:43.952 DEBUG:boto:Data: 2022-09-09T22:34:43.953 DEBUG:boto:Headers: {} 2022-09-09T22:34:43.954 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:34:43.955 DEBUG:boto:Port: 8000 2022-09-09T22:34:43.956 DEBUG:boto:Params: {} 2022-09-09T22:34:43.957 DEBUG:boto:Token: None 2022-09-09T22:34:43.958 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:34:43 GMT /jkcgbj-15/?versions 2022-09-09T22:34:43.959 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:U3kXZnatFvudN0j9RORrdCVhT5U= 2022-09-09T22:34:43.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:U3kXZnatFvudN0j9RORrdCVhT5U=', 'Content-Length': '0'} 2022-09-09T22:34:43.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d687bc8b78d6bfb-00631bbf83-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:34:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:43.968 DEBUG:boto:b'jkcgbj-15999false' 2022-09-09T22:34:43.969 DEBUG:boto:path=/jkcgbj-15/ 2022-09-09T22:34:43.970 DEBUG:boto:auth_path=/jkcgbj-15/ 2022-09-09T22:34:43.970 DEBUG:boto:path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:34:43.971 DEBUG:boto:auth_path=/jkcgbj-15/?versions&max-keys=999 2022-09-09T22:34:43.972 DEBUG:boto:Method: GET 2022-09-09T22:34:43.973 DEBUG:boto:Path: /jkcgbj-15/?versions&max-keys=999 2022-09-09T22:34:43.974 DEBUG:boto:Data: 2022-09-09T22:34:43.975 DEBUG:boto:Headers: {} 2022-09-09T22:34:43.976 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:34:43.977 DEBUG:boto:Port: 8001 2022-09-09T22:34:43.978 DEBUG:boto:Params: {} 2022-09-09T22:34:43.979 DEBUG:boto:Token: None 2022-09-09T22:34:43.980 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:34:43 GMT /jkcgbj-15/?versions 2022-09-09T22:34:43.981 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:U3kXZnatFvudN0j9RORrdCVhT5U= 2022-09-09T22:34:43.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:U3kXZnatFvudN0j9RORrdCVhT5U=', 'Content-Length': '0'} 2022-09-09T22:34:43.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087537907d5122904-00631bbf83-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:34:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:44.034 DEBUG:boto:b'jkcgbj-15999false' 2022-09-09T22:34:44.035 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-15 zones={test-zone2, test-zone3} 2022-09-09T22:34:44.036 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2022-09-09T22:34:44.037 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jkcgbj-16 2022-09-09T22:34:44.038 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:34:44.038 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:34:44.039 DEBUG:boto:Method: PUT 2022-09-09T22:34:44.039 DEBUG:boto:Path: /jkcgbj-16/ 2022-09-09T22:34:44.040 DEBUG:boto:Data: 2022-09-09T22:34:44.040 DEBUG:boto:Headers: {} 2022-09-09T22:34:44.041 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:34:44.041 DEBUG:boto:Port: 8000 2022-09-09T22:34:44.042 DEBUG:boto:Params: {} 2022-09-09T22:34:44.042 DEBUG:boto:Token: None 2022-09-09T22:34:44.043 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:34:44 GMT /jkcgbj-16/ 2022-09-09T22:34:44.043 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:SSuUXOeogM2RTCq7d2ytVvFaQos= 2022-09-09T22:34:44.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:SSuUXOeogM2RTCq7d2ytVvFaQos=', 'Content-Length': '0'} 2022-09-09T22:34:44.071 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002398d46cfee1d651-00631bbf84-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:34:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:44.072 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jkcgbj-17 2022-09-09T22:34:44.074 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:34:44.075 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:34:44.075 DEBUG:boto:Method: PUT 2022-09-09T22:34:44.076 DEBUG:boto:Path: /jkcgbj-17/ 2022-09-09T22:34:44.076 DEBUG:boto:Data: 2022-09-09T22:34:44.077 DEBUG:boto:Headers: {} 2022-09-09T22:34:44.077 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:34:44.078 DEBUG:boto:Port: 8000 2022-09-09T22:34:44.078 DEBUG:boto:Params: {} 2022-09-09T22:34:44.079 DEBUG:boto:Token: None 2022-09-09T22:34:44.080 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:34:44 GMT /jkcgbj-17/ 2022-09-09T22:34:44.080 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YneZucHHrf1JdWCLzpZOz63jQkA= 2022-09-09T22:34:44.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YneZucHHrf1JdWCLzpZOz63jQkA=', 'Content-Length': '0'} 2022-09-09T22:34:44.127 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000da6460517e50fc8f-00631bbf84-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:34:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:44.128 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jkcgbj-18 2022-09-09T22:34:44.129 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:34:44.129 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:34:44.130 DEBUG:boto:Method: PUT 2022-09-09T22:34:44.130 DEBUG:boto:Path: /jkcgbj-18/ 2022-09-09T22:34:44.131 DEBUG:boto:Data: 2022-09-09T22:34:44.131 DEBUG:boto:Headers: {} 2022-09-09T22:34:44.132 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:34:44.132 DEBUG:boto:Port: 8001 2022-09-09T22:34:44.133 DEBUG:boto:Params: {} 2022-09-09T22:34:44.133 DEBUG:boto:Token: None 2022-09-09T22:34:44.134 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:34:44 GMT /jkcgbj-18/ 2022-09-09T22:34:44.134 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:kr+kxqVZ6d46nSK0NMd4Mr9M7zc= 2022-09-09T22:34:44.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:kr+kxqVZ6d46nSK0NMd4Mr9M7zc=', 'Content-Length': '0'} 2022-09-09T22:34:44.183 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000957744a33c9be69e-00631bbf84-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:34:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:44.184 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:34:44.184 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:34:44.185 DEBUG:boto:path=/jkcgbj-16/?versioning 2022-09-09T22:34:44.185 DEBUG:boto:auth_path=/jkcgbj-16/?versioning 2022-09-09T22:34:44.186 DEBUG:boto:Method: PUT 2022-09-09T22:34:44.186 DEBUG:boto:Path: /jkcgbj-16/?versioning 2022-09-09T22:34:44.187 DEBUG:boto:Data: Enabled Disabled 2022-09-09T22:34:44.187 DEBUG:boto:Headers: {} 2022-09-09T22:34:44.188 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:34:44.188 DEBUG:boto:Port: 8000 2022-09-09T22:34:44.189 DEBUG:boto:Params: {} 2022-09-09T22:34:44.189 DEBUG:boto:Token: None 2022-09-09T22:34:44.190 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:34:44 GMT /jkcgbj-16/?versioning 2022-09-09T22:34:44.191 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:3o3lATxxkWE+82efV7Q1y1/thcY= 2022-09-09T22:34:44.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:3o3lATxxkWE+82efV7Q1y1/thcY=', 'Content-Length': '228'} 2022-09-09T22:34:44.201 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e785fa2bfbc29456-00631bbf84-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:34:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:44.201 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:34:44.202 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:34:44.203 DEBUG:boto:path=/jkcgbj-17/?versioning 2022-09-09T22:34:44.203 DEBUG:boto:auth_path=/jkcgbj-17/?versioning 2022-09-09T22:34:44.204 DEBUG:boto:Method: PUT 2022-09-09T22:34:44.204 DEBUG:boto:Path: /jkcgbj-17/?versioning 2022-09-09T22:34:44.205 DEBUG:boto:Data: Enabled Disabled 2022-09-09T22:34:44.205 DEBUG:boto:Headers: {} 2022-09-09T22:34:44.206 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:34:44.207 DEBUG:boto:Port: 8000 2022-09-09T22:34:44.207 DEBUG:boto:Params: {} 2022-09-09T22:34:44.208 DEBUG:boto:Token: None 2022-09-09T22:34:44.209 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:34:44 GMT /jkcgbj-17/?versioning 2022-09-09T22:34:44.209 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:L/m6aHURCZwwkldzuYLcORG4w9I= 2022-09-09T22:34:44.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:L/m6aHURCZwwkldzuYLcORG4w9I=', 'Content-Length': '228'} 2022-09-09T22:34:44.227 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ef7e99bb83472a7e-00631bbf84-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:34:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:44.228 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:34:44.228 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:34:44.229 DEBUG:boto:path=/jkcgbj-18/?versioning 2022-09-09T22:34:44.229 DEBUG:boto:auth_path=/jkcgbj-18/?versioning 2022-09-09T22:34:44.229 DEBUG:boto:Method: PUT 2022-09-09T22:34:44.230 DEBUG:boto:Path: /jkcgbj-18/?versioning 2022-09-09T22:34:44.230 DEBUG:boto:Data: Enabled Disabled 2022-09-09T22:34:44.231 DEBUG:boto:Headers: {} 2022-09-09T22:34:44.231 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:34:44.232 DEBUG:boto:Port: 8001 2022-09-09T22:34:44.232 DEBUG:boto:Params: {} 2022-09-09T22:34:44.233 DEBUG:boto:Token: None 2022-09-09T22:34:44.234 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:34:44 GMT /jkcgbj-18/?versioning 2022-09-09T22:34:44.234 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:3hLIAeuZcPh3at46FJCiLEsDI6k= 2022-09-09T22:34:44.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:3hLIAeuZcPh3at46FJCiLEsDI6k=', 'Content-Length': '228'} 2022-09-09T22:34:44.254 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005715e870624821f1-00631bbf84-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:34:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:44.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:34:44.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:34:44.256 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:34:44.301 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:34:44.301 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:34:44.365 INFO:teuthology.orchestra.run.smithi005.stderr:No --period given, using current period=503b1d61-4513-48c8-a308-17b76219610c 2022-09-09T22:34:44.381 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T22:34:44.382 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:34:44.382 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:34:44.382 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:28:06.146147Z" 2022-09-09T22:34:44.383 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:34:44.383 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:34:44.383 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:34:44.383 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:34:44.384 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:34:44.384 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:34:44.384 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:34:44.384 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:30:37.757014Z" 2022-09-09T22:34:44.385 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:34:44.385 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:34:44.385 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762884.244083_78.1", 2022-09-09T22:34:44.385 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:34:44.244083Z" 2022-09-09T22:34:44.386 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:34:44.386 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T22:34:44.387 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662762884.244083_78.1'} 2022-09-09T22:34:44.388 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-09T22:34:44.389 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:34:44.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:34:44.390 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:34:44.431 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:34:44.431 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:34:44.482 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:34:44.476+0000 7f6d3ad6adc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:34:44.484 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:34:44.476+0000 7f6d3ad6adc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:34:44.508 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:34:44.509 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:34:44.509 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:34:44.509 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:34:44.509 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:34:44.510 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:34:44.510 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:34:44.510 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:34:44.511 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:34:44.511 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:34:44.511 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:34:44.512 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:34:44.512 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:34:44.512 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:34:44.513 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:34:44.513 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:34:44.513 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:34:44.514 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:34:44.514 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:34:44.514 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:34:44.515 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:34:44.515 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:34:44.515 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:34:44.516 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:34:44.516 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:34:44.516 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:34:44.517 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:34:44.517 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:34:44.517 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:34:44.517 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:34:44.518 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:34:44.518 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:34:44.518 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:34:44.518 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:34:44.519 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:34:44.519 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:34:44.520 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:34:44.520 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:34:44.520 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:34:44.521 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:34:44.521 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:34:44.521 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:34:44.521 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:34:44.522 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:34:44.522 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:34:44.522 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:34:44.523 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:34:44.523 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:34:44.523 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:34:44.524 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762884.149665_69.1", 2022-09-09T22:34:44.524 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:34:44.524 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:34:44.525 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:34:44.525 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.149665Z", 2022-09-09T22:34:44.525 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:34:44.525 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:34:44.526 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:34:44.526 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:34:44.526 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:34:44.527 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:34:44.527 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:34:44.527 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:34:44.528 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:34:44.528 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:34:44Z" 2022-09-09T22:34:44.528 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:34:44.529 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662762884.149665_69.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:34:44.149665Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:34:44Z" } 2022-09-09T22:34:44.531 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662762884.149665_69.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:34:44.149665Z', 'realm_epoch': 2}}] 2022-09-09T22:34:44.531 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662762884.244083_78.1'} 2022-09-09T22:34:44.532 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662762884.149665_69.1'} 2022-09-09T22:34:44.532 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662762884.244083_78.1 target=1_1662762884.149665_69.1 2022-09-09T22:34:49.534 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:34:49.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:34:49.536 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:34:49.586 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:34:49.586 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:34:49.641 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:34:49.636+0000 7f55411c6dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:34:49.643 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:34:49.636+0000 7f55411c6dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:34:49.677 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:34:49.678 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:34:49.678 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:34:49.679 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:34:49.679 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:34:49.679 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:34:49.680 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:34:49.680 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:34:49.681 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:34:49.681 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:34:49.681 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:34:49.682 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:34:49.682 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:34:49.683 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:34:49.683 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:34:49.683 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:34:49.684 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:34:49.684 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:34:49.684 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:34:49.685 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:34:49.685 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:34:49.685 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:34:49.685 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:34:49.686 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:34:49.686 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:34:49.686 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:34:49.687 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:34:49.687 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:34:49.687 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:34:49.688 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:34:49.688 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:34:49.688 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:34:49.689 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:34:49.689 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:34:49.689 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:34:49.690 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:34:49.690 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:34:49.690 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:34:49.691 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:34:49.691 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:34:49.691 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:34:49.691 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:34:49.692 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:34:49.692 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:34:49.692 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:34:49.692 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:34:49.693 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:34:49.693 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:34:49.693 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:34:49.694 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762884.244083_78.1", 2022-09-09T22:34:49.694 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:34:49.694 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:34:49.695 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:34:49.695 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.244083Z", 2022-09-09T22:34:49.695 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:34:49.695 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:34:49.696 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:34:49.696 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:34:49.696 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:34:49.696 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:34:49.697 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:34:49.697 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:34:49.697 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:34:49.697 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:34:49Z" 2022-09-09T22:34:49.698 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:34:49.699 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662762884.244083_78.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:34:44.244083Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:34:49Z" } 2022-09-09T22:34:49.700 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662762884.244083_78.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:34:44.244083Z', 'realm_epoch': 2}}] 2022-09-09T22:34:49.700 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662762884.244083_78.1'} 2022-09-09T22:34:49.701 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662762884.244083_78.1'} 2022-09-09T22:34:49.701 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-09T22:34:49.702 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-09T22:34:49.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:34:49.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:34:49.703 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:34:49.752 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:34:49.753 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:34:49.804 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:34:49.794+0000 7f3113c16dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:34:49.806 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:34:49.798+0000 7f3113c16dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:34:49.831 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:34:49.831 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T22:34:49.831 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T22:34:49.832 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T22:34:49.832 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T22:34:49.832 INFO:teuthology.orchestra.run.smithi005.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:34:49.833 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:34:49.833 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:34:49.833 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T22:34:49.834 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:34:49.834 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T22:34:49.834 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:34:49.835 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:34:49.835 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:34:49.835 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:34:49.836 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 1, 2022-09-09T22:34:49.836 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:34:49.837 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:34:49.837 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:34:49.838 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:34:49.838 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:34:49.839 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:34:49.839 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T22:34:49.840 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:34:49.840 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:34:49.841 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:34:49.841 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:34:49.841 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:34:49.842 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:34:49.842 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:34:49.843 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:34:49.843 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:34:49.844 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:34:49.844 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:34:49.845 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T22:34:49.846 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:34:49.846 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:34:49.846 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:34:49.846 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:34:49.847 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:34:49.847 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:34:49.847 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:34:49.847 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:34:49.848 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:34:49.848 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:34:49.848 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:34:49.848 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T22:34:49.849 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:34:49.849 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:34:49.849 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762884.244083_78.1", 2022-09-09T22:34:49.849 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:34:49.850 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:34:49.850 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:34:49.850 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.244083Z", 2022-09-09T22:34:49.850 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:34:49.851 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:34:49.851 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:34:49.851 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:34:49.851 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:34:49.852 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T22:34:49.852 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 1, 2022-09-09T22:34:49.852 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 1 2022-09-09T22:34:49.852 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:34:49.853 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T22:34:49Z" 2022-09-09T22:34:49.853 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:34:49.854 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662762884.244083_78.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:34:44.244083Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:34:49Z" } 2022-09-09T22:34:49.855 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662762884.244083_78.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:34:44.244083Z', 'realm_epoch': 2}}] 2022-09-09T22:34:49.855 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662762884.244083_78.1'} 2022-09-09T22:34:49.856 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662762884.244083_78.1'} 2022-09-09T22:34:49.856 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-09T22:34:49.857 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-09T22:34:49.857 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:34:49.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:34:49.859 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:34:49.907 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:34:49.907 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:34:49.981 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:34:49.972+0000 7f141e457dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:34:49.983 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:34:49.976+0000 7f141e457dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:34:50.013 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:34:50.014 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:34:50.014 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:34:50.014 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:34:50.014 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:34:50.015 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:34:50.015 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:34:50.015 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:34:50.016 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:34:50.016 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:34:50.016 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:34:50.016 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:34:50.017 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:34:50.017 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:34:50.017 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:34:50.018 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:34:50.018 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:34:50.018 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:34:50.018 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:34:50.019 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:34:50.019 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:34:50.019 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:34:50.019 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:34:50.020 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:34:50.020 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:34:50.020 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:34:50.021 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:34:50.021 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:34:50.021 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:34:50.021 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:34:50.022 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:34:50.022 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:34:50.022 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:34:50.023 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:34:50.023 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:34:50.023 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:34:50.023 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:34:50.024 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:34:50.024 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:34:50.024 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:34:50.024 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:34:50.025 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:34:50.025 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:34:50.025 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:34:50.026 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:34:50.026 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:34:50.026 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:34:50.026 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:34:50.027 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:34:50.027 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762884.244083_78.1", 2022-09-09T22:34:50.027 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:34:50.027 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:34:50.028 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:34:50.028 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.244083Z", 2022-09-09T22:34:50.028 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:34:50.029 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:34:50.029 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:34:50.029 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:34:50.029 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:34:50.030 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:34:50.030 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:34:50.030 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:34:50.030 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:34:50.031 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:34:49Z" 2022-09-09T22:34:50.031 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:34:50.032 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662762884.244083_78.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:34:44.244083Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:34:49Z" } 2022-09-09T22:34:50.033 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662762884.244083_78.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:34:44.244083Z', 'realm_epoch': 2}}] 2022-09-09T22:34:50.033 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662762884.244083_78.1'} 2022-09-09T22:34:50.034 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662762884.244083_78.1'} 2022-09-09T22:34:50.034 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-09T22:34:50.035 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:34:50.035 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:34:50.036 DEBUG:boto:Method: HEAD 2022-09-09T22:34:50.036 DEBUG:boto:Path: /jkcgbj-16/ 2022-09-09T22:34:50.037 DEBUG:boto:Data: 2022-09-09T22:34:50.037 DEBUG:boto:Headers: {} 2022-09-09T22:34:50.038 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:34:50.038 DEBUG:boto:Port: 8000 2022-09-09T22:34:50.039 DEBUG:boto:Params: {} 2022-09-09T22:34:50.039 DEBUG:boto:Token: None 2022-09-09T22:34:50.040 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:34:50 GMT /jkcgbj-16/ 2022-09-09T22:34:50.041 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pIqv4emHXr5RSTdpX9vartf/1vc= 2022-09-09T22:34:50.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:34:50 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pIqv4emHXr5RSTdpX9vartf/1vc=', 'Content-Length': '0'} 2022-09-09T22:34:50.046 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e46f92f4a07754a0-00631bbf8a-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:50.047 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:34:50.048 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:34:50.048 DEBUG:boto:Method: PUT 2022-09-09T22:34:50.049 DEBUG:boto:Path: /jkcgbj-16/dummy 2022-09-09T22:34:50.049 DEBUG:boto:Data: 2022-09-09T22:34:50.050 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-09T22:34:50.050 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:34:50.051 DEBUG:boto:Port: 8000 2022-09-09T22:34:50.051 DEBUG:boto:Params: {} 2022-09-09T22:34:50.052 DEBUG:boto:Token: None 2022-09-09T22:34:50.052 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 09 Sep 2022 22:34:50 GMT /jkcgbj-16/dummy 2022-09-09T22:34:50.053 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8YJoeJS0AWqoufN1Csd98XMZ1v8= 2022-09-09T22:34:50.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:34:50 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8YJoeJS0AWqoufN1Csd98XMZ1v8='} 2022-09-09T22:34:50.098 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-amz-request-id', 'tx00000d6d20f58969625eb-00631bbf8a-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:34:50.099 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:34:50.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:34:50.101 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:34:50.154 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:34:50.154 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:34:50.218 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:34:50.212+0000 7f3a27c23dc0 1 waiting to reach incremental sync.. 2022-09-09T22:34:55.219 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:34:55.212+0000 7f3a27c23dc0 1 waiting to reach incremental sync.. 2022-09-09T22:35:00.224 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:35:00.216+0000 7f3a27c23dc0 1 bucket sync caught up with source: 2022-09-09T22:35:00.224 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000002.17.10, , , , , ] 2022-09-09T22:35:00.224 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000002.17.10, , , , , ] 2022-09-09T22:35:00.225 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:35:00.216+0000 7f3a27c23dc0 0 bucket checkpoint complete 2022-09-09T22:35:00.255 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:35:00.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:35:00.258 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:35:00.311 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:35:00.312 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:35:00.379 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:35:00.370+0000 7f1360887dc0 1 bucket sync caught up with source: 2022-09-09T22:35:00.380 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , 00000000002.17.10, , , , , ] 2022-09-09T22:35:00.380 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , 00000000002.17.10, , , , , ] 2022-09-09T22:35:00.381 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:35:00.370+0000 7f1360887dc0 0 bucket checkpoint complete 2022-09-09T22:35:00.404 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:35:00.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:35:00.405 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:35:00.459 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:35:00.459 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:35:00.532 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:35:00.524+0000 7fbdbcd81dc0 1 waiting to reach incremental sync.. 2022-09-09T22:35:05.536 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:35:05.527+0000 7fbdbcd81dc0 1 bucket sync caught up with source: 2022-09-09T22:35:05.536 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000002.17.10, , , , , ] 2022-09-09T22:35:05.537 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000002.17.10, , , , , ] 2022-09-09T22:35:05.537 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:35:05.527+0000 7fbdbcd81dc0 0 bucket checkpoint complete 2022-09-09T22:35:05.562 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:35:05.563 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:35:05.564 DEBUG:boto:Method: HEAD 2022-09-09T22:35:05.564 DEBUG:boto:Path: /jkcgbj-17/ 2022-09-09T22:35:05.565 DEBUG:boto:Data: 2022-09-09T22:35:05.565 DEBUG:boto:Headers: {} 2022-09-09T22:35:05.566 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:35:05.566 DEBUG:boto:Port: 8000 2022-09-09T22:35:05.567 DEBUG:boto:Params: {} 2022-09-09T22:35:05.568 DEBUG:boto:Token: None 2022-09-09T22:35:05.568 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:35:05 GMT /jkcgbj-17/ 2022-09-09T22:35:05.569 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:PNbZvR7mv85IIpFdqU+8NIF5Lp8= 2022-09-09T22:35:05.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:35:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:PNbZvR7mv85IIpFdqU+8NIF5Lp8=', 'Content-Length': '0'} 2022-09-09T22:35:05.573 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007e8050cc3a71ca9d-00631bbf99-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:35:05.574 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:35:05.575 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:35:05.575 DEBUG:boto:Method: PUT 2022-09-09T22:35:05.576 DEBUG:boto:Path: /jkcgbj-17/dummy 2022-09-09T22:35:05.577 DEBUG:boto:Data: 2022-09-09T22:35:05.577 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-09T22:35:05.578 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:35:05.578 DEBUG:boto:Port: 8000 2022-09-09T22:35:05.579 DEBUG:boto:Params: {} 2022-09-09T22:35:05.579 DEBUG:boto:Token: None 2022-09-09T22:35:05.580 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 09 Sep 2022 22:35:05 GMT /jkcgbj-17/dummy 2022-09-09T22:35:05.580 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:98mXXTLXp35pY6H8Czmcax5dQHs= 2022-09-09T22:35:05.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:35:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:98mXXTLXp35pY6H8Czmcax5dQHs='} 2022-09-09T22:35:05.607 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-amz-request-id', 'tx000003de6d36002e3fc8a-00631bbf99-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:35:05.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:35:05.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:35:05.609 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:35:05.666 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:35:05.666 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:35:05.724 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:35:05.717+0000 7f6e3ec47dc0 1 waiting to reach incremental sync.. 2022-09-09T22:35:10.725 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:35:10.717+0000 7f6e3ec47dc0 1 waiting to reach incremental sync.. 2022-09-09T22:35:15.726 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:35:15.721+0000 7f6e3ec47dc0 1 waiting to reach incremental sync.. 2022-09-09T22:35:20.731 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:35:20.725+0000 7f6e3ec47dc0 1 bucket sync caught up with source: 2022-09-09T22:35:20.731 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , 00000000002.20.10, , , , , ] 2022-09-09T22:35:20.731 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , 00000000002.20.10, , , , , ] 2022-09-09T22:35:20.732 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:35:20.725+0000 7f6e3ec47dc0 0 bucket checkpoint complete 2022-09-09T22:35:20.759 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:35:20.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:35:20.762 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:35:20.815 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:35:20.816 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:35:20.879 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:35:20.873+0000 7f4fe8004dc0 1 bucket sync caught up with source: 2022-09-09T22:35:20.880 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , 00000000002.20.10, , , , , ] 2022-09-09T22:35:20.880 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , 00000000002.20.10, , , , , ] 2022-09-09T22:35:20.880 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:35:20.873+0000 7f4fe8004dc0 0 bucket checkpoint complete 2022-09-09T22:35:20.911 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:35:20.912 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:35:20.913 DEBUG:boto:Method: HEAD 2022-09-09T22:35:20.913 DEBUG:boto:Path: /jkcgbj-18/ 2022-09-09T22:35:20.914 DEBUG:boto:Data: 2022-09-09T22:35:20.914 DEBUG:boto:Headers: {} 2022-09-09T22:35:20.915 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:35:20.915 DEBUG:boto:Port: 8001 2022-09-09T22:35:20.916 DEBUG:boto:Params: {} 2022-09-09T22:35:20.917 DEBUG:boto:Token: None 2022-09-09T22:35:20.917 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:35:20 GMT /jkcgbj-18/ 2022-09-09T22:35:20.918 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8CaULxjesYSOFaZmCdVxR9t3SAY= 2022-09-09T22:35:20.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:35:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8CaULxjesYSOFaZmCdVxR9t3SAY=', 'Content-Length': '0'} 2022-09-09T22:35:20.923 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000013c7fd708e8029f5-00631bbfa8-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:35:20.925 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:35:20.926 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:35:20.927 DEBUG:boto:Method: PUT 2022-09-09T22:35:20.927 DEBUG:boto:Path: /jkcgbj-18/dummy 2022-09-09T22:35:20.928 DEBUG:boto:Data: 2022-09-09T22:35:20.928 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-09T22:35:20.929 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:35:20.929 DEBUG:boto:Port: 8001 2022-09-09T22:35:20.930 DEBUG:boto:Params: {} 2022-09-09T22:35:20.930 DEBUG:boto:Token: None 2022-09-09T22:35:20.932 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 09 Sep 2022 22:35:20 GMT /jkcgbj-18/dummy 2022-09-09T22:35:20.932 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:oLudk97qOopuxhkhscCVQN3vNMI= 2022-09-09T22:35:20.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:35:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:oLudk97qOopuxhkhscCVQN3vNMI='} 2022-09-09T22:35:20.960 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-amz-request-id', 'tx00000e445245fdd9d1e62-00631bbfa8-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:35:20.961 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:35:20.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:35:20.963 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:35:21.017 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:35:21.018 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:35:21.082 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:35:21.077+0000 7f6270e4cdc0 1 waiting to reach incremental sync.. 2022-09-09T22:35:26.083 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:35:26.077+0000 7f6270e4cdc0 1 waiting to reach incremental sync.. 2022-09-09T22:35:31.084 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:35:31.077+0000 7f6270e4cdc0 1 waiting to reach incremental sync.. 2022-09-09T22:35:36.085 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:35:36.081+0000 7f6270e4cdc0 1 waiting to reach incremental sync.. 2022-09-09T22:35:41.089 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:35:41.084+0000 7f6270e4cdc0 1 bucket sync caught up with source: 2022-09-09T22:35:41.090 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , 00000000002.10.10, , , , , ] 2022-09-09T22:35:41.091 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , 00000000002.10.10, , , , , ] 2022-09-09T22:35:41.091 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:35:41.084+0000 7f6270e4cdc0 0 bucket checkpoint complete 2022-09-09T22:35:41.117 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:35:41.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:35:41.120 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:35:41.175 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:35:41.176 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:35:41.262 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:35:41.259+0000 7f0879989dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:35:41.263 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000002.10.10, , , , , ] 2022-09-09T22:35:41.263 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.14.9, , , , , ] 2022-09-09T22:35:46.265 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:35:46.263+0000 7f0879989dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:35:46.266 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000002.10.10, , , , , ] 2022-09-09T22:35:46.266 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.14.9, , , , , ] 2022-09-09T22:35:51.268 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:35:51.263+0000 7f0879989dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:35:51.269 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000002.10.10, , , , , ] 2022-09-09T22:35:51.269 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.14.9, , , , , ] 2022-09-09T22:35:56.271 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:35:56.267+0000 7f0879989dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:35:56.271 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000002.10.10, , , , , ] 2022-09-09T22:35:56.272 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.14.9, , , , , ] 2022-09-09T22:36:01.274 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:36:01.270+0000 7f0879989dc0 1 bucket sync caught up with source: 2022-09-09T22:36:01.275 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.14.9, , , , , ] 2022-09-09T22:36:01.275 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.14.9, , , , , ] 2022-09-09T22:36:01.276 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:36:01.270+0000 7f0879989dc0 0 bucket checkpoint complete 2022-09-09T22:36:01.303 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:36:01.303 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:36:01.304 DEBUG:boto:Method: HEAD 2022-09-09T22:36:01.305 DEBUG:boto:Path: /jkcgbj-16/ 2022-09-09T22:36:01.305 DEBUG:boto:Data: 2022-09-09T22:36:01.306 DEBUG:boto:Headers: {} 2022-09-09T22:36:01.306 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:36:01.307 DEBUG:boto:Port: 8000 2022-09-09T22:36:01.307 DEBUG:boto:Params: {} 2022-09-09T22:36:01.308 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T22:36:01.309 DEBUG:boto:Token: None 2022-09-09T22:36:01.310 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:36:01 GMT /jkcgbj-16/ 2022-09-09T22:36:01.310 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qoEROyGsdc4eL3L7h/9XtQMywWw= 2022-09-09T22:36:01.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:36:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qoEROyGsdc4eL3L7h/9XtQMywWw=', 'Content-Length': '0'} 2022-09-09T22:36:01.321 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000057a568adfa62accb-00631bbfd1-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:01.322 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:36:01.322 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:36:01.323 DEBUG:boto:Method: PUT 2022-09-09T22:36:01.323 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1 2022-09-09T22:36:01.324 DEBUG:boto:Data: 2022-09-09T22:36:01.324 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:36:01.325 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:36:01.325 DEBUG:boto:Port: 8000 2022-09-09T22:36:01.326 DEBUG:boto:Params: {} 2022-09-09T22:36:01.326 DEBUG:boto:Token: None 2022-09-09T22:36:01.327 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:36:01 GMT /jkcgbj-16/obj-test-zone1 2022-09-09T22:36:01.327 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:l0ikJ/pKRv01lhHH1EF3ATKgkE4= 2022-09-09T22:36:01.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:36:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:l0ikJ/pKRv01lhHH1EF3ATKgkE4='} 2022-09-09T22:36:01.354 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-amz-request-id', 'tx00000acf67e48a1e2590f-00631bbfd1-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:01.355 DEBUG:rgw_multi.tests:version1 id=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:36:01.356 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:36:01.356 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:36:01.357 DEBUG:boto:Method: PUT 2022-09-09T22:36:01.357 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1 2022-09-09T22:36:01.358 DEBUG:boto:Data: 2022-09-09T22:36:01.359 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:36:01.359 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:36:01.360 DEBUG:boto:Port: 8000 2022-09-09T22:36:01.360 DEBUG:boto:Params: {} 2022-09-09T22:36:01.361 DEBUG:boto:Token: None 2022-09-09T22:36:01.361 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 09 Sep 2022 22:36:01 GMT /jkcgbj-16/obj-test-zone1 2022-09-09T22:36:01.362 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6WSr/R+4B5/n+uK5B0q1rMmg4lU= 2022-09-09T22:36:01.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:36:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6WSr/R+4B5/n+uK5B0q1rMmg4lU='} 2022-09-09T22:36:01.386 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Lh6aMobv3i6UPRbVkCOayS7jfNEjusm'), ('x-amz-request-id', 'tx000006c147e5bba97100d-00631bbfd1-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:01.387 DEBUG:rgw_multi.tests:version2 id=Lh6aMobv3i6UPRbVkCOayS7jfNEjusm 2022-09-09T22:36:01.387 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:36:01.388 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:36:01.388 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=Lh6aMobv3i6UPRbVkCOayS7jfNEjusm 2022-09-09T22:36:01.389 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=Lh6aMobv3i6UPRbVkCOayS7jfNEjusm 2022-09-09T22:36:01.389 DEBUG:boto:Method: DELETE 2022-09-09T22:36:01.390 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=Lh6aMobv3i6UPRbVkCOayS7jfNEjusm 2022-09-09T22:36:01.390 DEBUG:boto:Data: 2022-09-09T22:36:01.391 DEBUG:boto:Headers: {} 2022-09-09T22:36:01.392 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:36:01.392 DEBUG:boto:Port: 8000 2022-09-09T22:36:01.393 DEBUG:boto:Params: {} 2022-09-09T22:36:01.393 DEBUG:boto:Token: None 2022-09-09T22:36:01.394 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:36:01 GMT /jkcgbj-16/obj-test-zone1?versionId=Lh6aMobv3i6UPRbVkCOayS7jfNEjusm 2022-09-09T22:36:01.394 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ruW8lqL1Eg0S/NZCK5XxDUCdMsg= 2022-09-09T22:36:01.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:36:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ruW8lqL1Eg0S/NZCK5XxDUCdMsg=', 'Content-Length': '0'} 2022-09-09T22:36:01.420 DEBUG:boto:Response headers: [('x-amz-version-id', 'Lh6aMobv3i6UPRbVkCOayS7jfNEjusm'), ('x-amz-request-id', 'tx00000b1808797948c7b8d-00631bbfd1-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:01.422 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:36:01.423 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:36:01.423 DEBUG:boto:Method: PUT 2022-09-09T22:36:01.424 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1 2022-09-09T22:36:01.424 DEBUG:boto:Data: 2022-09-09T22:36:01.425 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:36:01.425 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:36:01.426 DEBUG:boto:Port: 8000 2022-09-09T22:36:01.426 DEBUG:boto:Params: {} 2022-09-09T22:36:01.427 DEBUG:boto:Token: None 2022-09-09T22:36:01.428 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 09 Sep 2022 22:36:01 GMT /jkcgbj-16/obj-test-zone1 2022-09-09T22:36:01.429 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4h+Tr6XK4gfS28uw5+3IGR2+Od4= 2022-09-09T22:36:01.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:36:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4h+Tr6XK4gfS28uw5+3IGR2+Od4='} 2022-09-09T22:36:01.474 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '6659r-fxIPnPEvSiFHd12XTKFud1HKl'), ('x-amz-request-id', 'tx0000044d9be41d34dd0a6-00631bbfd1-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:01.475 DEBUG:rgw_multi.tests:version3 id=6659r-fxIPnPEvSiFHd12XTKFud1HKl 2022-09-09T22:36:01.476 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:36:01.476 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:36:01.477 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=6659r-fxIPnPEvSiFHd12XTKFud1HKl 2022-09-09T22:36:01.477 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=6659r-fxIPnPEvSiFHd12XTKFud1HKl 2022-09-09T22:36:01.478 DEBUG:boto:Method: DELETE 2022-09-09T22:36:01.478 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=6659r-fxIPnPEvSiFHd12XTKFud1HKl 2022-09-09T22:36:01.479 DEBUG:boto:Data: 2022-09-09T22:36:01.480 DEBUG:boto:Headers: {} 2022-09-09T22:36:01.480 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:36:01.481 DEBUG:boto:Port: 8000 2022-09-09T22:36:01.481 DEBUG:boto:Params: {} 2022-09-09T22:36:01.482 DEBUG:boto:Token: None 2022-09-09T22:36:01.482 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:36:01 GMT /jkcgbj-16/obj-test-zone1?versionId=6659r-fxIPnPEvSiFHd12XTKFud1HKl 2022-09-09T22:36:01.483 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JWv5V1/BJ2vd6fvZHeiOb+76fGo= 2022-09-09T22:36:01.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:36:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JWv5V1/BJ2vd6fvZHeiOb+76fGo=', 'Content-Length': '0'} 2022-09-09T22:36:01.511 DEBUG:boto:Response headers: [('x-amz-version-id', '6659r-fxIPnPEvSiFHd12XTKFud1HKl'), ('x-amz-request-id', 'tx000001966ee6283bdabad-00631bbfd1-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:01.512 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:36:01.512 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:36:01.513 DEBUG:boto:Method: HEAD 2022-09-09T22:36:01.513 DEBUG:boto:Path: /jkcgbj-16/ 2022-09-09T22:36:01.514 DEBUG:boto:Data: 2022-09-09T22:36:01.514 DEBUG:boto:Headers: {} 2022-09-09T22:36:01.515 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:36:01.515 DEBUG:boto:Port: 8000 2022-09-09T22:36:01.516 DEBUG:boto:Params: {} 2022-09-09T22:36:01.517 DEBUG:boto:Token: None 2022-09-09T22:36:01.517 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:36:01 GMT /jkcgbj-16/ 2022-09-09T22:36:01.518 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qoEROyGsdc4eL3L7h/9XtQMywWw= 2022-09-09T22:36:01.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:36:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qoEROyGsdc4eL3L7h/9XtQMywWw=', 'Content-Length': '0'} 2022-09-09T22:36:01.523 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000014097640223f44b3-00631bbfd1-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:01.524 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:36:01.524 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:36:01.525 DEBUG:boto:Method: PUT 2022-09-09T22:36:01.525 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2 2022-09-09T22:36:01.526 DEBUG:boto:Data: 2022-09-09T22:36:01.526 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:36:01.527 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:36:01.527 DEBUG:boto:Port: 8000 2022-09-09T22:36:01.528 DEBUG:boto:Params: {} 2022-09-09T22:36:01.528 DEBUG:boto:Token: None 2022-09-09T22:36:01.529 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:36:01 GMT /jkcgbj-16/obj-test-zone2 2022-09-09T22:36:01.529 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:yEeQHB8Um5r+1Su/MJpGQJtalEk= 2022-09-09T22:36:01.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:36:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:yEeQHB8Um5r+1Su/MJpGQJtalEk='} 2022-09-09T22:36:01.557 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-amz-request-id', 'tx000000a8fde602c61b43b-00631bbfd1-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:01.558 DEBUG:rgw_multi.tests:version1 id=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:36:01.559 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:36:01.559 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:36:01.560 DEBUG:boto:Method: PUT 2022-09-09T22:36:01.561 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2 2022-09-09T22:36:01.561 DEBUG:boto:Data: 2022-09-09T22:36:01.562 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:36:01.563 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:36:01.563 DEBUG:boto:Port: 8000 2022-09-09T22:36:01.564 DEBUG:boto:Params: {} 2022-09-09T22:36:01.564 DEBUG:boto:Token: None 2022-09-09T22:36:01.565 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 09 Sep 2022 22:36:01 GMT /jkcgbj-16/obj-test-zone2 2022-09-09T22:36:01.565 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:aK7jy/JngWuaq8fi3Ug0w7uV0jc= 2022-09-09T22:36:01.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:36:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:aK7jy/JngWuaq8fi3Ug0w7uV0jc='} 2022-09-09T22:36:01.610 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'KZSZiA2f-1JyRlQFnDm25z9tjCiQmOD'), ('x-amz-request-id', 'tx00000b2840c38ac6f3621-00631bbfd1-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:01.611 DEBUG:rgw_multi.tests:version2 id=KZSZiA2f-1JyRlQFnDm25z9tjCiQmOD 2022-09-09T22:36:01.612 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:36:01.612 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:36:01.613 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=KZSZiA2f-1JyRlQFnDm25z9tjCiQmOD 2022-09-09T22:36:01.613 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=KZSZiA2f-1JyRlQFnDm25z9tjCiQmOD 2022-09-09T22:36:01.613 DEBUG:boto:Method: DELETE 2022-09-09T22:36:01.614 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=KZSZiA2f-1JyRlQFnDm25z9tjCiQmOD 2022-09-09T22:36:01.614 DEBUG:boto:Data: 2022-09-09T22:36:01.615 DEBUG:boto:Headers: {} 2022-09-09T22:36:01.615 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:36:01.616 DEBUG:boto:Port: 8000 2022-09-09T22:36:01.616 DEBUG:boto:Params: {} 2022-09-09T22:36:01.617 DEBUG:boto:Token: None 2022-09-09T22:36:01.618 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:36:01 GMT /jkcgbj-16/obj-test-zone2?versionId=KZSZiA2f-1JyRlQFnDm25z9tjCiQmOD 2022-09-09T22:36:01.618 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:50+vH0yT4/MuTXxeZj0Y9clzmq0= 2022-09-09T22:36:01.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:36:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:50+vH0yT4/MuTXxeZj0Y9clzmq0=', 'Content-Length': '0'} 2022-09-09T22:36:01.643 DEBUG:boto:Response headers: [('x-amz-version-id', 'KZSZiA2f-1JyRlQFnDm25z9tjCiQmOD'), ('x-amz-request-id', 'tx00000648e915c0599c7fb-00631bbfd1-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:01.645 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:36:01.646 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:36:01.647 DEBUG:boto:Method: PUT 2022-09-09T22:36:01.648 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2 2022-09-09T22:36:01.648 DEBUG:boto:Data: 2022-09-09T22:36:01.649 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:36:01.650 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:36:01.651 DEBUG:boto:Port: 8000 2022-09-09T22:36:01.652 DEBUG:boto:Params: {} 2022-09-09T22:36:01.654 DEBUG:boto:Token: None 2022-09-09T22:36:01.655 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 09 Sep 2022 22:36:01 GMT /jkcgbj-16/obj-test-zone2 2022-09-09T22:36:01.656 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:SSUoaEgN5f4L6/iF28vmPd0HMBU= 2022-09-09T22:36:01.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:36:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:SSUoaEgN5f4L6/iF28vmPd0HMBU='} 2022-09-09T22:36:01.682 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ke0N8IjkA2t1nHA9g6pAsC4HxQMvkCo'), ('x-amz-request-id', 'tx0000019b6b8b46f561db3-00631bbfd1-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:01.683 DEBUG:rgw_multi.tests:version3 id=ke0N8IjkA2t1nHA9g6pAsC4HxQMvkCo 2022-09-09T22:36:01.684 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:36:01.685 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:36:01.685 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=ke0N8IjkA2t1nHA9g6pAsC4HxQMvkCo 2022-09-09T22:36:01.686 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=ke0N8IjkA2t1nHA9g6pAsC4HxQMvkCo 2022-09-09T22:36:01.687 DEBUG:boto:Method: DELETE 2022-09-09T22:36:01.687 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=ke0N8IjkA2t1nHA9g6pAsC4HxQMvkCo 2022-09-09T22:36:01.688 DEBUG:boto:Data: 2022-09-09T22:36:01.688 DEBUG:boto:Headers: {} 2022-09-09T22:36:01.689 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:36:01.690 DEBUG:boto:Port: 8000 2022-09-09T22:36:01.690 DEBUG:boto:Params: {} 2022-09-09T22:36:01.691 DEBUG:boto:Token: None 2022-09-09T22:36:01.691 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:36:01 GMT /jkcgbj-16/obj-test-zone2?versionId=ke0N8IjkA2t1nHA9g6pAsC4HxQMvkCo 2022-09-09T22:36:01.692 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sMX0BVGGYOK2NqsdAReyuT6w5w8= 2022-09-09T22:36:01.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:36:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sMX0BVGGYOK2NqsdAReyuT6w5w8=', 'Content-Length': '0'} 2022-09-09T22:36:01.716 DEBUG:boto:Response headers: [('x-amz-version-id', 'ke0N8IjkA2t1nHA9g6pAsC4HxQMvkCo'), ('x-amz-request-id', 'tx00000c070d7125eafb715-00631bbfd1-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:01.717 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:36:01.717 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:36:01.718 DEBUG:boto:Method: HEAD 2022-09-09T22:36:01.718 DEBUG:boto:Path: /jkcgbj-16/ 2022-09-09T22:36:01.719 DEBUG:boto:Data: 2022-09-09T22:36:01.719 DEBUG:boto:Headers: {} 2022-09-09T22:36:01.720 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:36:01.720 DEBUG:boto:Port: 8001 2022-09-09T22:36:01.721 DEBUG:boto:Params: {} 2022-09-09T22:36:01.721 DEBUG:boto:Token: None 2022-09-09T22:36:01.722 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:36:01 GMT /jkcgbj-16/ 2022-09-09T22:36:01.723 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qoEROyGsdc4eL3L7h/9XtQMywWw= 2022-09-09T22:36:01.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:36:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qoEROyGsdc4eL3L7h/9XtQMywWw=', 'Content-Length': '0'} 2022-09-09T22:36:01.729 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000954ab9d064b3d9a-00631bbfd1-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:01.730 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:36:01.731 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:36:01.732 DEBUG:boto:Method: PUT 2022-09-09T22:36:01.732 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3 2022-09-09T22:36:01.733 DEBUG:boto:Data: 2022-09-09T22:36:01.734 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:36:01.734 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:36:01.735 DEBUG:boto:Port: 8001 2022-09-09T22:36:01.736 DEBUG:boto:Params: {} 2022-09-09T22:36:01.736 DEBUG:boto:Token: None 2022-09-09T22:36:01.737 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:36:01 GMT /jkcgbj-16/obj-test-zone3 2022-09-09T22:36:01.738 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:T1JeM5uRNTjT+I11IT6f8hKrZ/g= 2022-09-09T22:36:01.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:36:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:T1JeM5uRNTjT+I11IT6f8hKrZ/g='} 2022-09-09T22:36:01.760 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-amz-request-id', 'tx00000fea7bd94f283e36a-00631bbfd1-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:01.761 DEBUG:rgw_multi.tests:version1 id=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:36:01.762 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:36:01.763 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:36:01.763 DEBUG:boto:Method: PUT 2022-09-09T22:36:01.764 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3 2022-09-09T22:36:01.764 DEBUG:boto:Data: 2022-09-09T22:36:01.765 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:36:01.765 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:36:01.766 DEBUG:boto:Port: 8001 2022-09-09T22:36:01.766 DEBUG:boto:Params: {} 2022-09-09T22:36:01.767 DEBUG:boto:Token: None 2022-09-09T22:36:01.768 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 09 Sep 2022 22:36:01 GMT /jkcgbj-16/obj-test-zone3 2022-09-09T22:36:01.768 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LRprdAN3Yg3H5dhIy4eTlh/0Vkk= 2022-09-09T22:36:01.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:36:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LRprdAN3Yg3H5dhIy4eTlh/0Vkk='} 2022-09-09T22:36:01.818 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '8mkr0KCHuvW8ckq8YtH2VsM.M8BWe5V'), ('x-amz-request-id', 'tx000006a930ec60f134ccb-00631bbfd1-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:01.819 DEBUG:rgw_multi.tests:version2 id=8mkr0KCHuvW8ckq8YtH2VsM.M8BWe5V 2022-09-09T22:36:01.820 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:36:01.820 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:36:01.821 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=8mkr0KCHuvW8ckq8YtH2VsM.M8BWe5V 2022-09-09T22:36:01.821 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=8mkr0KCHuvW8ckq8YtH2VsM.M8BWe5V 2022-09-09T22:36:01.822 DEBUG:boto:Method: DELETE 2022-09-09T22:36:01.823 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=8mkr0KCHuvW8ckq8YtH2VsM.M8BWe5V 2022-09-09T22:36:01.824 DEBUG:boto:Data: 2022-09-09T22:36:01.825 DEBUG:boto:Headers: {} 2022-09-09T22:36:01.825 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:36:01.826 DEBUG:boto:Port: 8001 2022-09-09T22:36:01.827 DEBUG:boto:Params: {} 2022-09-09T22:36:01.828 DEBUG:boto:Token: None 2022-09-09T22:36:01.829 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:36:01 GMT /jkcgbj-16/obj-test-zone3?versionId=8mkr0KCHuvW8ckq8YtH2VsM.M8BWe5V 2022-09-09T22:36:01.830 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:18py8S8VQ3OxBRaAtJcFWyB2Nug= 2022-09-09T22:36:01.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:36:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:18py8S8VQ3OxBRaAtJcFWyB2Nug=', 'Content-Length': '0'} 2022-09-09T22:36:01.858 DEBUG:boto:Response headers: [('x-amz-version-id', '8mkr0KCHuvW8ckq8YtH2VsM.M8BWe5V'), ('x-amz-request-id', 'tx0000089c3fde09d939d3f-00631bbfd1-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:01.860 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:36:01.861 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:36:01.862 DEBUG:boto:Method: PUT 2022-09-09T22:36:01.863 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3 2022-09-09T22:36:01.864 DEBUG:boto:Data: 2022-09-09T22:36:01.864 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:36:01.865 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:36:01.865 DEBUG:boto:Port: 8001 2022-09-09T22:36:01.866 DEBUG:boto:Params: {} 2022-09-09T22:36:01.867 DEBUG:boto:Token: None 2022-09-09T22:36:01.867 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 09 Sep 2022 22:36:01 GMT /jkcgbj-16/obj-test-zone3 2022-09-09T22:36:01.868 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rqi2cZWqobelGwllloLdtA5kx5s= 2022-09-09T22:36:01.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:36:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rqi2cZWqobelGwllloLdtA5kx5s='} 2022-09-09T22:36:01.894 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'zZMH2xIMyhULsuO-uqAseRF1AaQSnW4'), ('x-amz-request-id', 'tx00000e8a8b19293a50ca1-00631bbfd1-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:01.895 DEBUG:rgw_multi.tests:version3 id=zZMH2xIMyhULsuO-uqAseRF1AaQSnW4 2022-09-09T22:36:01.895 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:36:01.896 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:36:01.896 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=zZMH2xIMyhULsuO-uqAseRF1AaQSnW4 2022-09-09T22:36:01.897 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=zZMH2xIMyhULsuO-uqAseRF1AaQSnW4 2022-09-09T22:36:01.897 DEBUG:boto:Method: DELETE 2022-09-09T22:36:01.898 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=zZMH2xIMyhULsuO-uqAseRF1AaQSnW4 2022-09-09T22:36:01.898 DEBUG:boto:Data: 2022-09-09T22:36:01.899 DEBUG:boto:Headers: {} 2022-09-09T22:36:01.899 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:36:01.900 DEBUG:boto:Port: 8001 2022-09-09T22:36:01.900 DEBUG:boto:Params: {} 2022-09-09T22:36:01.901 DEBUG:boto:Token: None 2022-09-09T22:36:01.902 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:36:01 GMT /jkcgbj-16/obj-test-zone3?versionId=zZMH2xIMyhULsuO-uqAseRF1AaQSnW4 2022-09-09T22:36:01.903 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:HQFXyGw4F52T1IsFNKUxMRA17xg= 2022-09-09T22:36:01.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:36:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:HQFXyGw4F52T1IsFNKUxMRA17xg=', 'Content-Length': '0'} 2022-09-09T22:36:01.927 DEBUG:boto:Response headers: [('x-amz-version-id', 'zZMH2xIMyhULsuO-uqAseRF1AaQSnW4'), ('x-amz-request-id', 'tx000001b3fa2b8f0aa1213-00631bbfd1-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:01.928 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:36:01.929 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:36:01.929 DEBUG:boto:Method: HEAD 2022-09-09T22:36:01.930 DEBUG:boto:Path: /jkcgbj-17/ 2022-09-09T22:36:01.931 DEBUG:boto:Data: 2022-09-09T22:36:01.931 DEBUG:boto:Headers: {} 2022-09-09T22:36:01.932 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:36:01.932 DEBUG:boto:Port: 8000 2022-09-09T22:36:01.933 DEBUG:boto:Params: {} 2022-09-09T22:36:01.934 DEBUG:boto:Token: None 2022-09-09T22:36:01.935 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:36:01 GMT /jkcgbj-17/ 2022-09-09T22:36:01.935 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:hju5qNOATaykElpC2Yvb+OSChmc= 2022-09-09T22:36:01.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:36:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:hju5qNOATaykElpC2Yvb+OSChmc=', 'Content-Length': '0'} 2022-09-09T22:36:01.940 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084fa56848aa523be-00631bbfd1-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:01.941 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:36:01.942 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:36:01.942 DEBUG:boto:Method: PUT 2022-09-09T22:36:01.943 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1 2022-09-09T22:36:01.943 DEBUG:boto:Data: 2022-09-09T22:36:01.944 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:36:01.944 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:36:01.945 DEBUG:boto:Port: 8000 2022-09-09T22:36:01.945 DEBUG:boto:Params: {} 2022-09-09T22:36:01.946 DEBUG:boto:Token: None 2022-09-09T22:36:01.946 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:36:01 GMT /jkcgbj-17/obj-test-zone1 2022-09-09T22:36:01.947 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:/bT9OpUgJ4OLkqp0AEK0crkxdIk= 2022-09-09T22:36:01.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:36:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:/bT9OpUgJ4OLkqp0AEK0crkxdIk='} 2022-09-09T22:36:01.974 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-amz-request-id', 'tx0000062a505adf59d5bb9-00631bbfd1-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:01.975 DEBUG:rgw_multi.tests:version1 id=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:36:01.976 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:36:01.977 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:36:01.977 DEBUG:boto:Method: PUT 2022-09-09T22:36:01.978 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1 2022-09-09T22:36:01.978 DEBUG:boto:Data: 2022-09-09T22:36:01.979 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:36:01.980 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:36:01.980 DEBUG:boto:Port: 8000 2022-09-09T22:36:01.981 DEBUG:boto:Params: {} 2022-09-09T22:36:01.981 DEBUG:boto:Token: None 2022-09-09T22:36:01.982 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 09 Sep 2022 22:36:01 GMT /jkcgbj-17/obj-test-zone1 2022-09-09T22:36:01.983 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Yhe/IWJ86uLQh2kDbiCqEnuox0g= 2022-09-09T22:36:01.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:36:01 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Yhe/IWJ86uLQh2kDbiCqEnuox0g='} 2022-09-09T22:36:02.026 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'bzN445FQefi1d7D0uXnWMFUF.xtjX2c'), ('x-amz-request-id', 'tx00000959ffb511ddb7d6e-00631bbfd1-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:02.027 DEBUG:rgw_multi.tests:version2 id=bzN445FQefi1d7D0uXnWMFUF.xtjX2c 2022-09-09T22:36:02.028 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:36:02.028 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:36:02.029 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=bzN445FQefi1d7D0uXnWMFUF.xtjX2c 2022-09-09T22:36:02.029 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=bzN445FQefi1d7D0uXnWMFUF.xtjX2c 2022-09-09T22:36:02.030 DEBUG:boto:Method: DELETE 2022-09-09T22:36:02.030 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=bzN445FQefi1d7D0uXnWMFUF.xtjX2c 2022-09-09T22:36:02.031 DEBUG:boto:Data: 2022-09-09T22:36:02.031 DEBUG:boto:Headers: {} 2022-09-09T22:36:02.032 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:36:02.032 DEBUG:boto:Port: 8000 2022-09-09T22:36:02.033 DEBUG:boto:Params: {} 2022-09-09T22:36:02.034 DEBUG:boto:Token: None 2022-09-09T22:36:02.034 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:36:02 GMT /jkcgbj-17/obj-test-zone1?versionId=bzN445FQefi1d7D0uXnWMFUF.xtjX2c 2022-09-09T22:36:02.035 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:V5xmuunJJtTOmxeoXyJKy9e8xbs= 2022-09-09T22:36:02.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:36:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:V5xmuunJJtTOmxeoXyJKy9e8xbs=', 'Content-Length': '0'} 2022-09-09T22:36:02.061 DEBUG:boto:Response headers: [('x-amz-version-id', 'bzN445FQefi1d7D0uXnWMFUF.xtjX2c'), ('x-amz-request-id', 'tx00000fca9ab3c5baf4a4a-00631bbfd2-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:02.062 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:36:02.063 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:36:02.064 DEBUG:boto:Method: PUT 2022-09-09T22:36:02.065 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1 2022-09-09T22:36:02.066 DEBUG:boto:Data: 2022-09-09T22:36:02.067 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:36:02.068 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:36:02.069 DEBUG:boto:Port: 8000 2022-09-09T22:36:02.070 DEBUG:boto:Params: {} 2022-09-09T22:36:02.071 DEBUG:boto:Token: None 2022-09-09T22:36:02.072 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 09 Sep 2022 22:36:02 GMT /jkcgbj-17/obj-test-zone1 2022-09-09T22:36:02.073 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:PvvLRggdV9ylYtlWBUHJuvkkhfs= 2022-09-09T22:36:02.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:36:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:PvvLRggdV9ylYtlWBUHJuvkkhfs='} 2022-09-09T22:36:02.099 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'X95VsyV7VemfAG48Mr1I2oRJnwJwfoX'), ('x-amz-request-id', 'tx00000f0f54bd22cb829ff-00631bbfd2-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:02.100 DEBUG:rgw_multi.tests:version3 id=X95VsyV7VemfAG48Mr1I2oRJnwJwfoX 2022-09-09T22:36:02.100 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:36:02.101 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:36:02.101 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=X95VsyV7VemfAG48Mr1I2oRJnwJwfoX 2022-09-09T22:36:02.102 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=X95VsyV7VemfAG48Mr1I2oRJnwJwfoX 2022-09-09T22:36:02.103 DEBUG:boto:Method: DELETE 2022-09-09T22:36:02.103 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=X95VsyV7VemfAG48Mr1I2oRJnwJwfoX 2022-09-09T22:36:02.104 DEBUG:boto:Data: 2022-09-09T22:36:02.105 DEBUG:boto:Headers: {} 2022-09-09T22:36:02.105 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:36:02.106 DEBUG:boto:Port: 8000 2022-09-09T22:36:02.107 DEBUG:boto:Params: {} 2022-09-09T22:36:02.108 DEBUG:boto:Token: None 2022-09-09T22:36:02.109 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:36:02 GMT /jkcgbj-17/obj-test-zone1?versionId=X95VsyV7VemfAG48Mr1I2oRJnwJwfoX 2022-09-09T22:36:02.110 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:BICLcr5hslYBulSCgJBUOE/KNvE= 2022-09-09T22:36:02.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:36:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:BICLcr5hslYBulSCgJBUOE/KNvE=', 'Content-Length': '0'} 2022-09-09T22:36:02.141 DEBUG:boto:Response headers: [('x-amz-version-id', 'X95VsyV7VemfAG48Mr1I2oRJnwJwfoX'), ('x-amz-request-id', 'tx000003e7a3e191d1dc341-00631bbfd2-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:02.142 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:36:02.143 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:36:02.144 DEBUG:boto:Method: HEAD 2022-09-09T22:36:02.145 DEBUG:boto:Path: /jkcgbj-17/ 2022-09-09T22:36:02.146 DEBUG:boto:Data: 2022-09-09T22:36:02.147 DEBUG:boto:Headers: {} 2022-09-09T22:36:02.147 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:36:02.148 DEBUG:boto:Port: 8000 2022-09-09T22:36:02.149 DEBUG:boto:Params: {} 2022-09-09T22:36:02.150 DEBUG:boto:Token: None 2022-09-09T22:36:02.151 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:36:02 GMT /jkcgbj-17/ 2022-09-09T22:36:02.151 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6Qxlw136IB0oUTcyQ8DhUpsfoYg= 2022-09-09T22:36:02.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:36:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6Qxlw136IB0oUTcyQ8DhUpsfoYg=', 'Content-Length': '0'} 2022-09-09T22:36:02.157 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000276dcb2494b111ca-00631bbfd2-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:02.158 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:36:02.158 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:36:02.159 DEBUG:boto:Method: PUT 2022-09-09T22:36:02.159 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2 2022-09-09T22:36:02.160 DEBUG:boto:Data: 2022-09-09T22:36:02.160 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:36:02.161 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:36:02.161 DEBUG:boto:Port: 8000 2022-09-09T22:36:02.162 DEBUG:boto:Params: {} 2022-09-09T22:36:02.162 DEBUG:boto:Token: None 2022-09-09T22:36:02.163 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:36:02 GMT /jkcgbj-17/obj-test-zone2 2022-09-09T22:36:02.164 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:BcPVkR+m+dOVjBBFpyNge7er/sA= 2022-09-09T22:36:02.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:36:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:BcPVkR+m+dOVjBBFpyNge7er/sA='} 2022-09-09T22:36:02.198 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-amz-request-id', 'tx000003b4248150c0c5526-00631bbfd2-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:02.200 DEBUG:rgw_multi.tests:version1 id=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:36:02.200 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:36:02.201 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:36:02.202 DEBUG:boto:Method: PUT 2022-09-09T22:36:02.202 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2 2022-09-09T22:36:02.203 DEBUG:boto:Data: 2022-09-09T22:36:02.204 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:36:02.205 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:36:02.207 DEBUG:boto:Port: 8000 2022-09-09T22:36:02.207 DEBUG:boto:Params: {} 2022-09-09T22:36:02.208 DEBUG:boto:Token: None 2022-09-09T22:36:02.210 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 09 Sep 2022 22:36:02 GMT /jkcgbj-17/obj-test-zone2 2022-09-09T22:36:02.210 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Fts+8JFFS/z4r9JxNeyTHwXpVnM= 2022-09-09T22:36:02.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:36:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Fts+8JFFS/z4r9JxNeyTHwXpVnM='} 2022-09-09T22:36:02.236 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'no4HO2zy6AKHyJG-ZzF7xC8qLId-Pv4'), ('x-amz-request-id', 'tx00000cd33328f03af2d4e-00631bbfd2-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:02.237 DEBUG:rgw_multi.tests:version2 id=no4HO2zy6AKHyJG-ZzF7xC8qLId-Pv4 2022-09-09T22:36:02.239 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:36:02.240 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:36:02.240 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=no4HO2zy6AKHyJG-ZzF7xC8qLId-Pv4 2022-09-09T22:36:02.241 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=no4HO2zy6AKHyJG-ZzF7xC8qLId-Pv4 2022-09-09T22:36:02.242 DEBUG:boto:Method: DELETE 2022-09-09T22:36:02.242 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=no4HO2zy6AKHyJG-ZzF7xC8qLId-Pv4 2022-09-09T22:36:02.243 DEBUG:boto:Data: 2022-09-09T22:36:02.243 DEBUG:boto:Headers: {} 2022-09-09T22:36:02.244 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:36:02.244 DEBUG:boto:Port: 8000 2022-09-09T22:36:02.245 DEBUG:boto:Params: {} 2022-09-09T22:36:02.245 DEBUG:boto:Token: None 2022-09-09T22:36:02.246 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:36:02 GMT /jkcgbj-17/obj-test-zone2?versionId=no4HO2zy6AKHyJG-ZzF7xC8qLId-Pv4 2022-09-09T22:36:02.247 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:i3LGN19DEbbKK8kZx7dL59kZXfA= 2022-09-09T22:36:02.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:36:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:i3LGN19DEbbKK8kZx7dL59kZXfA=', 'Content-Length': '0'} 2022-09-09T22:36:02.271 DEBUG:boto:Response headers: [('x-amz-version-id', 'no4HO2zy6AKHyJG-ZzF7xC8qLId-Pv4'), ('x-amz-request-id', 'tx00000d676f6b9dea7ec3d-00631bbfd2-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:02.272 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:36:02.273 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:36:02.274 DEBUG:boto:Method: PUT 2022-09-09T22:36:02.274 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2 2022-09-09T22:36:02.275 DEBUG:boto:Data: 2022-09-09T22:36:02.275 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:36:02.276 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:36:02.276 DEBUG:boto:Port: 8000 2022-09-09T22:36:02.276 DEBUG:boto:Params: {} 2022-09-09T22:36:02.277 DEBUG:boto:Token: None 2022-09-09T22:36:02.278 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 09 Sep 2022 22:36:02 GMT /jkcgbj-17/obj-test-zone2 2022-09-09T22:36:02.278 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:EAao91o85s7PmNh02vCVNNmSyss= 2022-09-09T22:36:02.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:36:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:EAao91o85s7PmNh02vCVNNmSyss='} 2022-09-09T22:36:02.322 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'X6Aca0pJWarNJGV0Hd12.HwKxgm.0ht'), ('x-amz-request-id', 'tx000003de68c7bde88d2a2-00631bbfd2-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:02.323 DEBUG:rgw_multi.tests:version3 id=X6Aca0pJWarNJGV0Hd12.HwKxgm.0ht 2022-09-09T22:36:02.324 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:36:02.324 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:36:02.325 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=X6Aca0pJWarNJGV0Hd12.HwKxgm.0ht 2022-09-09T22:36:02.325 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=X6Aca0pJWarNJGV0Hd12.HwKxgm.0ht 2022-09-09T22:36:02.325 DEBUG:boto:Method: DELETE 2022-09-09T22:36:02.326 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=X6Aca0pJWarNJGV0Hd12.HwKxgm.0ht 2022-09-09T22:36:02.327 DEBUG:boto:Data: 2022-09-09T22:36:02.327 DEBUG:boto:Headers: {} 2022-09-09T22:36:02.328 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:36:02.329 DEBUG:boto:Port: 8000 2022-09-09T22:36:02.330 DEBUG:boto:Params: {} 2022-09-09T22:36:02.331 DEBUG:boto:Token: None 2022-09-09T22:36:02.331 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:36:02 GMT /jkcgbj-17/obj-test-zone2?versionId=X6Aca0pJWarNJGV0Hd12.HwKxgm.0ht 2022-09-09T22:36:02.333 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UPkQzSzbXTHz94jaNp2tJR9eVXs= 2022-09-09T22:36:02.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:36:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UPkQzSzbXTHz94jaNp2tJR9eVXs=', 'Content-Length': '0'} 2022-09-09T22:36:02.364 DEBUG:boto:Response headers: [('x-amz-version-id', 'X6Aca0pJWarNJGV0Hd12.HwKxgm.0ht'), ('x-amz-request-id', 'tx00000a5c33dab53010ba5-00631bbfd2-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:02.366 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:36:02.366 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:36:02.367 DEBUG:boto:Method: HEAD 2022-09-09T22:36:02.367 DEBUG:boto:Path: /jkcgbj-17/ 2022-09-09T22:36:02.368 DEBUG:boto:Data: 2022-09-09T22:36:02.369 DEBUG:boto:Headers: {} 2022-09-09T22:36:02.369 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:36:02.370 DEBUG:boto:Port: 8001 2022-09-09T22:36:02.371 DEBUG:boto:Params: {} 2022-09-09T22:36:02.371 DEBUG:boto:Token: None 2022-09-09T22:36:02.372 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:36:02 GMT /jkcgbj-17/ 2022-09-09T22:36:02.373 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6Qxlw136IB0oUTcyQ8DhUpsfoYg= 2022-09-09T22:36:02.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:36:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6Qxlw136IB0oUTcyQ8DhUpsfoYg=', 'Content-Length': '0'} 2022-09-09T22:36:02.379 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093ddc264389601c4-00631bbfd2-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:02.381 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:36:02.382 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:36:02.383 DEBUG:boto:Method: PUT 2022-09-09T22:36:02.384 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3 2022-09-09T22:36:02.385 DEBUG:boto:Data: 2022-09-09T22:36:02.386 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:36:02.387 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:36:02.388 DEBUG:boto:Port: 8001 2022-09-09T22:36:02.388 DEBUG:boto:Params: {} 2022-09-09T22:36:02.389 DEBUG:boto:Token: None 2022-09-09T22:36:02.390 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:36:02 GMT /jkcgbj-17/obj-test-zone3 2022-09-09T22:36:02.391 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TGv+hMG35BBnov5u28TTJOjXbo8= 2022-09-09T22:36:02.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:36:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TGv+hMG35BBnov5u28TTJOjXbo8='} 2022-09-09T22:36:02.418 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-amz-request-id', 'tx00000c828ff50d6835c91-00631bbfd2-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:02.419 DEBUG:rgw_multi.tests:version1 id=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:36:02.421 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:36:02.422 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:36:02.423 DEBUG:boto:Method: PUT 2022-09-09T22:36:02.424 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3 2022-09-09T22:36:02.425 DEBUG:boto:Data: 2022-09-09T22:36:02.426 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:36:02.426 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:36:02.427 DEBUG:boto:Port: 8001 2022-09-09T22:36:02.427 DEBUG:boto:Params: {} 2022-09-09T22:36:02.428 DEBUG:boto:Token: None 2022-09-09T22:36:02.429 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 09 Sep 2022 22:36:02 GMT /jkcgbj-17/obj-test-zone3 2022-09-09T22:36:02.429 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:lEjfKcx7y8iy7FY7ERSRdUwIcE0= 2022-09-09T22:36:02.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:36:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:lEjfKcx7y8iy7FY7ERSRdUwIcE0='} 2022-09-09T22:36:02.454 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ELbjNxRIvjat4JotX71tIbhx8Gs5zjW'), ('x-amz-request-id', 'tx00000256e04989e280745-00631bbfd2-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:02.455 DEBUG:rgw_multi.tests:version2 id=ELbjNxRIvjat4JotX71tIbhx8Gs5zjW 2022-09-09T22:36:02.455 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:36:02.456 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:36:02.456 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=ELbjNxRIvjat4JotX71tIbhx8Gs5zjW 2022-09-09T22:36:02.457 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=ELbjNxRIvjat4JotX71tIbhx8Gs5zjW 2022-09-09T22:36:02.457 DEBUG:boto:Method: DELETE 2022-09-09T22:36:02.458 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=ELbjNxRIvjat4JotX71tIbhx8Gs5zjW 2022-09-09T22:36:02.458 DEBUG:boto:Data: 2022-09-09T22:36:02.459 DEBUG:boto:Headers: {} 2022-09-09T22:36:02.459 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:36:02.460 DEBUG:boto:Port: 8001 2022-09-09T22:36:02.460 DEBUG:boto:Params: {} 2022-09-09T22:36:02.461 DEBUG:boto:Token: None 2022-09-09T22:36:02.461 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:36:02 GMT /jkcgbj-17/obj-test-zone3?versionId=ELbjNxRIvjat4JotX71tIbhx8Gs5zjW 2022-09-09T22:36:02.462 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:e/rAPRRK/tG4bY5jjpGVTwRKYiw= 2022-09-09T22:36:02.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:36:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:e/rAPRRK/tG4bY5jjpGVTwRKYiw=', 'Content-Length': '0'} 2022-09-09T22:36:02.488 DEBUG:boto:Response headers: [('x-amz-version-id', 'ELbjNxRIvjat4JotX71tIbhx8Gs5zjW'), ('x-amz-request-id', 'tx00000d981e8bbd9a8e9f0-00631bbfd2-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:02.489 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:36:02.490 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:36:02.490 DEBUG:boto:Method: PUT 2022-09-09T22:36:02.491 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3 2022-09-09T22:36:02.491 DEBUG:boto:Data: 2022-09-09T22:36:02.492 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:36:02.492 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:36:02.493 DEBUG:boto:Port: 8001 2022-09-09T22:36:02.493 DEBUG:boto:Params: {} 2022-09-09T22:36:02.494 DEBUG:boto:Token: None 2022-09-09T22:36:02.494 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 09 Sep 2022 22:36:02 GMT /jkcgbj-17/obj-test-zone3 2022-09-09T22:36:02.495 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+SW7bpOdD+/9ymCWlSzS08azecY= 2022-09-09T22:36:02.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:36:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+SW7bpOdD+/9ymCWlSzS08azecY='} 2022-09-09T22:36:02.542 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'amsJyQsiie1dDFYEBI9u8csITniotyh'), ('x-amz-request-id', 'tx00000a6edaf34e15f3b16-00631bbfd2-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:02.543 DEBUG:rgw_multi.tests:version3 id=amsJyQsiie1dDFYEBI9u8csITniotyh 2022-09-09T22:36:02.544 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:36:02.545 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:36:02.545 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=amsJyQsiie1dDFYEBI9u8csITniotyh 2022-09-09T22:36:02.546 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=amsJyQsiie1dDFYEBI9u8csITniotyh 2022-09-09T22:36:02.546 DEBUG:boto:Method: DELETE 2022-09-09T22:36:02.547 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=amsJyQsiie1dDFYEBI9u8csITniotyh 2022-09-09T22:36:02.547 DEBUG:boto:Data: 2022-09-09T22:36:02.548 DEBUG:boto:Headers: {} 2022-09-09T22:36:02.548 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:36:02.549 DEBUG:boto:Port: 8001 2022-09-09T22:36:02.549 DEBUG:boto:Params: {} 2022-09-09T22:36:02.550 DEBUG:boto:Token: None 2022-09-09T22:36:02.551 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:36:02 GMT /jkcgbj-17/obj-test-zone3?versionId=amsJyQsiie1dDFYEBI9u8csITniotyh 2022-09-09T22:36:02.551 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Xti4q/jihkBsYDxGUdRSBaXQV5U= 2022-09-09T22:36:02.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:36:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Xti4q/jihkBsYDxGUdRSBaXQV5U=', 'Content-Length': '0'} 2022-09-09T22:36:02.578 DEBUG:boto:Response headers: [('x-amz-version-id', 'amsJyQsiie1dDFYEBI9u8csITniotyh'), ('x-amz-request-id', 'tx00000cb2d152b4ce08052-00631bbfd2-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:02.579 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:36:02.579 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:36:02.580 DEBUG:boto:Method: HEAD 2022-09-09T22:36:02.580 DEBUG:boto:Path: /jkcgbj-18/ 2022-09-09T22:36:02.581 DEBUG:boto:Data: 2022-09-09T22:36:02.582 DEBUG:boto:Headers: {} 2022-09-09T22:36:02.583 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:36:02.583 DEBUG:boto:Port: 8000 2022-09-09T22:36:02.584 DEBUG:boto:Params: {} 2022-09-09T22:36:02.584 DEBUG:boto:Token: None 2022-09-09T22:36:02.585 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:36:02 GMT /jkcgbj-18/ 2022-09-09T22:36:02.586 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+anEht8pw3g7pYXYpWznj7HxtLc= 2022-09-09T22:36:02.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:36:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+anEht8pw3g7pYXYpWznj7HxtLc=', 'Content-Length': '0'} 2022-09-09T22:36:02.592 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016f512430b2fe990-00631bbfd2-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:02.593 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:36:02.594 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:36:02.595 DEBUG:boto:Method: PUT 2022-09-09T22:36:02.596 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1 2022-09-09T22:36:02.596 DEBUG:boto:Data: 2022-09-09T22:36:02.597 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:36:02.597 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:36:02.598 DEBUG:boto:Port: 8000 2022-09-09T22:36:02.598 DEBUG:boto:Params: {} 2022-09-09T22:36:02.599 DEBUG:boto:Token: None 2022-09-09T22:36:02.599 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:36:02 GMT /jkcgbj-18/obj-test-zone1 2022-09-09T22:36:02.600 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:I7WPE3r52oaQNpudaeUlOSPjqog= 2022-09-09T22:36:02.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:36:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:I7WPE3r52oaQNpudaeUlOSPjqog='} 2022-09-09T22:36:02.625 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-amz-request-id', 'tx00000261794cf13f79b34-00631bbfd2-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:02.626 DEBUG:rgw_multi.tests:version1 id=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:36:02.626 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:36:02.627 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:36:02.628 DEBUG:boto:Method: PUT 2022-09-09T22:36:02.628 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1 2022-09-09T22:36:02.629 DEBUG:boto:Data: 2022-09-09T22:36:02.629 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:36:02.630 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:36:02.630 DEBUG:boto:Port: 8000 2022-09-09T22:36:02.631 DEBUG:boto:Params: {} 2022-09-09T22:36:02.631 DEBUG:boto:Token: None 2022-09-09T22:36:02.632 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 09 Sep 2022 22:36:02 GMT /jkcgbj-18/obj-test-zone1 2022-09-09T22:36:02.632 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:EV11PYVKaR8IdSN53vG9QPUFOMA= 2022-09-09T22:36:02.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:36:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:EV11PYVKaR8IdSN53vG9QPUFOMA='} 2022-09-09T22:36:02.654 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '86Z6JrIN6dpM7ui6wM9gTQmJ3Xf7p7l'), ('x-amz-request-id', 'tx000003d9f6bd46443ab6e-00631bbfd2-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:02.655 DEBUG:rgw_multi.tests:version2 id=86Z6JrIN6dpM7ui6wM9gTQmJ3Xf7p7l 2022-09-09T22:36:02.656 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:36:02.656 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:36:02.656 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=86Z6JrIN6dpM7ui6wM9gTQmJ3Xf7p7l 2022-09-09T22:36:02.657 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=86Z6JrIN6dpM7ui6wM9gTQmJ3Xf7p7l 2022-09-09T22:36:02.657 DEBUG:boto:Method: DELETE 2022-09-09T22:36:02.658 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=86Z6JrIN6dpM7ui6wM9gTQmJ3Xf7p7l 2022-09-09T22:36:02.658 DEBUG:boto:Data: 2022-09-09T22:36:02.659 DEBUG:boto:Headers: {} 2022-09-09T22:36:02.659 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:36:02.660 DEBUG:boto:Port: 8000 2022-09-09T22:36:02.660 DEBUG:boto:Params: {} 2022-09-09T22:36:02.661 DEBUG:boto:Token: None 2022-09-09T22:36:02.661 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:36:02 GMT /jkcgbj-18/obj-test-zone1?versionId=86Z6JrIN6dpM7ui6wM9gTQmJ3Xf7p7l 2022-09-09T22:36:02.662 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:04xzJ48YorZSXd0gchK9/j9QBKo= 2022-09-09T22:36:02.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:36:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:04xzJ48YorZSXd0gchK9/j9QBKo=', 'Content-Length': '0'} 2022-09-09T22:36:02.686 DEBUG:boto:Response headers: [('x-amz-version-id', '86Z6JrIN6dpM7ui6wM9gTQmJ3Xf7p7l'), ('x-amz-request-id', 'tx0000083c98e07bbf33763-00631bbfd2-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:02.687 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:36:02.688 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:36:02.689 DEBUG:boto:Method: PUT 2022-09-09T22:36:02.689 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1 2022-09-09T22:36:02.690 DEBUG:boto:Data: 2022-09-09T22:36:02.690 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:36:02.691 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:36:02.691 DEBUG:boto:Port: 8000 2022-09-09T22:36:02.692 DEBUG:boto:Params: {} 2022-09-09T22:36:02.693 DEBUG:boto:Token: None 2022-09-09T22:36:02.694 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 09 Sep 2022 22:36:02 GMT /jkcgbj-18/obj-test-zone1 2022-09-09T22:36:02.694 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9v6gSpdbvkFh5IE8j63wddQvfxk= 2022-09-09T22:36:02.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:36:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9v6gSpdbvkFh5IE8j63wddQvfxk='} 2022-09-09T22:36:02.742 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'wd1OGEAf83RL10aoUY7ai8V5xLcS48p'), ('x-amz-request-id', 'tx00000b76200ad28afc3f2-00631bbfd2-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:02.743 DEBUG:rgw_multi.tests:version3 id=wd1OGEAf83RL10aoUY7ai8V5xLcS48p 2022-09-09T22:36:02.744 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:36:02.744 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:36:02.745 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=wd1OGEAf83RL10aoUY7ai8V5xLcS48p 2022-09-09T22:36:02.746 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=wd1OGEAf83RL10aoUY7ai8V5xLcS48p 2022-09-09T22:36:02.747 DEBUG:boto:Method: DELETE 2022-09-09T22:36:02.748 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=wd1OGEAf83RL10aoUY7ai8V5xLcS48p 2022-09-09T22:36:02.748 DEBUG:boto:Data: 2022-09-09T22:36:02.749 DEBUG:boto:Headers: {} 2022-09-09T22:36:02.750 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:36:02.751 DEBUG:boto:Port: 8000 2022-09-09T22:36:02.751 DEBUG:boto:Params: {} 2022-09-09T22:36:02.752 DEBUG:boto:Token: None 2022-09-09T22:36:02.753 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:36:02 GMT /jkcgbj-18/obj-test-zone1?versionId=wd1OGEAf83RL10aoUY7ai8V5xLcS48p 2022-09-09T22:36:02.754 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:zln8yvoVyB0jDWMDMexnbmHMwDo= 2022-09-09T22:36:02.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:36:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:zln8yvoVyB0jDWMDMexnbmHMwDo=', 'Content-Length': '0'} 2022-09-09T22:36:02.776 DEBUG:boto:Response headers: [('x-amz-version-id', 'wd1OGEAf83RL10aoUY7ai8V5xLcS48p'), ('x-amz-request-id', 'tx000004adbfac934ecac7b-00631bbfd2-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:02.777 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:36:02.778 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:36:02.779 DEBUG:boto:Method: HEAD 2022-09-09T22:36:02.779 DEBUG:boto:Path: /jkcgbj-18/ 2022-09-09T22:36:02.780 DEBUG:boto:Data: 2022-09-09T22:36:02.780 DEBUG:boto:Headers: {} 2022-09-09T22:36:02.781 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:36:02.781 DEBUG:boto:Port: 8000 2022-09-09T22:36:02.782 DEBUG:boto:Params: {} 2022-09-09T22:36:02.783 DEBUG:boto:Token: None 2022-09-09T22:36:02.783 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:36:02 GMT /jkcgbj-18/ 2022-09-09T22:36:02.784 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+anEht8pw3g7pYXYpWznj7HxtLc= 2022-09-09T22:36:02.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:36:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+anEht8pw3g7pYXYpWznj7HxtLc=', 'Content-Length': '0'} 2022-09-09T22:36:02.789 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000891cc0c2d8f597e7-00631bbfd2-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:02.790 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:36:02.791 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:36:02.791 DEBUG:boto:Method: PUT 2022-09-09T22:36:02.792 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2 2022-09-09T22:36:02.792 DEBUG:boto:Data: 2022-09-09T22:36:02.793 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:36:02.793 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:36:02.794 DEBUG:boto:Port: 8000 2022-09-09T22:36:02.794 DEBUG:boto:Params: {} 2022-09-09T22:36:02.795 DEBUG:boto:Token: None 2022-09-09T22:36:02.796 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:36:02 GMT /jkcgbj-18/obj-test-zone2 2022-09-09T22:36:02.796 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4ICHR72ktVieubVI2J7QHMALl8c= 2022-09-09T22:36:02.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:36:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4ICHR72ktVieubVI2J7QHMALl8c='} 2022-09-09T22:36:02.822 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-amz-request-id', 'tx00000dbf4e1bc64502818-00631bbfd2-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:02.823 DEBUG:rgw_multi.tests:version1 id=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:36:02.824 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:36:02.825 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:36:02.826 DEBUG:boto:Method: PUT 2022-09-09T22:36:02.827 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2 2022-09-09T22:36:02.827 DEBUG:boto:Data: 2022-09-09T22:36:02.828 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:36:02.829 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:36:02.829 DEBUG:boto:Port: 8000 2022-09-09T22:36:02.830 DEBUG:boto:Params: {} 2022-09-09T22:36:02.830 DEBUG:boto:Token: None 2022-09-09T22:36:02.831 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 09 Sep 2022 22:36:02 GMT /jkcgbj-18/obj-test-zone2 2022-09-09T22:36:02.832 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:oHbtLgra72y6hu7Pnxbvz5AwKZU= 2022-09-09T22:36:02.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:36:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:oHbtLgra72y6hu7Pnxbvz5AwKZU='} 2022-09-09T22:36:02.882 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'OznvmO5qFUcmMh9.lHmcNUq1F7NgIVB'), ('x-amz-request-id', 'tx0000022ca0cfb730cd888-00631bbfd2-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:02.883 DEBUG:rgw_multi.tests:version2 id=OznvmO5qFUcmMh9.lHmcNUq1F7NgIVB 2022-09-09T22:36:02.884 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:36:02.885 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:36:02.886 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=OznvmO5qFUcmMh9.lHmcNUq1F7NgIVB 2022-09-09T22:36:02.887 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=OznvmO5qFUcmMh9.lHmcNUq1F7NgIVB 2022-09-09T22:36:02.888 DEBUG:boto:Method: DELETE 2022-09-09T22:36:02.889 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=OznvmO5qFUcmMh9.lHmcNUq1F7NgIVB 2022-09-09T22:36:02.889 DEBUG:boto:Data: 2022-09-09T22:36:02.890 DEBUG:boto:Headers: {} 2022-09-09T22:36:02.891 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:36:02.892 DEBUG:boto:Port: 8000 2022-09-09T22:36:02.893 DEBUG:boto:Params: {} 2022-09-09T22:36:02.895 DEBUG:boto:Token: None 2022-09-09T22:36:02.896 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:36:02 GMT /jkcgbj-18/obj-test-zone2?versionId=OznvmO5qFUcmMh9.lHmcNUq1F7NgIVB 2022-09-09T22:36:02.897 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:M3aejb2MMr8fTh5QeQPNsCTp7gw= 2022-09-09T22:36:02.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:36:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:M3aejb2MMr8fTh5QeQPNsCTp7gw=', 'Content-Length': '0'} 2022-09-09T22:36:02.927 DEBUG:boto:Response headers: [('x-amz-version-id', 'OznvmO5qFUcmMh9.lHmcNUq1F7NgIVB'), ('x-amz-request-id', 'tx0000060b6bf69f8c7fc70-00631bbfd2-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:02.929 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:36:02.929 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:36:02.930 DEBUG:boto:Method: PUT 2022-09-09T22:36:02.930 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2 2022-09-09T22:36:02.931 DEBUG:boto:Data: 2022-09-09T22:36:02.932 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:36:02.932 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:36:02.933 DEBUG:boto:Port: 8000 2022-09-09T22:36:02.933 DEBUG:boto:Params: {} 2022-09-09T22:36:02.934 DEBUG:boto:Token: None 2022-09-09T22:36:02.935 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 09 Sep 2022 22:36:02 GMT /jkcgbj-18/obj-test-zone2 2022-09-09T22:36:02.935 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2Ln6eky1fjyWgKEnOKVihjh4jRw= 2022-09-09T22:36:02.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:36:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2Ln6eky1fjyWgKEnOKVihjh4jRw='} 2022-09-09T22:36:02.958 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'auDS-GkWadQgJIMh48MoaqUmJh98Yco'), ('x-amz-request-id', 'tx00000f98587a275052e8d-00631bbfd2-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:02.959 DEBUG:rgw_multi.tests:version3 id=auDS-GkWadQgJIMh48MoaqUmJh98Yco 2022-09-09T22:36:02.960 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:36:02.960 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:36:02.961 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=auDS-GkWadQgJIMh48MoaqUmJh98Yco 2022-09-09T22:36:02.961 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=auDS-GkWadQgJIMh48MoaqUmJh98Yco 2022-09-09T22:36:02.962 DEBUG:boto:Method: DELETE 2022-09-09T22:36:02.962 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=auDS-GkWadQgJIMh48MoaqUmJh98Yco 2022-09-09T22:36:02.963 DEBUG:boto:Data: 2022-09-09T22:36:02.963 DEBUG:boto:Headers: {} 2022-09-09T22:36:02.964 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:36:02.964 DEBUG:boto:Port: 8000 2022-09-09T22:36:02.965 DEBUG:boto:Params: {} 2022-09-09T22:36:02.966 DEBUG:boto:Token: None 2022-09-09T22:36:02.966 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:36:02 GMT /jkcgbj-18/obj-test-zone2?versionId=auDS-GkWadQgJIMh48MoaqUmJh98Yco 2022-09-09T22:36:02.967 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Bdp217e614fS/RgMZfmicLBz82U= 2022-09-09T22:36:02.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:36:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Bdp217e614fS/RgMZfmicLBz82U=', 'Content-Length': '0'} 2022-09-09T22:36:02.995 DEBUG:boto:Response headers: [('x-amz-version-id', 'auDS-GkWadQgJIMh48MoaqUmJh98Yco'), ('x-amz-request-id', 'tx0000081f5d5735d4bdd16-00631bbfd2-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:02.996 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:36:02.996 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:36:02.997 DEBUG:boto:Method: HEAD 2022-09-09T22:36:02.998 DEBUG:boto:Path: /jkcgbj-18/ 2022-09-09T22:36:02.999 DEBUG:boto:Data: 2022-09-09T22:36:03.000 DEBUG:boto:Headers: {} 2022-09-09T22:36:03.000 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:36:03.001 DEBUG:boto:Port: 8001 2022-09-09T22:36:03.002 DEBUG:boto:Params: {} 2022-09-09T22:36:03.003 DEBUG:boto:Token: None 2022-09-09T22:36:03.003 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:36:03 GMT /jkcgbj-18/ 2022-09-09T22:36:03.004 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1MavObUfQoWCVOM3XGenSzOqjsk= 2022-09-09T22:36:03.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:36:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1MavObUfQoWCVOM3XGenSzOqjsk=', 'Content-Length': '0'} 2022-09-09T22:36:03.009 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d2bf7c00e7d8f21-00631bbfd3-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:03.009 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:36:03.010 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:36:03.011 DEBUG:boto:Method: PUT 2022-09-09T22:36:03.011 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3 2022-09-09T22:36:03.012 DEBUG:boto:Data: 2022-09-09T22:36:03.012 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:36:03.013 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:36:03.013 DEBUG:boto:Port: 8001 2022-09-09T22:36:03.014 DEBUG:boto:Params: {} 2022-09-09T22:36:03.014 DEBUG:boto:Token: None 2022-09-09T22:36:03.015 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:36:03 GMT /jkcgbj-18/obj-test-zone3 2022-09-09T22:36:03.015 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:yLARAzJoDtR6zQCgNc96wxPn2ug= 2022-09-09T22:36:03.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:36:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:yLARAzJoDtR6zQCgNc96wxPn2ug='} 2022-09-09T22:36:03.040 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-amz-request-id', 'tx0000050fd659a143e862f-00631bbfd3-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:03.041 DEBUG:rgw_multi.tests:version1 id=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:36:03.042 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:36:03.042 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:36:03.043 DEBUG:boto:Method: PUT 2022-09-09T22:36:03.043 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3 2022-09-09T22:36:03.044 DEBUG:boto:Data: 2022-09-09T22:36:03.044 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:36:03.045 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:36:03.045 DEBUG:boto:Port: 8001 2022-09-09T22:36:03.046 DEBUG:boto:Params: {} 2022-09-09T22:36:03.047 DEBUG:boto:Token: None 2022-09-09T22:36:03.047 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 09 Sep 2022 22:36:03 GMT /jkcgbj-18/obj-test-zone3 2022-09-09T22:36:03.048 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:MP9k363ackPtvXsdiG2oD5PHI6E= 2022-09-09T22:36:03.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:36:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:MP9k363ackPtvXsdiG2oD5PHI6E='} 2022-09-09T22:36:03.094 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Y3LmWI0JdR2-9ljlzePXf5aEvFmt4fJ'), ('x-amz-request-id', 'tx00000856bce51bbbfac74-00631bbfd3-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:03.095 DEBUG:rgw_multi.tests:version2 id=Y3LmWI0JdR2-9ljlzePXf5aEvFmt4fJ 2022-09-09T22:36:03.096 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:36:03.096 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:36:03.097 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=Y3LmWI0JdR2-9ljlzePXf5aEvFmt4fJ 2022-09-09T22:36:03.097 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=Y3LmWI0JdR2-9ljlzePXf5aEvFmt4fJ 2022-09-09T22:36:03.098 DEBUG:boto:Method: DELETE 2022-09-09T22:36:03.098 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=Y3LmWI0JdR2-9ljlzePXf5aEvFmt4fJ 2022-09-09T22:36:03.099 DEBUG:boto:Data: 2022-09-09T22:36:03.100 DEBUG:boto:Headers: {} 2022-09-09T22:36:03.100 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:36:03.101 DEBUG:boto:Port: 8001 2022-09-09T22:36:03.101 DEBUG:boto:Params: {} 2022-09-09T22:36:03.102 DEBUG:boto:Token: None 2022-09-09T22:36:03.103 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:36:03 GMT /jkcgbj-18/obj-test-zone3?versionId=Y3LmWI0JdR2-9ljlzePXf5aEvFmt4fJ 2022-09-09T22:36:03.103 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KtAZXoZCuCholl04QAzM5It/KI8= 2022-09-09T22:36:03.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:36:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KtAZXoZCuCholl04QAzM5It/KI8=', 'Content-Length': '0'} 2022-09-09T22:36:03.130 DEBUG:boto:Response headers: [('x-amz-version-id', 'Y3LmWI0JdR2-9ljlzePXf5aEvFmt4fJ'), ('x-amz-request-id', 'tx00000a418e1f02ad1d4e2-00631bbfd3-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:03.132 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:36:03.133 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:36:03.134 DEBUG:boto:Method: PUT 2022-09-09T22:36:03.135 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3 2022-09-09T22:36:03.135 DEBUG:boto:Data: 2022-09-09T22:36:03.136 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:36:03.137 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:36:03.138 DEBUG:boto:Port: 8001 2022-09-09T22:36:03.139 DEBUG:boto:Params: {} 2022-09-09T22:36:03.140 DEBUG:boto:Token: None 2022-09-09T22:36:03.141 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 09 Sep 2022 22:36:03 GMT /jkcgbj-18/obj-test-zone3 2022-09-09T22:36:03.143 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:/vIqkRBtK/depiFJNRCUEcL6le4= 2022-09-09T22:36:03.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:36:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:/vIqkRBtK/depiFJNRCUEcL6le4='} 2022-09-09T22:36:03.167 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'EDW-kp8GSGv8WLIFTJluRuZ8JAEtxxk'), ('x-amz-request-id', 'tx0000099ed049a47e0cd97-00631bbfd3-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:03.168 DEBUG:rgw_multi.tests:version3 id=EDW-kp8GSGv8WLIFTJluRuZ8JAEtxxk 2022-09-09T22:36:03.169 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:36:03.170 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:36:03.171 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=EDW-kp8GSGv8WLIFTJluRuZ8JAEtxxk 2022-09-09T22:36:03.172 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=EDW-kp8GSGv8WLIFTJluRuZ8JAEtxxk 2022-09-09T22:36:03.173 DEBUG:boto:Method: DELETE 2022-09-09T22:36:03.174 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=EDW-kp8GSGv8WLIFTJluRuZ8JAEtxxk 2022-09-09T22:36:03.175 DEBUG:boto:Data: 2022-09-09T22:36:03.176 DEBUG:boto:Headers: {} 2022-09-09T22:36:03.177 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:36:03.178 DEBUG:boto:Port: 8001 2022-09-09T22:36:03.179 DEBUG:boto:Params: {} 2022-09-09T22:36:03.180 DEBUG:boto:Token: None 2022-09-09T22:36:03.181 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:36:03 GMT /jkcgbj-18/obj-test-zone3?versionId=EDW-kp8GSGv8WLIFTJluRuZ8JAEtxxk 2022-09-09T22:36:03.182 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KXmN7wvUqqTe/WiT7RAnDRx4SIs= 2022-09-09T22:36:03.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:36:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KXmN7wvUqqTe/WiT7RAnDRx4SIs=', 'Content-Length': '0'} 2022-09-09T22:36:03.209 DEBUG:boto:Response headers: [('x-amz-version-id', 'EDW-kp8GSGv8WLIFTJluRuZ8JAEtxxk'), ('x-amz-request-id', 'tx00000e8393ff8ec4ca950-00631bbfd3-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:36:03.210 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jkcgbj-16 2022-09-09T22:36:03.212 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:36:03.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:36:03.214 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:36:03.267 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:36:03.268 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:36:03.348 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:36:03.342+0000 7f343114ddc0 1 waiting for incremental sync to catch up: 2022-09-09T22:36:03.349 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.21.9, , , , , ] 2022-09-09T22:36:03.349 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000010.24.5, , , , 00000000004.21.9, , , , , ] 2022-09-09T22:36:08.352 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:36:08.346+0000 7f343114ddc0 1 waiting for incremental sync to catch up: 2022-09-09T22:36:08.353 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.21.9, , , , , ] 2022-09-09T22:36:08.353 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000010.24.5, , , , 00000000004.21.9, , , , , ] 2022-09-09T22:36:13.354 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:36:13.350+0000 7f343114ddc0 1 waiting for incremental sync to catch up: 2022-09-09T22:36:13.355 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.21.9, , , , , ] 2022-09-09T22:36:13.356 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000010.24.5, , , , 00000000004.21.9, , , , , ] 2022-09-09T22:36:18.357 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:36:18.354+0000 7f343114ddc0 1 waiting for incremental sync to catch up: 2022-09-09T22:36:18.358 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.21.9, , , , , ] 2022-09-09T22:36:18.358 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000010.24.5, , , , 00000000004.21.9, , , , , ] 2022-09-09T22:36:23.360 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:36:23.354+0000 7f343114ddc0 1 waiting for incremental sync to catch up: 2022-09-09T22:36:23.360 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000008.20.11, , , , 00000000007.28.9, , , , , ] 2022-09-09T22:36:23.360 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000010.24.5, , , , 00000000004.21.9, , , , , ] 2022-09-09T22:36:28.363 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:36:28.358+0000 7f343114ddc0 1 waiting for incremental sync to catch up: 2022-09-09T22:36:28.363 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000008.20.11, , , , 00000000007.28.9, , , , , ] 2022-09-09T22:36:28.364 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000010.24.5, , , , 00000000004.21.9, , , , , ] 2022-09-09T22:36:33.365 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:36:33.362+0000 7f343114ddc0 1 waiting for incremental sync to catch up: 2022-09-09T22:36:33.366 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000008.20.11, , , , 00000000007.28.9, , , , , ] 2022-09-09T22:36:33.366 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000010.24.5, , , , 00000000004.21.9, , , , , ] 2022-09-09T22:36:38.368 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:36:38.362+0000 7f343114ddc0 1 waiting for incremental sync to catch up: 2022-09-09T22:36:38.369 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000008.20.11, , , , 00000000007.28.9, , , , , ] 2022-09-09T22:36:38.369 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000010.24.5, , , , 00000000004.21.9, , , , , ] 2022-09-09T22:36:43.371 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:36:43.366+0000 7f343114ddc0 1 waiting for incremental sync to catch up: 2022-09-09T22:36:43.372 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000008.20.11, , 00000000003.33.9, , 00000000007.28.9, , , , , ] 2022-09-09T22:36:43.372 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000010.24.5, , , , 00000000004.21.9, , , , , ] 2022-09-09T22:36:48.375 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:36:48.370+0000 7f343114ddc0 1 waiting for incremental sync to catch up: 2022-09-09T22:36:48.376 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000008.20.11, , 00000000003.33.9, , 00000000007.28.9, , , , , ] 2022-09-09T22:36:48.376 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000010.24.5, , , , 00000000004.21.9, , , , , ] 2022-09-09T22:36:53.378 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:36:53.373+0000 7f343114ddc0 1 waiting for incremental sync to catch up: 2022-09-09T22:36:53.378 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000008.20.11, , 00000000003.33.9, , 00000000007.28.9, , , , , ] 2022-09-09T22:36:53.379 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000010.24.5, , , , 00000000004.21.9, , , , , ] 2022-09-09T22:36:58.381 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:36:58.377+0000 7f343114ddc0 1 waiting for incremental sync to catch up: 2022-09-09T22:36:58.381 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000008.20.11, , 00000000003.33.9, , 00000000007.28.9, , , , , ] 2022-09-09T22:36:58.381 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000010.24.5, , , , 00000000004.21.9, , , , , ] 2022-09-09T22:37:03.383 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:37:03.377+0000 7f343114ddc0 1 bucket sync caught up with source: 2022-09-09T22:37:03.384 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000010.24.5, , 00000000003.33.9, , 00000000007.28.9, , , , , ] 2022-09-09T22:37:03.384 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000010.24.5, , , , 00000000004.21.9, , , , , ] 2022-09-09T22:37:03.384 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:37:03.377+0000 7f343114ddc0 0 bucket checkpoint complete 2022-09-09T22:37:03.411 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jkcgbj-16 2022-09-09T22:37:03.412 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:37:03.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:37:03.414 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:37:03.468 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:37:03.469 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:37:03.539 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:37:03.530+0000 7f38945a1dc0 1 bucket sync caught up with source: 2022-09-09T22:37:03.540 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000010.24.5, , 00000000003.33.9, , 00000000007.28.9, , , , , ] 2022-09-09T22:37:03.540 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000010.24.5, , 00000000003.33.9, , 00000000007.28.9, , , , , ] 2022-09-09T22:37:03.540 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:37:03.530+0000 7f38945a1dc0 0 bucket checkpoint complete 2022-09-09T22:37:03.564 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jkcgbj-16 2022-09-09T22:37:03.565 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:37:03.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:37:03.566 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:37:03.623 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:37:03.623 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:37:03.696 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:37:03.693+0000 7f99c48d7dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:37:03.697 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000010.24.5, , 00000000002.31.10, , 00000000007.28.9, , , , , ] 2022-09-09T22:37:03.697 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000010.24.5, , 00000000003.33.9, , 00000000007.28.9, , , , , ] 2022-09-09T22:37:08.699 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:37:08.693+0000 7f99c48d7dc0 1 bucket sync caught up with source: 2022-09-09T22:37:08.700 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000010.24.5, , 00000000003.33.9, , 00000000007.28.9, , , , , ] 2022-09-09T22:37:08.700 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000010.24.5, , 00000000003.33.9, , 00000000007.28.9, , , , , ] 2022-09-09T22:37:08.701 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:37:08.693+0000 7f99c48d7dc0 0 bucket checkpoint complete 2022-09-09T22:37:08.728 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jkcgbj-16 2022-09-09T22:37:08.729 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:37:08.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:37:08.731 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:37:08.790 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:37:08.790 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:37:08.853 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:37:08.846+0000 7f85a8ff9dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:37:08.854 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000003.34.9, , 00000000004.37.11, , 00000000007.55.9, , , , , ] 2022-09-09T22:37:08.855 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000003.34.9, , 00000000010.49.5, , 00000000007.55.9, , , , , ] 2022-09-09T22:37:13.857 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:37:13.850+0000 7f85a8ff9dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:37:13.858 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000003.34.9, , 00000000004.37.11, , 00000000007.55.9, , , , , ] 2022-09-09T22:37:13.858 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000003.34.9, , 00000000010.49.5, , 00000000007.55.9, , , , , ] 2022-09-09T22:37:18.860 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:37:18.850+0000 7f85a8ff9dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:37:18.861 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000003.34.9, , 00000000004.37.11, , 00000000007.55.9, , , , , ] 2022-09-09T22:37:18.861 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000003.34.9, , 00000000010.49.5, , 00000000007.55.9, , , , , ] 2022-09-09T22:37:23.863 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:37:23.854+0000 7f85a8ff9dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:37:23.863 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000003.34.9, , 00000000004.37.11, , 00000000007.55.9, , , , , ] 2022-09-09T22:37:23.864 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000003.34.9, , 00000000010.49.5, , 00000000007.55.9, , , , , ] 2022-09-09T22:37:28.867 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:37:28.858+0000 7f85a8ff9dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:37:29.010 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000003.34.9, , 00000000004.37.11, , 00000000007.55.9, , , , , ] 2022-09-09T22:37:29.010 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000003.34.9, , 00000000010.49.5, , 00000000007.55.9, , , , , ] 2022-09-09T22:37:33.869 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:37:33.861+0000 7f85a8ff9dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:37:33.872 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000003.34.9, , 00000000004.37.11, , 00000000007.55.9, , , , , ] 2022-09-09T22:37:33.873 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000003.34.9, , 00000000010.49.5, , 00000000007.55.9, , , , , ] 2022-09-09T22:37:38.872 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:37:38.861+0000 7f85a8ff9dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:37:38.873 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000003.34.9, , 00000000004.37.11, , 00000000007.55.9, , , , , ] 2022-09-09T22:37:38.873 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000003.34.9, , 00000000010.49.5, , 00000000007.55.9, , , , , ] 2022-09-09T22:37:43.875 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:37:43.869+0000 7f85a8ff9dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:37:43.876 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000003.34.9, , 00000000004.37.11, , 00000000007.55.9, , , , , ] 2022-09-09T22:37:43.876 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000003.34.9, , 00000000010.49.5, , 00000000007.55.9, , , , , ] 2022-09-09T22:37:48.879 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:37:48.873+0000 7f85a8ff9dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:37:48.881 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000003.34.9, , 00000000004.37.11, , 00000000007.55.9, , , , , ] 2022-09-09T22:37:48.881 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000003.34.9, , 00000000010.49.5, , 00000000007.55.9, , , , , ] 2022-09-09T22:37:53.882 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:37:53.877+0000 7f85a8ff9dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:37:53.882 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000003.34.9, , 00000000004.37.11, , 00000000007.55.9, , , , , ] 2022-09-09T22:37:53.883 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000003.34.9, , 00000000010.49.5, , 00000000007.55.9, , , , , ] 2022-09-09T22:37:58.885 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:37:58.881+0000 7f85a8ff9dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:37:58.885 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000003.34.9, , 00000000004.37.11, , 00000000007.55.9, , , , , ] 2022-09-09T22:37:58.885 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000003.34.9, , 00000000010.49.5, , 00000000007.55.9, , , , , ] 2022-09-09T22:38:03.887 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:38:03.881+0000 7f85a8ff9dc0 1 bucket sync caught up with source: 2022-09-09T22:38:03.888 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000003.34.9, , 00000000010.49.5, , 00000000007.55.9, , , , , ] 2022-09-09T22:38:03.888 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000003.34.9, , 00000000010.49.5, , 00000000007.55.9, , , , , ] 2022-09-09T22:38:03.888 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:38:03.881+0000 7f85a8ff9dc0 0 bucket checkpoint complete 2022-09-09T22:38:03.919 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jkcgbj-16 2022-09-09T22:38:03.920 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:03.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:03.921 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:38:03.974 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:38:03.975 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:38:04.040 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:38:04.033+0000 7f053137cdc0 1 bucket sync caught up with source: 2022-09-09T22:38:04.041 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000003.34.9, , 00000000010.49.5, , 00000000007.55.9, , , , , ] 2022-09-09T22:38:04.041 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000003.34.9, , 00000000010.49.5, , 00000000007.55.9, , , , , ] 2022-09-09T22:38:04.041 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:38:04.033+0000 7f053137cdc0 0 bucket checkpoint complete 2022-09-09T22:38:04.067 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jkcgbj-16 2022-09-09T22:38:04.068 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jkcgbj-16 2022-09-09T22:38:04.069 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:04.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:04.070 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:38:04.123 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:38:04.124 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:38:04.193 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:38:04.189+0000 7f60525c5dc0 1 bucket sync caught up with source: 2022-09-09T22:38:04.193 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000003.34.9, , 00000000003.57.9, , 00000000014.51.5, , , , , ] 2022-09-09T22:38:04.194 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000003.34.9, , 00000000003.57.9, , 00000000014.51.5, , , , , ] 2022-09-09T22:38:04.194 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:38:04.189+0000 7f60525c5dc0 0 bucket checkpoint complete 2022-09-09T22:38:04.215 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jkcgbj-16 2022-09-09T22:38:04.216 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:04.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:04.217 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:38:04.270 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:38:04.271 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:38:04.343 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:38:04.336+0000 7ff0d1422dc0 1 bucket sync caught up with source: 2022-09-09T22:38:04.344 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000003.34.9, , 00000000003.57.9, , 00000000014.51.5, , , , , ] 2022-09-09T22:38:04.344 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000003.34.9, , 00000000003.57.9, , 00000000014.51.5, , , , , ] 2022-09-09T22:38:04.344 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:38:04.336+0000 7ff0d1422dc0 0 bucket checkpoint complete 2022-09-09T22:38:04.369 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jkcgbj-16 2022-09-09T22:38:04.370 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-16 zones={test-zone2, test-zone1} 2022-09-09T22:38:04.371 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:04.372 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:04.373 DEBUG:boto:Method: HEAD 2022-09-09T22:38:04.373 DEBUG:boto:Path: /jkcgbj-16/ 2022-09-09T22:38:04.374 DEBUG:boto:Data: 2022-09-09T22:38:04.375 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.375 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:04.376 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.376 DEBUG:boto:Params: {} 2022-09-09T22:38:04.377 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T22:38:04.378 DEBUG:boto:Token: None 2022-09-09T22:38:04.379 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/ 2022-09-09T22:38:04.379 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:GEgBS25aMsbWgtiWE21R5zzetO4= 2022-09-09T22:38:04.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:GEgBS25aMsbWgtiWE21R5zzetO4=', 'Content-Length': '0'} 2022-09-09T22:38:04.387 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001729d6c39afe786b-00631bc04c-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.388 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:04.389 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:04.390 DEBUG:boto:Method: HEAD 2022-09-09T22:38:04.391 DEBUG:boto:Path: /jkcgbj-16/ 2022-09-09T22:38:04.391 DEBUG:boto:Data: 2022-09-09T22:38:04.392 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.393 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:04.394 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.394 DEBUG:boto:Params: {} 2022-09-09T22:38:04.395 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T22:38:04.396 DEBUG:boto:Token: None 2022-09-09T22:38:04.396 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/ 2022-09-09T22:38:04.397 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:GEgBS25aMsbWgtiWE21R5zzetO4= 2022-09-09T22:38:04.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:GEgBS25aMsbWgtiWE21R5zzetO4=', 'Content-Length': '0'} 2022-09-09T22:38:04.403 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000095a27bfe666d7818-00631bc04c-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.404 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:38:04.404 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:04.405 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:04.405 DEBUG:boto:path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:04.406 DEBUG:boto:auth_path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:04.406 DEBUG:boto:Method: GET 2022-09-09T22:38:04.407 DEBUG:boto:Path: /jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:04.408 DEBUG:boto:Data: 2022-09-09T22:38:04.409 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.410 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:04.410 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.411 DEBUG:boto:Params: {} 2022-09-09T22:38:04.412 DEBUG:boto:Token: None 2022-09-09T22:38:04.413 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/?versions 2022-09-09T22:38:04.414 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JO7vAvytQ0MUlwdwyPG+EV4IcCQ= 2022-09-09T22:38:04.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JO7vAvytQ0MUlwdwyPG+EV4IcCQ=', 'Content-Length': '0'} 2022-09-09T22:38:04.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010b84beaa41f4415-00631bc04c-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.423 DEBUG:boto:b'jkcgbj-16999falsedummyJ2BEYVUeYOtJqYkdQX3amzkINxiyxtstrue2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1123zMLKlbUHrE3mBcpUCJblKnWDeBXDtrue2022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2MZ.b57oN5HI9SlBmYt.8.2016m6aPMftrue2022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3AesdIyOhh8xrq24fWFi6fMM39PxhXM8true2022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:04.425 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:38:04.425 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:38:04.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:38:04.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:38:04.427 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:38:04.427 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:04.428 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:04.428 DEBUG:boto:path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:04.429 DEBUG:boto:auth_path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:04.429 DEBUG:boto:Method: GET 2022-09-09T22:38:04.430 DEBUG:boto:Path: /jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:04.430 DEBUG:boto:Data: 2022-09-09T22:38:04.431 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.431 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:04.432 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.432 DEBUG:boto:Params: {} 2022-09-09T22:38:04.433 DEBUG:boto:Token: None 2022-09-09T22:38:04.433 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/?versions 2022-09-09T22:38:04.434 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JO7vAvytQ0MUlwdwyPG+EV4IcCQ= 2022-09-09T22:38:04.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JO7vAvytQ0MUlwdwyPG+EV4IcCQ=', 'Content-Length': '0'} 2022-09-09T22:38:04.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000467e839c1dec2dff-00631bc04c-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.441 DEBUG:boto:b'jkcgbj-16999falsedummyJ2BEYVUeYOtJqYkdQX3amzkINxiyxtstrue2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1123zMLKlbUHrE3mBcpUCJblKnWDeBXDtrue2022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2MZ.b57oN5HI9SlBmYt.8.2016m6aPMftrue2022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3AesdIyOhh8xrq24fWFi6fMM39PxhXM8true2022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:04.442 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:38:04.442 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:38:04.443 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:38:04.443 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:38:04.444 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:04.444 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:04.445 DEBUG:boto:path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:04.445 DEBUG:boto:auth_path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:04.446 DEBUG:boto:Method: GET 2022-09-09T22:38:04.446 DEBUG:boto:Path: /jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:04.447 DEBUG:boto:Data: 2022-09-09T22:38:04.447 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.448 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:04.448 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.449 DEBUG:boto:Params: {} 2022-09-09T22:38:04.449 DEBUG:boto:Token: None 2022-09-09T22:38:04.450 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/?versions 2022-09-09T22:38:04.451 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JO7vAvytQ0MUlwdwyPG+EV4IcCQ= 2022-09-09T22:38:04.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JO7vAvytQ0MUlwdwyPG+EV4IcCQ=', 'Content-Length': '0'} 2022-09-09T22:38:04.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bdcef219c3be40f-00631bc04c-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.459 DEBUG:boto:b'jkcgbj-16999falsedummyJ2BEYVUeYOtJqYkdQX3amzkINxiyxtstrue2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1123zMLKlbUHrE3mBcpUCJblKnWDeBXDtrue2022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2MZ.b57oN5HI9SlBmYt.8.2016m6aPMftrue2022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3AesdIyOhh8xrq24fWFi6fMM39PxhXM8true2022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:04.460 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:04.460 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:04.461 DEBUG:boto:path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:04.461 DEBUG:boto:auth_path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:04.462 DEBUG:boto:Method: GET 2022-09-09T22:38:04.463 DEBUG:boto:Path: /jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:04.463 DEBUG:boto:Data: 2022-09-09T22:38:04.463 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.464 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:04.464 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.465 DEBUG:boto:Params: {} 2022-09-09T22:38:04.466 DEBUG:boto:Token: None 2022-09-09T22:38:04.466 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/?versions 2022-09-09T22:38:04.467 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JO7vAvytQ0MUlwdwyPG+EV4IcCQ= 2022-09-09T22:38:04.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JO7vAvytQ0MUlwdwyPG+EV4IcCQ=', 'Content-Length': '0'} 2022-09-09T22:38:04.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b92fd92d83f9dacb-00631bc04c-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.475 DEBUG:boto:b'jkcgbj-16999falsedummyJ2BEYVUeYOtJqYkdQX3amzkINxiyxtstrue2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1123zMLKlbUHrE3mBcpUCJblKnWDeBXDtrue2022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2MZ.b57oN5HI9SlBmYt.8.2016m6aPMftrue2022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3AesdIyOhh8xrq24fWFi6fMM39PxhXM8true2022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:04.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:38:04.477 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:38:04.478 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:38:04.478 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.479 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.479 DEBUG:boto:Method: GET 2022-09-09T22:38:04.480 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.480 DEBUG:boto:Data: 2022-09-09T22:38:04.481 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.481 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:04.482 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.482 DEBUG:boto:Params: {} 2022-09-09T22:38:04.483 DEBUG:boto:Token: None 2022-09-09T22:38:04.484 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.484 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:quDCLX31GTofxC8TVTq7YnqVdMs= 2022-09-09T22:38:04.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:quDCLX31GTofxC8TVTq7YnqVdMs=', 'Content-Length': '0'} 2022-09-09T22:38:04.488 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000808dc0cc32a46e5f-00631bc04c-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.489 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:38:04.489 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:38:04.490 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.490 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.491 DEBUG:boto:Method: GET 2022-09-09T22:38:04.491 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.492 DEBUG:boto:Data: 2022-09-09T22:38:04.492 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.493 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:04.493 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.494 DEBUG:boto:Params: {} 2022-09-09T22:38:04.494 DEBUG:boto:Token: None 2022-09-09T22:38:04.495 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.496 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:quDCLX31GTofxC8TVTq7YnqVdMs= 2022-09-09T22:38:04.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:quDCLX31GTofxC8TVTq7YnqVdMs=', 'Content-Length': '0'} 2022-09-09T22:38:04.500 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b46abb5944bbba9f-00631bc04c-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.500 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:38:04.501 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:38:04.501 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.502 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.502 DEBUG:boto:Method: HEAD 2022-09-09T22:38:04.503 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.503 DEBUG:boto:Data: 2022-09-09T22:38:04.504 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.504 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:04.505 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.505 DEBUG:boto:Params: {} 2022-09-09T22:38:04.506 DEBUG:boto:Token: None 2022-09-09T22:38:04.506 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.507 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CrLyyWct11QrNnkTGoFmu3/Hpjo= 2022-09-09T22:38:04.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CrLyyWct11QrNnkTGoFmu3/Hpjo=', 'Content-Length': '0'} 2022-09-09T22:38:04.510 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006d3637e0d046024e-00631bc04c-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.511 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:38:04.511 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:38:04.512 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.512 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.513 DEBUG:boto:Method: HEAD 2022-09-09T22:38:04.513 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.514 DEBUG:boto:Data: 2022-09-09T22:38:04.514 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.515 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:04.515 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.516 DEBUG:boto:Params: {} 2022-09-09T22:38:04.516 DEBUG:boto:Token: None 2022-09-09T22:38:04.517 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.517 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CrLyyWct11QrNnkTGoFmu3/Hpjo= 2022-09-09T22:38:04.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CrLyyWct11QrNnkTGoFmu3/Hpjo=', 'Content-Length': '0'} 2022-09-09T22:38:04.521 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002f9320e20a5bdb6c-00631bc04c-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:38:04.522 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:38:04.523 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:38:04.523 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.524 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.524 DEBUG:boto:Method: GET 2022-09-09T22:38:04.525 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.525 DEBUG:boto:Data: 2022-09-09T22:38:04.526 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.526 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:04.527 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.527 DEBUG:boto:Params: {} 2022-09-09T22:38:04.528 DEBUG:boto:Token: None 2022-09-09T22:38:04.528 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.529 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:quDCLX31GTofxC8TVTq7YnqVdMs= 2022-09-09T22:38:04.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:quDCLX31GTofxC8TVTq7YnqVdMs=', 'Content-Length': '0'} 2022-09-09T22:38:04.533 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008b679d2b5162c9f6-00631bc04c-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.534 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:38:04.535 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:38:04.536 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.537 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.538 DEBUG:boto:Method: GET 2022-09-09T22:38:04.539 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.540 DEBUG:boto:Data: 2022-09-09T22:38:04.541 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.542 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:04.543 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.544 DEBUG:boto:Params: {} 2022-09-09T22:38:04.545 DEBUG:boto:Token: None 2022-09-09T22:38:04.546 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.547 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:quDCLX31GTofxC8TVTq7YnqVdMs= 2022-09-09T22:38:04.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:quDCLX31GTofxC8TVTq7YnqVdMs=', 'Content-Length': '0'} 2022-09-09T22:38:04.553 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000014dcb161769081bb-00631bc04c-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.554 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:04.555 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:04.556 DEBUG:boto:path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:38:04.557 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:38:04.558 DEBUG:boto:Method: GET 2022-09-09T22:38:04.559 DEBUG:boto:Path: /jkcgbj-16/?prefix=dummy 2022-09-09T22:38:04.560 DEBUG:boto:Data: 2022-09-09T22:38:04.561 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.562 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:04.563 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.564 DEBUG:boto:Params: {} 2022-09-09T22:38:04.565 DEBUG:boto:Token: None 2022-09-09T22:38:04.566 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/ 2022-09-09T22:38:04.567 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:yPzBsmCFWYf6ccDzO48AqzmEvKc= 2022-09-09T22:38:04.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:yPzBsmCFWYf6ccDzO48AqzmEvKc=', 'Content-Length': '0'} 2022-09-09T22:38:04.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3fd3ce5fd6001e2-00631bc04c-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.614 DEBUG:boto:b'jkcgbj-16dummy1000falsedummy2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:04.615 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:04.616 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:04.616 DEBUG:boto:path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:38:04.617 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:38:04.619 DEBUG:boto:Method: GET 2022-09-09T22:38:04.620 DEBUG:boto:Path: /jkcgbj-16/?prefix=dummy 2022-09-09T22:38:04.621 DEBUG:boto:Data: 2022-09-09T22:38:04.622 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.623 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:04.624 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.625 DEBUG:boto:Params: {} 2022-09-09T22:38:04.626 DEBUG:boto:Token: None 2022-09-09T22:38:04.627 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/ 2022-09-09T22:38:04.628 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:yPzBsmCFWYf6ccDzO48AqzmEvKc= 2022-09-09T22:38:04.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:yPzBsmCFWYf6ccDzO48AqzmEvKc=', 'Content-Length': '0'} 2022-09-09T22:38:04.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005163ce789982b269-00631bc04c-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.636 DEBUG:boto:b'jkcgbj-16dummy1000falsedummy2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:04.638 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:38:04.639 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:38:04.640 DEBUG:boto:Method: HEAD 2022-09-09T22:38:04.641 DEBUG:boto:Path: /jkcgbj-16/dummy 2022-09-09T22:38:04.642 DEBUG:boto:Data: 2022-09-09T22:38:04.643 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.643 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:04.644 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.645 DEBUG:boto:Params: {} 2022-09-09T22:38:04.647 DEBUG:boto:Token: None 2022-09-09T22:38:04.648 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/dummy 2022-09-09T22:38:04.649 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KQi+HKo90tokMXjV0q9sWoeu2cU= 2022-09-09T22:38:04.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KQi+HKo90tokMXjV0q9sWoeu2cU=', 'Content-Length': '0'} 2022-09-09T22:38:04.654 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000e5a956880a3fa14-00631bc04c-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.655 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:38:04.656 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:38:04.657 DEBUG:boto:Method: HEAD 2022-09-09T22:38:04.658 DEBUG:boto:Path: /jkcgbj-16/dummy 2022-09-09T22:38:04.659 DEBUG:boto:Data: 2022-09-09T22:38:04.660 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.661 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:04.661 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.662 DEBUG:boto:Params: {} 2022-09-09T22:38:04.664 DEBUG:boto:Token: None 2022-09-09T22:38:04.664 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/dummy 2022-09-09T22:38:04.665 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KQi+HKo90tokMXjV0q9sWoeu2cU= 2022-09-09T22:38:04.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KQi+HKo90tokMXjV0q9sWoeu2cU=', 'Content-Length': '0'} 2022-09-09T22:38:04.670 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000d2ca300a650aece-00631bc04c-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:38:04.671 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:38:04.672 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:38:04.681 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.682 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.683 DEBUG:boto:Method: GET 2022-09-09T22:38:04.683 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.684 DEBUG:boto:Data: 2022-09-09T22:38:04.684 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.685 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:04.685 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.686 DEBUG:boto:Params: {} 2022-09-09T22:38:04.687 DEBUG:boto:Token: None 2022-09-09T22:38:04.687 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.688 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:quDCLX31GTofxC8TVTq7YnqVdMs= 2022-09-09T22:38:04.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:quDCLX31GTofxC8TVTq7YnqVdMs=', 'Content-Length': '0'} 2022-09-09T22:38:04.691 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a629c61668737da2-00631bc04c-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.692 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:38:04.693 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:38:04.693 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.694 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.694 DEBUG:boto:Method: GET 2022-09-09T22:38:04.695 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.696 DEBUG:boto:Data: 2022-09-09T22:38:04.696 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.697 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:04.697 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.698 DEBUG:boto:Params: {} 2022-09-09T22:38:04.698 DEBUG:boto:Token: None 2022-09-09T22:38:04.699 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:04.700 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:quDCLX31GTofxC8TVTq7YnqVdMs= 2022-09-09T22:38:04.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:quDCLX31GTofxC8TVTq7YnqVdMs=', 'Content-Length': '0'} 2022-09-09T22:38:04.704 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000802c8a3a579df1c7-00631bc04c-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.704 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:04.705 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:04.705 DEBUG:boto:path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:38:04.706 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:38:04.707 DEBUG:boto:Method: GET 2022-09-09T22:38:04.707 DEBUG:boto:Path: /jkcgbj-16/?prefix=dummy 2022-09-09T22:38:04.708 DEBUG:boto:Data: 2022-09-09T22:38:04.708 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.709 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:04.709 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.710 DEBUG:boto:Params: {} 2022-09-09T22:38:04.711 DEBUG:boto:Token: None 2022-09-09T22:38:04.711 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/ 2022-09-09T22:38:04.712 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:yPzBsmCFWYf6ccDzO48AqzmEvKc= 2022-09-09T22:38:04.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:yPzBsmCFWYf6ccDzO48AqzmEvKc=', 'Content-Length': '0'} 2022-09-09T22:38:04.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000383de2986cd84691-00631bc04c-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.719 DEBUG:boto:b'jkcgbj-16dummy1000falsedummy2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:04.720 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:04.720 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:04.721 DEBUG:boto:path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:38:04.722 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:38:04.723 DEBUG:boto:Method: GET 2022-09-09T22:38:04.724 DEBUG:boto:Path: /jkcgbj-16/?prefix=dummy 2022-09-09T22:38:04.725 DEBUG:boto:Data: 2022-09-09T22:38:04.726 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.727 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:04.728 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.729 DEBUG:boto:Params: {} 2022-09-09T22:38:04.730 DEBUG:boto:Token: None 2022-09-09T22:38:04.731 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/ 2022-09-09T22:38:04.732 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:yPzBsmCFWYf6ccDzO48AqzmEvKc= 2022-09-09T22:38:04.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:yPzBsmCFWYf6ccDzO48AqzmEvKc=', 'Content-Length': '0'} 2022-09-09T22:38:04.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003655b5db10d65d55-00631bc04c-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.739 DEBUG:boto:b'jkcgbj-16dummy1000falsedummy2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:04.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:38:04.741 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:04.742 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:04.742 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.743 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.744 DEBUG:boto:Method: GET 2022-09-09T22:38:04.744 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.745 DEBUG:boto:Data: 2022-09-09T22:38:04.745 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.746 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:04.746 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.747 DEBUG:boto:Params: {} 2022-09-09T22:38:04.748 DEBUG:boto:Token: None 2022-09-09T22:38:04.748 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.749 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ywwazIbxLD9th055096SlZVVZCY= 2022-09-09T22:38:04.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ywwazIbxLD9th055096SlZVVZCY=', 'Content-Length': '0'} 2022-09-09T22:38:04.753 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e45de1ff8dcccebc-00631bc04c-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.794 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:04.795 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:04.796 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.797 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.799 DEBUG:boto:Method: GET 2022-09-09T22:38:04.800 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.801 DEBUG:boto:Data: 2022-09-09T22:38:04.802 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.802 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:04.803 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.803 DEBUG:boto:Params: {} 2022-09-09T22:38:04.804 DEBUG:boto:Token: None 2022-09-09T22:38:04.805 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.805 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ywwazIbxLD9th055096SlZVVZCY= 2022-09-09T22:38:04.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ywwazIbxLD9th055096SlZVVZCY=', 'Content-Length': '0'} 2022-09-09T22:38:04.809 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e31deafff60388f-00631bc04c-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.810 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:04.811 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:04.811 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.812 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.812 DEBUG:boto:Method: HEAD 2022-09-09T22:38:04.813 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.813 DEBUG:boto:Data: 2022-09-09T22:38:04.814 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.814 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:04.815 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.815 DEBUG:boto:Params: {} 2022-09-09T22:38:04.816 DEBUG:boto:Token: None 2022-09-09T22:38:04.816 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.817 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:QTCF7wQXBZANKgGqPDUzdEZyTu8= 2022-09-09T22:38:04.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:QTCF7wQXBZANKgGqPDUzdEZyTu8=', 'Content-Length': '0'} 2022-09-09T22:38:04.820 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000094fcd54072380a82-00631bc04c-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.821 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:04.822 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:04.822 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.823 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.823 DEBUG:boto:Method: HEAD 2022-09-09T22:38:04.824 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.824 DEBUG:boto:Data: 2022-09-09T22:38:04.825 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.825 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:04.826 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.826 DEBUG:boto:Params: {} 2022-09-09T22:38:04.827 DEBUG:boto:Token: None 2022-09-09T22:38:04.827 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.828 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:QTCF7wQXBZANKgGqPDUzdEZyTu8= 2022-09-09T22:38:04.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:QTCF7wQXBZANKgGqPDUzdEZyTu8=', 'Content-Length': '0'} 2022-09-09T22:38:04.833 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d9b6bc33713a809-00631bc04c-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:38:04.836 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:04.837 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:04.837 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.838 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.838 DEBUG:boto:Method: GET 2022-09-09T22:38:04.839 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.839 DEBUG:boto:Data: 2022-09-09T22:38:04.840 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.840 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:04.841 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.842 DEBUG:boto:Params: {} 2022-09-09T22:38:04.842 DEBUG:boto:Token: None 2022-09-09T22:38:04.843 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.843 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ywwazIbxLD9th055096SlZVVZCY= 2022-09-09T22:38:04.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ywwazIbxLD9th055096SlZVVZCY=', 'Content-Length': '0'} 2022-09-09T22:38:04.847 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c008c0a7fd20bc61-00631bc04c-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.848 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:04.849 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:04.849 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.850 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.850 DEBUG:boto:Method: GET 2022-09-09T22:38:04.851 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.851 DEBUG:boto:Data: 2022-09-09T22:38:04.852 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.852 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:04.853 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.853 DEBUG:boto:Params: {} 2022-09-09T22:38:04.854 DEBUG:boto:Token: None 2022-09-09T22:38:04.854 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.855 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ywwazIbxLD9th055096SlZVVZCY= 2022-09-09T22:38:04.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ywwazIbxLD9th055096SlZVVZCY=', 'Content-Length': '0'} 2022-09-09T22:38:04.859 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000edbca744c2242d4a-00631bc04c-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.860 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:04.860 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:04.861 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:04.861 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:04.862 DEBUG:boto:Method: GET 2022-09-09T22:38:04.862 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:04.863 DEBUG:boto:Data: 2022-09-09T22:38:04.863 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.864 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:04.864 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.865 DEBUG:boto:Params: {} 2022-09-09T22:38:04.866 DEBUG:boto:Token: None 2022-09-09T22:38:04.866 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/ 2022-09-09T22:38:04.867 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:yPzBsmCFWYf6ccDzO48AqzmEvKc= 2022-09-09T22:38:04.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:yPzBsmCFWYf6ccDzO48AqzmEvKc=', 'Content-Length': '0'} 2022-09-09T22:38:04.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee31295502603807-00631bc04c-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.874 DEBUG:boto:b'jkcgbj-16obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:04.875 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:04.875 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:04.876 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:04.876 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:04.877 DEBUG:boto:Method: GET 2022-09-09T22:38:04.877 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:04.878 DEBUG:boto:Data: 2022-09-09T22:38:04.878 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.879 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:04.879 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.880 DEBUG:boto:Params: {} 2022-09-09T22:38:04.881 DEBUG:boto:Token: None 2022-09-09T22:38:04.881 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/ 2022-09-09T22:38:04.882 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:yPzBsmCFWYf6ccDzO48AqzmEvKc= 2022-09-09T22:38:04.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:yPzBsmCFWYf6ccDzO48AqzmEvKc=', 'Content-Length': '0'} 2022-09-09T22:38:04.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae7f4aa393fffdf0-00631bc04c-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.934 DEBUG:boto:b'jkcgbj-16obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:04.935 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:04.936 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:04.936 DEBUG:boto:Method: HEAD 2022-09-09T22:38:04.937 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1 2022-09-09T22:38:04.937 DEBUG:boto:Data: 2022-09-09T22:38:04.938 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.938 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:04.939 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.939 DEBUG:boto:Params: {} 2022-09-09T22:38:04.940 DEBUG:boto:Token: None 2022-09-09T22:38:04.940 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/obj-test-zone1 2022-09-09T22:38:04.941 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rcw0TYOnJzsokMYI5FsD4XjeeX0= 2022-09-09T22:38:04.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rcw0TYOnJzsokMYI5FsD4XjeeX0=', 'Content-Length': '0'} 2022-09-09T22:38:04.946 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c482ff22d66fc187-00631bc04c-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.946 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:04.947 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:04.947 DEBUG:boto:Method: HEAD 2022-09-09T22:38:04.948 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1 2022-09-09T22:38:04.948 DEBUG:boto:Data: 2022-09-09T22:38:04.949 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.949 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:04.950 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.950 DEBUG:boto:Params: {} 2022-09-09T22:38:04.951 DEBUG:boto:Token: None 2022-09-09T22:38:04.952 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/obj-test-zone1 2022-09-09T22:38:04.952 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rcw0TYOnJzsokMYI5FsD4XjeeX0= 2022-09-09T22:38:04.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rcw0TYOnJzsokMYI5FsD4XjeeX0=', 'Content-Length': '0'} 2022-09-09T22:38:04.957 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e74bca132453e1e0-00631bc04c-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:38:04.958 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:04.959 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:04.959 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.960 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.960 DEBUG:boto:Method: GET 2022-09-09T22:38:04.961 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.961 DEBUG:boto:Data: 2022-09-09T22:38:04.962 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.962 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:04.963 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.963 DEBUG:boto:Params: {} 2022-09-09T22:38:04.964 DEBUG:boto:Token: None 2022-09-09T22:38:04.965 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.965 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ywwazIbxLD9th055096SlZVVZCY= 2022-09-09T22:38:04.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ywwazIbxLD9th055096SlZVVZCY=', 'Content-Length': '0'} 2022-09-09T22:38:04.969 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eaaf91772e33b442-00631bc04c-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.970 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:04.971 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:04.971 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.972 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.972 DEBUG:boto:Method: GET 2022-09-09T22:38:04.973 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.974 DEBUG:boto:Data: 2022-09-09T22:38:04.974 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.975 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:04.975 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.976 DEBUG:boto:Params: {} 2022-09-09T22:38:04.976 DEBUG:boto:Token: None 2022-09-09T22:38:04.977 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:04.978 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ywwazIbxLD9th055096SlZVVZCY= 2022-09-09T22:38:04.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ywwazIbxLD9th055096SlZVVZCY=', 'Content-Length': '0'} 2022-09-09T22:38:04.981 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001cbeca31709e25f1-00631bc04c-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.982 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:04.983 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:04.983 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:04.984 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:04.984 DEBUG:boto:Method: GET 2022-09-09T22:38:04.985 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:04.985 DEBUG:boto:Data: 2022-09-09T22:38:04.986 DEBUG:boto:Headers: {} 2022-09-09T22:38:04.986 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:04.987 DEBUG:boto:Port: 8000 2022-09-09T22:38:04.987 DEBUG:boto:Params: {} 2022-09-09T22:38:04.988 DEBUG:boto:Token: None 2022-09-09T22:38:04.988 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:04 GMT /jkcgbj-16/ 2022-09-09T22:38:04.989 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:yPzBsmCFWYf6ccDzO48AqzmEvKc= 2022-09-09T22:38:04.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:yPzBsmCFWYf6ccDzO48AqzmEvKc=', 'Content-Length': '0'} 2022-09-09T22:38:04.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e4b2fe19b3c8bfe-00631bc04c-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:04.997 DEBUG:boto:b'jkcgbj-16obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:04.998 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:04.998 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:04.999 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:04.999 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:05.000 DEBUG:boto:Method: GET 2022-09-09T22:38:05.000 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:05.001 DEBUG:boto:Data: 2022-09-09T22:38:05.001 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.002 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:05.002 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.003 DEBUG:boto:Params: {} 2022-09-09T22:38:05.003 DEBUG:boto:Token: None 2022-09-09T22:38:05.004 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/ 2022-09-09T22:38:05.005 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1UGCSIQDgmldL2bcf832iLWJLfM= 2022-09-09T22:38:05.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1UGCSIQDgmldL2bcf832iLWJLfM=', 'Content-Length': '0'} 2022-09-09T22:38:05.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001daf8c65e9972795-00631bc04d-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.012 DEBUG:boto:b'jkcgbj-16obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:05.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:38:05.013 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:05.014 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:05.014 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.015 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.015 DEBUG:boto:Method: GET 2022-09-09T22:38:05.016 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.016 DEBUG:boto:Data: 2022-09-09T22:38:05.017 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.017 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:05.018 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.018 DEBUG:boto:Params: {} 2022-09-09T22:38:05.019 DEBUG:boto:Token: None 2022-09-09T22:38:05.020 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.020 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:HtIjRLyOAX28CpH2uMk1ienZBlU= 2022-09-09T22:38:05.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:HtIjRLyOAX28CpH2uMk1ienZBlU=', 'Content-Length': '0'} 2022-09-09T22:38:05.024 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000005f827bbb610b678-00631bc04d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.066 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:05.067 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:05.067 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.068 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.068 DEBUG:boto:Method: GET 2022-09-09T22:38:05.069 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.069 DEBUG:boto:Data: 2022-09-09T22:38:05.070 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.070 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:05.071 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.072 DEBUG:boto:Params: {} 2022-09-09T22:38:05.072 DEBUG:boto:Token: None 2022-09-09T22:38:05.073 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.073 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:HtIjRLyOAX28CpH2uMk1ienZBlU= 2022-09-09T22:38:05.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:HtIjRLyOAX28CpH2uMk1ienZBlU=', 'Content-Length': '0'} 2022-09-09T22:38:05.077 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000009d450897c95effa-00631bc04d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.079 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:05.079 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:05.080 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.080 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.081 DEBUG:boto:Method: HEAD 2022-09-09T22:38:05.082 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.083 DEBUG:boto:Data: 2022-09-09T22:38:05.084 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.085 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:05.086 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.086 DEBUG:boto:Params: {} 2022-09-09T22:38:05.087 DEBUG:boto:Token: None 2022-09-09T22:38:05.088 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.089 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:b2eWBjb9jBekXOQQ9t6O/OtK0ts= 2022-09-09T22:38:05.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:b2eWBjb9jBekXOQQ9t6O/OtK0ts=', 'Content-Length': '0'} 2022-09-09T22:38:05.093 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000089af075edd60173b-00631bc04d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.093 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:05.094 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:05.094 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.095 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.095 DEBUG:boto:Method: HEAD 2022-09-09T22:38:05.096 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.097 DEBUG:boto:Data: 2022-09-09T22:38:05.098 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.098 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:05.099 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.099 DEBUG:boto:Params: {} 2022-09-09T22:38:05.100 DEBUG:boto:Token: None 2022-09-09T22:38:05.100 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.101 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:b2eWBjb9jBekXOQQ9t6O/OtK0ts= 2022-09-09T22:38:05.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:b2eWBjb9jBekXOQQ9t6O/OtK0ts=', 'Content-Length': '0'} 2022-09-09T22:38:05.105 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b15673f70057ad7d-00631bc04d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:38:05.106 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:05.107 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:05.108 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.108 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.109 DEBUG:boto:Method: GET 2022-09-09T22:38:05.109 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.110 DEBUG:boto:Data: 2022-09-09T22:38:05.110 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.111 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:05.112 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.112 DEBUG:boto:Params: {} 2022-09-09T22:38:05.113 DEBUG:boto:Token: None 2022-09-09T22:38:05.114 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.114 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:HtIjRLyOAX28CpH2uMk1ienZBlU= 2022-09-09T22:38:05.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:HtIjRLyOAX28CpH2uMk1ienZBlU=', 'Content-Length': '0'} 2022-09-09T22:38:05.118 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e5733f584cbe14ed-00631bc04d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.119 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:05.120 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:05.120 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.121 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.122 DEBUG:boto:Method: GET 2022-09-09T22:38:05.123 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.123 DEBUG:boto:Data: 2022-09-09T22:38:05.124 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.124 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:05.125 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.125 DEBUG:boto:Params: {} 2022-09-09T22:38:05.126 DEBUG:boto:Token: None 2022-09-09T22:38:05.127 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.128 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:HtIjRLyOAX28CpH2uMk1ienZBlU= 2022-09-09T22:38:05.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:HtIjRLyOAX28CpH2uMk1ienZBlU=', 'Content-Length': '0'} 2022-09-09T22:38:05.132 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1d6f7f269078734-00631bc04d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.133 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:05.133 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:05.134 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:05.135 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:05.135 DEBUG:boto:Method: GET 2022-09-09T22:38:05.136 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:05.137 DEBUG:boto:Data: 2022-09-09T22:38:05.138 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.138 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:05.139 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.139 DEBUG:boto:Params: {} 2022-09-09T22:38:05.140 DEBUG:boto:Token: None 2022-09-09T22:38:05.141 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/ 2022-09-09T22:38:05.141 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1UGCSIQDgmldL2bcf832iLWJLfM= 2022-09-09T22:38:05.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1UGCSIQDgmldL2bcf832iLWJLfM=', 'Content-Length': '0'} 2022-09-09T22:38:05.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cdafbc2c623a9be-00631bc04d-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.149 DEBUG:boto:b'jkcgbj-16obj-test-zone21000falseobj-test-zone22022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:05.149 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:05.150 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:05.150 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:05.151 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:05.151 DEBUG:boto:Method: GET 2022-09-09T22:38:05.152 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:05.152 DEBUG:boto:Data: 2022-09-09T22:38:05.153 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.153 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:05.154 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.154 DEBUG:boto:Params: {} 2022-09-09T22:38:05.155 DEBUG:boto:Token: None 2022-09-09T22:38:05.156 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/ 2022-09-09T22:38:05.156 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1UGCSIQDgmldL2bcf832iLWJLfM= 2022-09-09T22:38:05.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1UGCSIQDgmldL2bcf832iLWJLfM=', 'Content-Length': '0'} 2022-09-09T22:38:05.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c76d446e8fb068d-00631bc04d-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.164 DEBUG:boto:b'jkcgbj-16obj-test-zone21000falseobj-test-zone22022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:05.165 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:05.165 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:05.166 DEBUG:boto:Method: HEAD 2022-09-09T22:38:05.166 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2 2022-09-09T22:38:05.167 DEBUG:boto:Data: 2022-09-09T22:38:05.167 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.168 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:05.168 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.169 DEBUG:boto:Params: {} 2022-09-09T22:38:05.169 DEBUG:boto:Token: None 2022-09-09T22:38:05.170 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/obj-test-zone2 2022-09-09T22:38:05.170 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dByRx46tEw5gVGN4uPWxO+D4his= 2022-09-09T22:38:05.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dByRx46tEw5gVGN4uPWxO+D4his=', 'Content-Length': '0'} 2022-09-09T22:38:05.175 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cef02d5a38a58ec4-00631bc04d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.176 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:05.176 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:05.177 DEBUG:boto:Method: HEAD 2022-09-09T22:38:05.177 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2 2022-09-09T22:38:05.178 DEBUG:boto:Data: 2022-09-09T22:38:05.178 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.179 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:05.179 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.180 DEBUG:boto:Params: {} 2022-09-09T22:38:05.180 DEBUG:boto:Token: None 2022-09-09T22:38:05.181 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/obj-test-zone2 2022-09-09T22:38:05.181 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dByRx46tEw5gVGN4uPWxO+D4his= 2022-09-09T22:38:05.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dByRx46tEw5gVGN4uPWxO+D4his=', 'Content-Length': '0'} 2022-09-09T22:38:05.186 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c367b7f98f23338b-00631bc04d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:38:05.187 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:05.188 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:05.188 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.189 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.189 DEBUG:boto:Method: GET 2022-09-09T22:38:05.190 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.190 DEBUG:boto:Data: 2022-09-09T22:38:05.191 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.191 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:05.192 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.192 DEBUG:boto:Params: {} 2022-09-09T22:38:05.193 DEBUG:boto:Token: None 2022-09-09T22:38:05.194 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.194 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:HtIjRLyOAX28CpH2uMk1ienZBlU= 2022-09-09T22:38:05.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:HtIjRLyOAX28CpH2uMk1ienZBlU=', 'Content-Length': '0'} 2022-09-09T22:38:05.198 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037c993e846f5773a-00631bc04d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.242 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:05.243 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:05.243 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.244 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.244 DEBUG:boto:Method: GET 2022-09-09T22:38:05.245 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.246 DEBUG:boto:Data: 2022-09-09T22:38:05.246 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.247 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:05.247 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.248 DEBUG:boto:Params: {} 2022-09-09T22:38:05.248 DEBUG:boto:Token: None 2022-09-09T22:38:05.249 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:05.250 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:HtIjRLyOAX28CpH2uMk1ienZBlU= 2022-09-09T22:38:05.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:HtIjRLyOAX28CpH2uMk1ienZBlU=', 'Content-Length': '0'} 2022-09-09T22:38:05.253 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000095f1c2918e7517e7-00631bc04d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.254 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:05.255 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:05.255 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:05.256 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:05.256 DEBUG:boto:Method: GET 2022-09-09T22:38:05.257 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:05.257 DEBUG:boto:Data: 2022-09-09T22:38:05.258 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.258 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:05.259 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.259 DEBUG:boto:Params: {} 2022-09-09T22:38:05.260 DEBUG:boto:Token: None 2022-09-09T22:38:05.261 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/ 2022-09-09T22:38:05.261 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1UGCSIQDgmldL2bcf832iLWJLfM= 2022-09-09T22:38:05.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1UGCSIQDgmldL2bcf832iLWJLfM=', 'Content-Length': '0'} 2022-09-09T22:38:05.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdc5d6e7182c6bdc-00631bc04d-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.268 DEBUG:boto:b'jkcgbj-16obj-test-zone21000falseobj-test-zone22022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:05.269 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:05.269 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:05.270 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:05.270 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:05.271 DEBUG:boto:Method: GET 2022-09-09T22:38:05.272 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:05.272 DEBUG:boto:Data: 2022-09-09T22:38:05.273 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.273 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:05.274 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.274 DEBUG:boto:Params: {} 2022-09-09T22:38:05.275 DEBUG:boto:Token: None 2022-09-09T22:38:05.275 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/ 2022-09-09T22:38:05.276 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1UGCSIQDgmldL2bcf832iLWJLfM= 2022-09-09T22:38:05.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1UGCSIQDgmldL2bcf832iLWJLfM=', 'Content-Length': '0'} 2022-09-09T22:38:05.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000470a0fa8aeed0fab-00631bc04d-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.283 DEBUG:boto:b'jkcgbj-16obj-test-zone21000falseobj-test-zone22022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:05.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:38:05.284 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:05.285 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:05.285 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.286 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.287 DEBUG:boto:Method: GET 2022-09-09T22:38:05.287 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.288 DEBUG:boto:Data: 2022-09-09T22:38:05.288 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.289 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:05.289 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.290 DEBUG:boto:Params: {} 2022-09-09T22:38:05.290 DEBUG:boto:Token: None 2022-09-09T22:38:05.291 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.291 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:irlum+V5wSiVwzPWEtgQDr3VdGI= 2022-09-09T22:38:05.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:irlum+V5wSiVwzPWEtgQDr3VdGI=', 'Content-Length': '0'} 2022-09-09T22:38:05.295 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e626da686903608-00631bc04d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.296 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:05.297 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:05.297 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.298 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.298 DEBUG:boto:Method: GET 2022-09-09T22:38:05.299 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.299 DEBUG:boto:Data: 2022-09-09T22:38:05.300 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.300 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:05.301 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.301 DEBUG:boto:Params: {} 2022-09-09T22:38:05.302 DEBUG:boto:Token: None 2022-09-09T22:38:05.303 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.303 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:irlum+V5wSiVwzPWEtgQDr3VdGI= 2022-09-09T22:38:05.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:irlum+V5wSiVwzPWEtgQDr3VdGI=', 'Content-Length': '0'} 2022-09-09T22:38:05.307 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a26d759476d1ae39-00631bc04d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.308 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:05.308 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:05.309 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.309 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.310 DEBUG:boto:Method: HEAD 2022-09-09T22:38:05.310 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.311 DEBUG:boto:Data: 2022-09-09T22:38:05.311 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.312 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:05.312 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.313 DEBUG:boto:Params: {} 2022-09-09T22:38:05.314 DEBUG:boto:Token: None 2022-09-09T22:38:05.314 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.315 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8W+RHTFlfDNguORUo59OmjUl0oc= 2022-09-09T22:38:05.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8W+RHTFlfDNguORUo59OmjUl0oc=', 'Content-Length': '0'} 2022-09-09T22:38:05.318 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d6262d07701f39e-00631bc04d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.319 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:05.319 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:05.320 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.320 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.321 DEBUG:boto:Method: HEAD 2022-09-09T22:38:05.321 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.322 DEBUG:boto:Data: 2022-09-09T22:38:05.322 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.323 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:05.323 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.324 DEBUG:boto:Params: {} 2022-09-09T22:38:05.325 DEBUG:boto:Token: None 2022-09-09T22:38:05.325 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.326 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8W+RHTFlfDNguORUo59OmjUl0oc= 2022-09-09T22:38:05.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8W+RHTFlfDNguORUo59OmjUl0oc=', 'Content-Length': '0'} 2022-09-09T22:38:05.329 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000081cd9c413c66be79-00631bc04d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:38:05.331 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:05.331 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:05.332 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.332 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.333 DEBUG:boto:Method: GET 2022-09-09T22:38:05.333 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.334 DEBUG:boto:Data: 2022-09-09T22:38:05.335 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.335 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:05.336 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.336 DEBUG:boto:Params: {} 2022-09-09T22:38:05.337 DEBUG:boto:Token: None 2022-09-09T22:38:05.337 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.338 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:irlum+V5wSiVwzPWEtgQDr3VdGI= 2022-09-09T22:38:05.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:irlum+V5wSiVwzPWEtgQDr3VdGI=', 'Content-Length': '0'} 2022-09-09T22:38:05.342 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf36f4c1652495c2-00631bc04d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.382 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:05.383 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:05.383 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.384 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.384 DEBUG:boto:Method: GET 2022-09-09T22:38:05.385 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.385 DEBUG:boto:Data: 2022-09-09T22:38:05.386 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.387 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:05.387 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.388 DEBUG:boto:Params: {} 2022-09-09T22:38:05.388 DEBUG:boto:Token: None 2022-09-09T22:38:05.389 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.390 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:irlum+V5wSiVwzPWEtgQDr3VdGI= 2022-09-09T22:38:05.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:irlum+V5wSiVwzPWEtgQDr3VdGI=', 'Content-Length': '0'} 2022-09-09T22:38:05.394 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005bd7acb8e9bbfd47-00631bc04d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.394 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:05.395 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:05.396 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:05.396 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:05.397 DEBUG:boto:Method: GET 2022-09-09T22:38:05.397 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:05.398 DEBUG:boto:Data: 2022-09-09T22:38:05.398 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.399 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:05.400 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.400 DEBUG:boto:Params: {} 2022-09-09T22:38:05.401 DEBUG:boto:Token: None 2022-09-09T22:38:05.401 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/ 2022-09-09T22:38:05.402 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1UGCSIQDgmldL2bcf832iLWJLfM= 2022-09-09T22:38:05.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1UGCSIQDgmldL2bcf832iLWJLfM=', 'Content-Length': '0'} 2022-09-09T22:38:05.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c31e53afeb9a57db-00631bc04d-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.409 DEBUG:boto:b'jkcgbj-16obj-test-zone31000falseobj-test-zone32022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:05.410 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:05.411 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:05.411 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:05.412 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:05.412 DEBUG:boto:Method: GET 2022-09-09T22:38:05.413 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:05.414 DEBUG:boto:Data: 2022-09-09T22:38:05.414 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.415 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:05.415 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.416 DEBUG:boto:Params: {} 2022-09-09T22:38:05.416 DEBUG:boto:Token: None 2022-09-09T22:38:05.417 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/ 2022-09-09T22:38:05.417 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1UGCSIQDgmldL2bcf832iLWJLfM= 2022-09-09T22:38:05.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1UGCSIQDgmldL2bcf832iLWJLfM=', 'Content-Length': '0'} 2022-09-09T22:38:05.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094347f6734612dfb-00631bc04d-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.425 DEBUG:boto:b'jkcgbj-16obj-test-zone31000falseobj-test-zone32022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:05.426 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:05.426 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:05.427 DEBUG:boto:Method: HEAD 2022-09-09T22:38:05.427 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3 2022-09-09T22:38:05.428 DEBUG:boto:Data: 2022-09-09T22:38:05.428 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.429 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:05.429 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.430 DEBUG:boto:Params: {} 2022-09-09T22:38:05.431 DEBUG:boto:Token: None 2022-09-09T22:38:05.431 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/obj-test-zone3 2022-09-09T22:38:05.432 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vIYG2eEJd7B/Bh4rPg0NEOlcbsM= 2022-09-09T22:38:05.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vIYG2eEJd7B/Bh4rPg0NEOlcbsM=', 'Content-Length': '0'} 2022-09-09T22:38:05.436 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ff465024c55dca6-00631bc04d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.437 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:05.438 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:05.438 DEBUG:boto:Method: HEAD 2022-09-09T22:38:05.439 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3 2022-09-09T22:38:05.439 DEBUG:boto:Data: 2022-09-09T22:38:05.440 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.440 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:05.441 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.441 DEBUG:boto:Params: {} 2022-09-09T22:38:05.442 DEBUG:boto:Token: None 2022-09-09T22:38:05.443 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/obj-test-zone3 2022-09-09T22:38:05.443 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vIYG2eEJd7B/Bh4rPg0NEOlcbsM= 2022-09-09T22:38:05.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vIYG2eEJd7B/Bh4rPg0NEOlcbsM=', 'Content-Length': '0'} 2022-09-09T22:38:05.448 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a672c9c25a6f4fff-00631bc04d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:38:05.449 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:05.450 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:05.450 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.451 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.451 DEBUG:boto:Method: GET 2022-09-09T22:38:05.452 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.453 DEBUG:boto:Data: 2022-09-09T22:38:05.453 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.454 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:05.454 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.455 DEBUG:boto:Params: {} 2022-09-09T22:38:05.455 DEBUG:boto:Token: None 2022-09-09T22:38:05.456 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.457 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:irlum+V5wSiVwzPWEtgQDr3VdGI= 2022-09-09T22:38:05.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:irlum+V5wSiVwzPWEtgQDr3VdGI=', 'Content-Length': '0'} 2022-09-09T22:38:05.460 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008fadef3989b3f20a-00631bc04d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.506 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:05.507 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:05.507 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.508 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.508 DEBUG:boto:Method: GET 2022-09-09T22:38:05.509 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.510 DEBUG:boto:Data: 2022-09-09T22:38:05.510 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.511 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:05.511 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.512 DEBUG:boto:Params: {} 2022-09-09T22:38:05.512 DEBUG:boto:Token: None 2022-09-09T22:38:05.513 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:05.513 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:irlum+V5wSiVwzPWEtgQDr3VdGI= 2022-09-09T22:38:05.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:irlum+V5wSiVwzPWEtgQDr3VdGI=', 'Content-Length': '0'} 2022-09-09T22:38:05.517 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037598470cea6d7bc-00631bc04d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.518 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:05.519 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:05.519 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:05.520 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:05.520 DEBUG:boto:Method: GET 2022-09-09T22:38:05.521 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:05.521 DEBUG:boto:Data: 2022-09-09T22:38:05.522 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.522 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:05.523 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.523 DEBUG:boto:Params: {} 2022-09-09T22:38:05.524 DEBUG:boto:Token: None 2022-09-09T22:38:05.525 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/ 2022-09-09T22:38:05.526 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1UGCSIQDgmldL2bcf832iLWJLfM= 2022-09-09T22:38:05.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1UGCSIQDgmldL2bcf832iLWJLfM=', 'Content-Length': '0'} 2022-09-09T22:38:05.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e8b278e6978b1da-00631bc04d-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.533 DEBUG:boto:b'jkcgbj-16obj-test-zone31000falseobj-test-zone32022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:05.534 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:05.534 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:05.535 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:05.535 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:05.536 DEBUG:boto:Method: GET 2022-09-09T22:38:05.536 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:05.537 DEBUG:boto:Data: 2022-09-09T22:38:05.537 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.538 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:05.539 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.539 DEBUG:boto:Params: {} 2022-09-09T22:38:05.540 DEBUG:boto:Token: None 2022-09-09T22:38:05.540 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/ 2022-09-09T22:38:05.541 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1UGCSIQDgmldL2bcf832iLWJLfM= 2022-09-09T22:38:05.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1UGCSIQDgmldL2bcf832iLWJLfM=', 'Content-Length': '0'} 2022-09-09T22:38:05.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007004a3a7c11801b5-00631bc04d-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.548 DEBUG:boto:b'jkcgbj-16obj-test-zone31000falseobj-test-zone32022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:05.549 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-16 zones={test-zone2, test-zone1} 2022-09-09T22:38:05.550 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-16 zones={test-zone3, test-zone1} 2022-09-09T22:38:05.550 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:05.551 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:05.551 DEBUG:boto:Method: HEAD 2022-09-09T22:38:05.552 DEBUG:boto:Path: /jkcgbj-16/ 2022-09-09T22:38:05.552 DEBUG:boto:Data: 2022-09-09T22:38:05.553 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.553 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:05.554 DEBUG:boto:Port: 8001 2022-09-09T22:38:05.554 DEBUG:boto:Params: {} 2022-09-09T22:38:05.555 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-09T22:38:05.555 DEBUG:boto:Token: None 2022-09-09T22:38:05.556 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/ 2022-09-09T22:38:05.556 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:fXPPYwYz+C4z7+pEiofwSoAhNnM= 2022-09-09T22:38:05.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:fXPPYwYz+C4z7+pEiofwSoAhNnM=', 'Content-Length': '0'} 2022-09-09T22:38:05.562 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e3072928ca255a36-00631bc04d-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.563 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:05.564 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:05.564 DEBUG:boto:Method: HEAD 2022-09-09T22:38:05.565 DEBUG:boto:Path: /jkcgbj-16/ 2022-09-09T22:38:05.565 DEBUG:boto:Data: 2022-09-09T22:38:05.566 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.566 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:05.567 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.567 DEBUG:boto:Params: {} 2022-09-09T22:38:05.568 DEBUG:boto:Token: None 2022-09-09T22:38:05.568 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/ 2022-09-09T22:38:05.569 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:fXPPYwYz+C4z7+pEiofwSoAhNnM= 2022-09-09T22:38:05.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:fXPPYwYz+C4z7+pEiofwSoAhNnM=', 'Content-Length': '0'} 2022-09-09T22:38:05.575 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b21cf9444ff0df30-00631bc04d-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.576 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:38:05.576 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:05.577 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:05.577 DEBUG:boto:path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:05.578 DEBUG:boto:auth_path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:05.578 DEBUG:boto:Method: GET 2022-09-09T22:38:05.579 DEBUG:boto:Path: /jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:05.579 DEBUG:boto:Data: 2022-09-09T22:38:05.580 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.580 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:05.581 DEBUG:boto:Port: 8001 2022-09-09T22:38:05.581 DEBUG:boto:Params: {} 2022-09-09T22:38:05.582 DEBUG:boto:Token: None 2022-09-09T22:38:05.582 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/?versions 2022-09-09T22:38:05.583 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dTFWw+hcQGnPxALDdvotQXBDWgU= 2022-09-09T22:38:05.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dTFWw+hcQGnPxALDdvotQXBDWgU=', 'Content-Length': '0'} 2022-09-09T22:38:05.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001832c92ffba21258-00631bc04d-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.593 DEBUG:boto:b'jkcgbj-16999falsedummyJ2BEYVUeYOtJqYkdQX3amzkINxiyxtstrue2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1123zMLKlbUHrE3mBcpUCJblKnWDeBXDtrue2022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2MZ.b57oN5HI9SlBmYt.8.2016m6aPMftrue2022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3AesdIyOhh8xrq24fWFi6fMM39PxhXM8true2022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:05.594 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:38:05.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:38:05.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:38:05.596 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:38:05.596 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:38:05.597 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:05.597 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:05.598 DEBUG:boto:path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:05.599 DEBUG:boto:auth_path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:05.599 DEBUG:boto:Method: GET 2022-09-09T22:38:05.600 DEBUG:boto:Path: /jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:05.600 DEBUG:boto:Data: 2022-09-09T22:38:05.601 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.601 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:05.602 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.602 DEBUG:boto:Params: {} 2022-09-09T22:38:05.603 DEBUG:boto:Token: None 2022-09-09T22:38:05.603 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/?versions 2022-09-09T22:38:05.604 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dTFWw+hcQGnPxALDdvotQXBDWgU= 2022-09-09T22:38:05.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dTFWw+hcQGnPxALDdvotQXBDWgU=', 'Content-Length': '0'} 2022-09-09T22:38:05.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d61d1dc5caaa5959-00631bc04d-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.610 DEBUG:boto:b'jkcgbj-16999falsedummyJ2BEYVUeYOtJqYkdQX3amzkINxiyxtstrue2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1123zMLKlbUHrE3mBcpUCJblKnWDeBXDtrue2022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2MZ.b57oN5HI9SlBmYt.8.2016m6aPMftrue2022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3AesdIyOhh8xrq24fWFi6fMM39PxhXM8true2022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:05.611 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:38:05.612 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:38:05.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:38:05.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:38:05.614 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:05.614 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:05.615 DEBUG:boto:path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:05.616 DEBUG:boto:auth_path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:05.617 DEBUG:boto:Method: GET 2022-09-09T22:38:05.617 DEBUG:boto:Path: /jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:05.618 DEBUG:boto:Data: 2022-09-09T22:38:05.619 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.619 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:05.619 DEBUG:boto:Port: 8001 2022-09-09T22:38:05.620 DEBUG:boto:Params: {} 2022-09-09T22:38:05.621 DEBUG:boto:Token: None 2022-09-09T22:38:05.622 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/?versions 2022-09-09T22:38:05.623 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dTFWw+hcQGnPxALDdvotQXBDWgU= 2022-09-09T22:38:05.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dTFWw+hcQGnPxALDdvotQXBDWgU=', 'Content-Length': '0'} 2022-09-09T22:38:05.629 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c12ff5f762d6734-00631bc04d-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.629 DEBUG:boto:b'jkcgbj-16999falsedummyJ2BEYVUeYOtJqYkdQX3amzkINxiyxtstrue2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1123zMLKlbUHrE3mBcpUCJblKnWDeBXDtrue2022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2MZ.b57oN5HI9SlBmYt.8.2016m6aPMftrue2022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3AesdIyOhh8xrq24fWFi6fMM39PxhXM8true2022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:05.630 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:05.631 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:05.631 DEBUG:boto:path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:05.632 DEBUG:boto:auth_path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:05.632 DEBUG:boto:Method: GET 2022-09-09T22:38:05.633 DEBUG:boto:Path: /jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:05.633 DEBUG:boto:Data: 2022-09-09T22:38:05.634 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.635 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:05.635 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.635 DEBUG:boto:Params: {} 2022-09-09T22:38:05.636 DEBUG:boto:Token: None 2022-09-09T22:38:05.637 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/?versions 2022-09-09T22:38:05.637 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dTFWw+hcQGnPxALDdvotQXBDWgU= 2022-09-09T22:38:05.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dTFWw+hcQGnPxALDdvotQXBDWgU=', 'Content-Length': '0'} 2022-09-09T22:38:05.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e9d9ce240d8047f-00631bc04d-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.643 DEBUG:boto:b'jkcgbj-16999falsedummyJ2BEYVUeYOtJqYkdQX3amzkINxiyxtstrue2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1123zMLKlbUHrE3mBcpUCJblKnWDeBXDtrue2022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2MZ.b57oN5HI9SlBmYt.8.2016m6aPMftrue2022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3AesdIyOhh8xrq24fWFi6fMM39PxhXM8true2022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:05.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:38:05.644 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:38:05.645 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:38:05.645 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.646 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.647 DEBUG:boto:Method: GET 2022-09-09T22:38:05.647 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.648 DEBUG:boto:Data: 2022-09-09T22:38:05.648 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.649 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:05.649 DEBUG:boto:Port: 8001 2022-09-09T22:38:05.650 DEBUG:boto:Params: {} 2022-09-09T22:38:05.650 DEBUG:boto:Token: None 2022-09-09T22:38:05.651 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.651 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pLxIKObeoU+jbyAm4D3av9DlRh4= 2022-09-09T22:38:05.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pLxIKObeoU+jbyAm4D3av9DlRh4=', 'Content-Length': '0'} 2022-09-09T22:38:05.655 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000056908b4c7d11c900-00631bc04d-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.656 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:38:05.656 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:38:05.657 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.657 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.658 DEBUG:boto:Method: GET 2022-09-09T22:38:05.659 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.659 DEBUG:boto:Data: 2022-09-09T22:38:05.660 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.660 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:05.661 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.661 DEBUG:boto:Params: {} 2022-09-09T22:38:05.662 DEBUG:boto:Token: None 2022-09-09T22:38:05.662 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.663 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pLxIKObeoU+jbyAm4D3av9DlRh4= 2022-09-09T22:38:05.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pLxIKObeoU+jbyAm4D3av9DlRh4=', 'Content-Length': '0'} 2022-09-09T22:38:05.666 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000094bdfd563790e29e-00631bc04d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.667 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:38:05.668 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:38:05.668 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.669 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.669 DEBUG:boto:Method: HEAD 2022-09-09T22:38:05.670 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.670 DEBUG:boto:Data: 2022-09-09T22:38:05.671 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.671 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:05.672 DEBUG:boto:Port: 8001 2022-09-09T22:38:05.672 DEBUG:boto:Params: {} 2022-09-09T22:38:05.673 DEBUG:boto:Token: None 2022-09-09T22:38:05.673 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.674 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:X2wKn37WBNBgyQSU2LuqpXA/hZE= 2022-09-09T22:38:05.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:X2wKn37WBNBgyQSU2LuqpXA/hZE=', 'Content-Length': '0'} 2022-09-09T22:38:05.678 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000df53948a532f4519-00631bc04d-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.679 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:38:05.679 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:38:05.680 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.680 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.681 DEBUG:boto:Method: HEAD 2022-09-09T22:38:05.681 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.682 DEBUG:boto:Data: 2022-09-09T22:38:05.682 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.683 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:05.683 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.684 DEBUG:boto:Params: {} 2022-09-09T22:38:05.684 DEBUG:boto:Token: None 2022-09-09T22:38:05.685 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.685 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:X2wKn37WBNBgyQSU2LuqpXA/hZE= 2022-09-09T22:38:05.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:X2wKn37WBNBgyQSU2LuqpXA/hZE=', 'Content-Length': '0'} 2022-09-09T22:38:05.690 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007324be0015daaa59-00631bc04d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:38:05.691 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:38:05.691 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:38:05.692 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.692 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.693 DEBUG:boto:Method: GET 2022-09-09T22:38:05.693 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.694 DEBUG:boto:Data: 2022-09-09T22:38:05.695 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.695 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:05.696 DEBUG:boto:Port: 8001 2022-09-09T22:38:05.696 DEBUG:boto:Params: {} 2022-09-09T22:38:05.697 DEBUG:boto:Token: None 2022-09-09T22:38:05.698 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.698 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pLxIKObeoU+jbyAm4D3av9DlRh4= 2022-09-09T22:38:05.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pLxIKObeoU+jbyAm4D3av9DlRh4=', 'Content-Length': '0'} 2022-09-09T22:38:05.702 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007e4f068df33458d9-00631bc04d-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.703 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:38:05.703 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:38:05.704 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.704 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.705 DEBUG:boto:Method: GET 2022-09-09T22:38:05.705 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.706 DEBUG:boto:Data: 2022-09-09T22:38:05.707 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.707 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:05.708 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.708 DEBUG:boto:Params: {} 2022-09-09T22:38:05.709 DEBUG:boto:Token: None 2022-09-09T22:38:05.709 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.710 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pLxIKObeoU+jbyAm4D3av9DlRh4= 2022-09-09T22:38:05.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pLxIKObeoU+jbyAm4D3av9DlRh4=', 'Content-Length': '0'} 2022-09-09T22:38:05.714 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000813b6463bf8c3f5c-00631bc04d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.715 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:05.715 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:05.716 DEBUG:boto:path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:38:05.716 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:38:05.717 DEBUG:boto:Method: GET 2022-09-09T22:38:05.717 DEBUG:boto:Path: /jkcgbj-16/?prefix=dummy 2022-09-09T22:38:05.718 DEBUG:boto:Data: 2022-09-09T22:38:05.718 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.719 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:05.719 DEBUG:boto:Port: 8001 2022-09-09T22:38:05.720 DEBUG:boto:Params: {} 2022-09-09T22:38:05.721 DEBUG:boto:Token: None 2022-09-09T22:38:05.721 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/ 2022-09-09T22:38:05.722 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1UGCSIQDgmldL2bcf832iLWJLfM= 2022-09-09T22:38:05.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1UGCSIQDgmldL2bcf832iLWJLfM=', 'Content-Length': '0'} 2022-09-09T22:38:05.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bcd788e0e17ddfd-00631bc04d-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.774 DEBUG:boto:b'jkcgbj-16dummy1000falsedummy2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:05.775 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:05.776 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:05.776 DEBUG:boto:path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:38:05.777 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:38:05.778 DEBUG:boto:Method: GET 2022-09-09T22:38:05.778 DEBUG:boto:Path: /jkcgbj-16/?prefix=dummy 2022-09-09T22:38:05.779 DEBUG:boto:Data: 2022-09-09T22:38:05.779 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.780 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:05.780 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.781 DEBUG:boto:Params: {} 2022-09-09T22:38:05.782 DEBUG:boto:Token: None 2022-09-09T22:38:05.782 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/ 2022-09-09T22:38:05.783 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1UGCSIQDgmldL2bcf832iLWJLfM= 2022-09-09T22:38:05.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1UGCSIQDgmldL2bcf832iLWJLfM=', 'Content-Length': '0'} 2022-09-09T22:38:05.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0fe0dcaaa82655d-00631bc04d-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.791 DEBUG:boto:b'jkcgbj-16dummy1000falsedummy2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:05.792 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:38:05.792 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:38:05.793 DEBUG:boto:Method: HEAD 2022-09-09T22:38:05.793 DEBUG:boto:Path: /jkcgbj-16/dummy 2022-09-09T22:38:05.794 DEBUG:boto:Data: 2022-09-09T22:38:05.794 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.795 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:05.795 DEBUG:boto:Port: 8001 2022-09-09T22:38:05.796 DEBUG:boto:Params: {} 2022-09-09T22:38:05.797 DEBUG:boto:Token: None 2022-09-09T22:38:05.797 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/dummy 2022-09-09T22:38:05.798 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:r1usOLAW9jnaiJSLEnUXisnoa3k= 2022-09-09T22:38:05.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:r1usOLAW9jnaiJSLEnUXisnoa3k=', 'Content-Length': '0'} 2022-09-09T22:38:05.803 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fbf6efef322cbe51-00631bc04d-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.804 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:38:05.804 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:38:05.805 DEBUG:boto:Method: HEAD 2022-09-09T22:38:05.805 DEBUG:boto:Path: /jkcgbj-16/dummy 2022-09-09T22:38:05.806 DEBUG:boto:Data: 2022-09-09T22:38:05.806 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.807 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:05.807 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.808 DEBUG:boto:Params: {} 2022-09-09T22:38:05.809 DEBUG:boto:Token: None 2022-09-09T22:38:05.809 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/dummy 2022-09-09T22:38:05.810 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:r1usOLAW9jnaiJSLEnUXisnoa3k= 2022-09-09T22:38:05.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:r1usOLAW9jnaiJSLEnUXisnoa3k=', 'Content-Length': '0'} 2022-09-09T22:38:05.815 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007473e7c4791c4bfe-00631bc04d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:38:05.816 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:38:05.817 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:38:05.817 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.818 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.819 DEBUG:boto:Method: GET 2022-09-09T22:38:05.820 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.821 DEBUG:boto:Data: 2022-09-09T22:38:05.822 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.823 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:05.824 DEBUG:boto:Port: 8001 2022-09-09T22:38:05.825 DEBUG:boto:Params: {} 2022-09-09T22:38:05.826 DEBUG:boto:Token: None 2022-09-09T22:38:05.827 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.828 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pLxIKObeoU+jbyAm4D3av9DlRh4= 2022-09-09T22:38:05.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pLxIKObeoU+jbyAm4D3av9DlRh4=', 'Content-Length': '0'} 2022-09-09T22:38:05.833 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000f93cd1222a9ced4-00631bc04d-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.834 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:38:05.834 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:38:05.835 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.835 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.836 DEBUG:boto:Method: GET 2022-09-09T22:38:05.836 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.837 DEBUG:boto:Data: 2022-09-09T22:38:05.837 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.838 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:05.839 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.839 DEBUG:boto:Params: {} 2022-09-09T22:38:05.840 DEBUG:boto:Token: None 2022-09-09T22:38:05.841 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:05.841 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pLxIKObeoU+jbyAm4D3av9DlRh4= 2022-09-09T22:38:05.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pLxIKObeoU+jbyAm4D3av9DlRh4=', 'Content-Length': '0'} 2022-09-09T22:38:05.845 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000b48c2acfea71a3b-00631bc04d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.846 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:05.847 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:05.847 DEBUG:boto:path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:38:05.848 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:38:05.848 DEBUG:boto:Method: GET 2022-09-09T22:38:05.849 DEBUG:boto:Path: /jkcgbj-16/?prefix=dummy 2022-09-09T22:38:05.849 DEBUG:boto:Data: 2022-09-09T22:38:05.850 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.850 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:05.851 DEBUG:boto:Port: 8001 2022-09-09T22:38:05.851 DEBUG:boto:Params: {} 2022-09-09T22:38:05.852 DEBUG:boto:Token: None 2022-09-09T22:38:05.853 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/ 2022-09-09T22:38:05.854 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1UGCSIQDgmldL2bcf832iLWJLfM= 2022-09-09T22:38:05.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1UGCSIQDgmldL2bcf832iLWJLfM=', 'Content-Length': '0'} 2022-09-09T22:38:05.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d93b1410067dc8d-00631bc04d-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.862 DEBUG:boto:b'jkcgbj-16dummy1000falsedummy2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:05.863 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:05.863 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:05.864 DEBUG:boto:path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:38:05.864 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:38:05.865 DEBUG:boto:Method: GET 2022-09-09T22:38:05.865 DEBUG:boto:Path: /jkcgbj-16/?prefix=dummy 2022-09-09T22:38:05.866 DEBUG:boto:Data: 2022-09-09T22:38:05.867 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.867 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:05.868 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.868 DEBUG:boto:Params: {} 2022-09-09T22:38:05.869 DEBUG:boto:Token: None 2022-09-09T22:38:05.870 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/ 2022-09-09T22:38:05.870 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1UGCSIQDgmldL2bcf832iLWJLfM= 2022-09-09T22:38:05.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1UGCSIQDgmldL2bcf832iLWJLfM=', 'Content-Length': '0'} 2022-09-09T22:38:05.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095f373e56cb05631-00631bc04d-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.879 DEBUG:boto:b'jkcgbj-16dummy1000falsedummy2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:05.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:38:05.881 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:05.881 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:05.882 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:05.882 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:05.883 DEBUG:boto:Method: GET 2022-09-09T22:38:05.884 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:05.884 DEBUG:boto:Data: 2022-09-09T22:38:05.885 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.885 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:05.886 DEBUG:boto:Port: 8001 2022-09-09T22:38:05.886 DEBUG:boto:Params: {} 2022-09-09T22:38:05.887 DEBUG:boto:Token: None 2022-09-09T22:38:05.887 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:05.888 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ZpNRGH/vTK+iagP3lZqo9ngdsiU= 2022-09-09T22:38:05.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ZpNRGH/vTK+iagP3lZqo9ngdsiU=', 'Content-Length': '0'} 2022-09-09T22:38:05.892 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a46bcbabb06457ea-00631bc04d-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.934 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:05.935 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:05.936 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:05.937 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:05.939 DEBUG:boto:Method: GET 2022-09-09T22:38:05.940 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:05.941 DEBUG:boto:Data: 2022-09-09T22:38:05.942 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.943 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:05.944 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.945 DEBUG:boto:Params: {} 2022-09-09T22:38:05.946 DEBUG:boto:Token: None 2022-09-09T22:38:05.948 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:05.949 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ZpNRGH/vTK+iagP3lZqo9ngdsiU= 2022-09-09T22:38:05.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ZpNRGH/vTK+iagP3lZqo9ngdsiU=', 'Content-Length': '0'} 2022-09-09T22:38:05.954 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000595d08ac7056ee9-00631bc04d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.956 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:05.957 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:05.958 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:05.959 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:05.960 DEBUG:boto:Method: HEAD 2022-09-09T22:38:05.961 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:05.962 DEBUG:boto:Data: 2022-09-09T22:38:05.962 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.963 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:05.964 DEBUG:boto:Port: 8001 2022-09-09T22:38:05.965 DEBUG:boto:Params: {} 2022-09-09T22:38:05.966 DEBUG:boto:Token: None 2022-09-09T22:38:05.967 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:05.968 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:uddPwGe1e65+pkif8tCMQHQm9tA= 2022-09-09T22:38:05.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:uddPwGe1e65+pkif8tCMQHQm9tA=', 'Content-Length': '0'} 2022-09-09T22:38:05.973 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064fe19bab89f8f3e-00631bc04d-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.974 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:05.974 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:05.975 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:05.975 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:05.976 DEBUG:boto:Method: HEAD 2022-09-09T22:38:05.976 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:05.977 DEBUG:boto:Data: 2022-09-09T22:38:05.977 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.978 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:05.978 DEBUG:boto:Port: 8000 2022-09-09T22:38:05.979 DEBUG:boto:Params: {} 2022-09-09T22:38:05.979 DEBUG:boto:Token: None 2022-09-09T22:38:05.980 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:05.981 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:uddPwGe1e65+pkif8tCMQHQm9tA= 2022-09-09T22:38:05.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:uddPwGe1e65+pkif8tCMQHQm9tA=', 'Content-Length': '0'} 2022-09-09T22:38:05.984 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000456af673cd09a5af-00631bc04d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:38:05.985 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:05.986 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:05.987 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:05.987 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:05.988 DEBUG:boto:Method: GET 2022-09-09T22:38:05.988 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:05.989 DEBUG:boto:Data: 2022-09-09T22:38:05.989 DEBUG:boto:Headers: {} 2022-09-09T22:38:05.990 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:05.990 DEBUG:boto:Port: 8001 2022-09-09T22:38:05.991 DEBUG:boto:Params: {} 2022-09-09T22:38:05.991 DEBUG:boto:Token: None 2022-09-09T22:38:05.992 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:05 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:05.993 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ZpNRGH/vTK+iagP3lZqo9ngdsiU= 2022-09-09T22:38:05.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ZpNRGH/vTK+iagP3lZqo9ngdsiU=', 'Content-Length': '0'} 2022-09-09T22:38:05.996 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d1da136140f7e66d-00631bc04d-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:05.997 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:05.998 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:05.998 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:05.999 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:05.999 DEBUG:boto:Method: GET 2022-09-09T22:38:06.000 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:06.000 DEBUG:boto:Data: 2022-09-09T22:38:06.001 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.001 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:06.002 DEBUG:boto:Port: 8000 2022-09-09T22:38:06.002 DEBUG:boto:Params: {} 2022-09-09T22:38:06.003 DEBUG:boto:Token: None 2022-09-09T22:38:06.003 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:06.004 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:QBnArGGSx7gnQ9MZ5AU8t60rhO4= 2022-09-09T22:38:06.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:QBnArGGSx7gnQ9MZ5AU8t60rhO4=', 'Content-Length': '0'} 2022-09-09T22:38:06.008 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006400d149535688eb-00631bc04e-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.050 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:06.051 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:06.052 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:06.053 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:06.054 DEBUG:boto:Method: GET 2022-09-09T22:38:06.055 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:06.056 DEBUG:boto:Data: 2022-09-09T22:38:06.057 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.058 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:06.059 DEBUG:boto:Port: 8001 2022-09-09T22:38:06.060 DEBUG:boto:Params: {} 2022-09-09T22:38:06.060 DEBUG:boto:Token: None 2022-09-09T22:38:06.061 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/ 2022-09-09T22:38:06.062 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JA6Ij0lw2WdAvLLLh0jRGEh5b+c= 2022-09-09T22:38:06.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JA6Ij0lw2WdAvLLLh0jRGEh5b+c=', 'Content-Length': '0'} 2022-09-09T22:38:06.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c824610c78300bf-00631bc04e-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.069 DEBUG:boto:b'jkcgbj-16obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:06.070 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:06.070 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:06.071 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:06.071 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:06.072 DEBUG:boto:Method: GET 2022-09-09T22:38:06.072 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:06.073 DEBUG:boto:Data: 2022-09-09T22:38:06.073 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.074 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:06.074 DEBUG:boto:Port: 8000 2022-09-09T22:38:06.075 DEBUG:boto:Params: {} 2022-09-09T22:38:06.075 DEBUG:boto:Token: None 2022-09-09T22:38:06.076 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/ 2022-09-09T22:38:06.077 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JA6Ij0lw2WdAvLLLh0jRGEh5b+c= 2022-09-09T22:38:06.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JA6Ij0lw2WdAvLLLh0jRGEh5b+c=', 'Content-Length': '0'} 2022-09-09T22:38:06.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2ee8f624c579eb3-00631bc04e-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.084 DEBUG:boto:b'jkcgbj-16obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:06.085 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:06.085 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:06.086 DEBUG:boto:Method: HEAD 2022-09-09T22:38:06.086 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1 2022-09-09T22:38:06.087 DEBUG:boto:Data: 2022-09-09T22:38:06.087 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.088 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:06.088 DEBUG:boto:Port: 8001 2022-09-09T22:38:06.089 DEBUG:boto:Params: {} 2022-09-09T22:38:06.089 DEBUG:boto:Token: None 2022-09-09T22:38:06.090 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/obj-test-zone1 2022-09-09T22:38:06.091 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ND0UEWQIbNQSFs8A6D65Q4OiuSE= 2022-09-09T22:38:06.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ND0UEWQIbNQSFs8A6D65Q4OiuSE=', 'Content-Length': '0'} 2022-09-09T22:38:06.098 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000dca5da18a6904fa-00631bc04e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.099 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:06.099 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:06.100 DEBUG:boto:Method: HEAD 2022-09-09T22:38:06.100 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1 2022-09-09T22:38:06.101 DEBUG:boto:Data: 2022-09-09T22:38:06.101 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.102 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:06.102 DEBUG:boto:Port: 8000 2022-09-09T22:38:06.102 DEBUG:boto:Params: {} 2022-09-09T22:38:06.103 DEBUG:boto:Token: None 2022-09-09T22:38:06.104 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/obj-test-zone1 2022-09-09T22:38:06.104 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ND0UEWQIbNQSFs8A6D65Q4OiuSE= 2022-09-09T22:38:06.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ND0UEWQIbNQSFs8A6D65Q4OiuSE=', 'Content-Length': '0'} 2022-09-09T22:38:06.109 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003937d3411b3e536c-00631bc04e-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:38:06.110 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:06.111 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:06.111 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:06.112 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:06.112 DEBUG:boto:Method: GET 2022-09-09T22:38:06.113 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:06.113 DEBUG:boto:Data: 2022-09-09T22:38:06.114 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.114 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:06.115 DEBUG:boto:Port: 8001 2022-09-09T22:38:06.115 DEBUG:boto:Params: {} 2022-09-09T22:38:06.116 DEBUG:boto:Token: None 2022-09-09T22:38:06.116 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:06.117 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:QBnArGGSx7gnQ9MZ5AU8t60rhO4= 2022-09-09T22:38:06.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:QBnArGGSx7gnQ9MZ5AU8t60rhO4=', 'Content-Length': '0'} 2022-09-09T22:38:06.121 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d4adb8eadfdb1f92-00631bc04e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.162 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:06.163 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:06.163 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:06.164 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:06.164 DEBUG:boto:Method: GET 2022-09-09T22:38:06.165 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:06.165 DEBUG:boto:Data: 2022-09-09T22:38:06.166 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.167 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:06.167 DEBUG:boto:Port: 8000 2022-09-09T22:38:06.168 DEBUG:boto:Params: {} 2022-09-09T22:38:06.168 DEBUG:boto:Token: None 2022-09-09T22:38:06.169 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:06.170 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:QBnArGGSx7gnQ9MZ5AU8t60rhO4= 2022-09-09T22:38:06.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:QBnArGGSx7gnQ9MZ5AU8t60rhO4=', 'Content-Length': '0'} 2022-09-09T22:38:06.174 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cf392306237396c7-00631bc04e-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.175 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:06.175 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:06.176 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:06.177 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:06.178 DEBUG:boto:Method: GET 2022-09-09T22:38:06.178 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:06.179 DEBUG:boto:Data: 2022-09-09T22:38:06.179 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.180 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:06.180 DEBUG:boto:Port: 8001 2022-09-09T22:38:06.181 DEBUG:boto:Params: {} 2022-09-09T22:38:06.182 DEBUG:boto:Token: None 2022-09-09T22:38:06.182 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/ 2022-09-09T22:38:06.183 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JA6Ij0lw2WdAvLLLh0jRGEh5b+c= 2022-09-09T22:38:06.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JA6Ij0lw2WdAvLLLh0jRGEh5b+c=', 'Content-Length': '0'} 2022-09-09T22:38:06.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2ad5430f832c157-00631bc04e-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.190 DEBUG:boto:b'jkcgbj-16obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:06.192 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:06.192 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:06.193 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:06.194 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:06.194 DEBUG:boto:Method: GET 2022-09-09T22:38:06.195 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:06.195 DEBUG:boto:Data: 2022-09-09T22:38:06.196 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.197 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:06.198 DEBUG:boto:Port: 8000 2022-09-09T22:38:06.199 DEBUG:boto:Params: {} 2022-09-09T22:38:06.201 DEBUG:boto:Token: None 2022-09-09T22:38:06.202 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/ 2022-09-09T22:38:06.203 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JA6Ij0lw2WdAvLLLh0jRGEh5b+c= 2022-09-09T22:38:06.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JA6Ij0lw2WdAvLLLh0jRGEh5b+c=', 'Content-Length': '0'} 2022-09-09T22:38:06.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001365794b886707de-00631bc04e-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.254 DEBUG:boto:b'jkcgbj-16obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:06.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:38:06.256 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:06.256 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:06.257 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.258 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.259 DEBUG:boto:Method: GET 2022-09-09T22:38:06.260 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.261 DEBUG:boto:Data: 2022-09-09T22:38:06.262 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.263 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:06.263 DEBUG:boto:Port: 8001 2022-09-09T22:38:06.264 DEBUG:boto:Params: {} 2022-09-09T22:38:06.264 DEBUG:boto:Token: None 2022-09-09T22:38:06.265 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.265 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:faqR9rB9qscOWlU5OrDjL0V3g6o= 2022-09-09T22:38:06.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:faqR9rB9qscOWlU5OrDjL0V3g6o=', 'Content-Length': '0'} 2022-09-09T22:38:06.270 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004096f1f369729216-00631bc04e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.271 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:06.272 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:06.274 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.275 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.276 DEBUG:boto:Method: GET 2022-09-09T22:38:06.277 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.278 DEBUG:boto:Data: 2022-09-09T22:38:06.279 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.280 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:06.281 DEBUG:boto:Port: 8000 2022-09-09T22:38:06.282 DEBUG:boto:Params: {} 2022-09-09T22:38:06.283 DEBUG:boto:Token: None 2022-09-09T22:38:06.284 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.285 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:faqR9rB9qscOWlU5OrDjL0V3g6o= 2022-09-09T22:38:06.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:faqR9rB9qscOWlU5OrDjL0V3g6o=', 'Content-Length': '0'} 2022-09-09T22:38:06.290 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c287b839ae840525-00631bc04e-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.291 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:06.291 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:06.292 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.292 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.293 DEBUG:boto:Method: HEAD 2022-09-09T22:38:06.294 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.294 DEBUG:boto:Data: 2022-09-09T22:38:06.295 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.296 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:06.297 DEBUG:boto:Port: 8001 2022-09-09T22:38:06.297 DEBUG:boto:Params: {} 2022-09-09T22:38:06.298 DEBUG:boto:Token: None 2022-09-09T22:38:06.298 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.299 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UvbdWzV6pNa75PP3Vl5tSKcZv5s= 2022-09-09T22:38:06.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UvbdWzV6pNa75PP3Vl5tSKcZv5s=', 'Content-Length': '0'} 2022-09-09T22:38:06.303 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e7fc2cc319f62bc-00631bc04e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.303 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:06.304 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:06.304 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.305 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.306 DEBUG:boto:Method: HEAD 2022-09-09T22:38:06.307 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.308 DEBUG:boto:Data: 2022-09-09T22:38:06.308 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.309 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:06.309 DEBUG:boto:Port: 8000 2022-09-09T22:38:06.310 DEBUG:boto:Params: {} 2022-09-09T22:38:06.310 DEBUG:boto:Token: None 2022-09-09T22:38:06.311 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.312 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UvbdWzV6pNa75PP3Vl5tSKcZv5s= 2022-09-09T22:38:06.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UvbdWzV6pNa75PP3Vl5tSKcZv5s=', 'Content-Length': '0'} 2022-09-09T22:38:06.315 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000936773968c0fcfbe-00631bc04e-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:38:06.317 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:06.317 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:06.318 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.319 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.320 DEBUG:boto:Method: GET 2022-09-09T22:38:06.321 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.322 DEBUG:boto:Data: 2022-09-09T22:38:06.323 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.324 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:06.325 DEBUG:boto:Port: 8001 2022-09-09T22:38:06.326 DEBUG:boto:Params: {} 2022-09-09T22:38:06.327 DEBUG:boto:Token: None 2022-09-09T22:38:06.328 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.330 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:faqR9rB9qscOWlU5OrDjL0V3g6o= 2022-09-09T22:38:06.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:faqR9rB9qscOWlU5OrDjL0V3g6o=', 'Content-Length': '0'} 2022-09-09T22:38:06.334 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f35eaf77cd2e34b-00631bc04e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.336 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:06.337 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:06.337 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.338 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.339 DEBUG:boto:Method: GET 2022-09-09T22:38:06.340 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.341 DEBUG:boto:Data: 2022-09-09T22:38:06.342 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.343 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:06.344 DEBUG:boto:Port: 8000 2022-09-09T22:38:06.345 DEBUG:boto:Params: {} 2022-09-09T22:38:06.346 DEBUG:boto:Token: None 2022-09-09T22:38:06.347 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.348 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:faqR9rB9qscOWlU5OrDjL0V3g6o= 2022-09-09T22:38:06.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:faqR9rB9qscOWlU5OrDjL0V3g6o=', 'Content-Length': '0'} 2022-09-09T22:38:06.353 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b7f482894ba41e5-00631bc04e-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.394 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:06.395 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:06.396 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:06.397 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:06.398 DEBUG:boto:Method: GET 2022-09-09T22:38:06.399 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:06.399 DEBUG:boto:Data: 2022-09-09T22:38:06.400 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.400 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:06.401 DEBUG:boto:Port: 8001 2022-09-09T22:38:06.401 DEBUG:boto:Params: {} 2022-09-09T22:38:06.402 DEBUG:boto:Token: None 2022-09-09T22:38:06.403 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/ 2022-09-09T22:38:06.403 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JA6Ij0lw2WdAvLLLh0jRGEh5b+c= 2022-09-09T22:38:06.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JA6Ij0lw2WdAvLLLh0jRGEh5b+c=', 'Content-Length': '0'} 2022-09-09T22:38:06.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe3287d5d6219cbc-00631bc04e-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.411 DEBUG:boto:b'jkcgbj-16obj-test-zone21000falseobj-test-zone22022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:06.411 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:06.412 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:06.413 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:06.413 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:06.414 DEBUG:boto:Method: GET 2022-09-09T22:38:06.414 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:06.415 DEBUG:boto:Data: 2022-09-09T22:38:06.415 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.416 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:06.416 DEBUG:boto:Port: 8000 2022-09-09T22:38:06.417 DEBUG:boto:Params: {} 2022-09-09T22:38:06.418 DEBUG:boto:Token: None 2022-09-09T22:38:06.418 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/ 2022-09-09T22:38:06.419 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JA6Ij0lw2WdAvLLLh0jRGEh5b+c= 2022-09-09T22:38:06.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JA6Ij0lw2WdAvLLLh0jRGEh5b+c=', 'Content-Length': '0'} 2022-09-09T22:38:06.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046da8156cb59f5c6-00631bc04e-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.428 DEBUG:boto:b'jkcgbj-16obj-test-zone21000falseobj-test-zone22022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:06.430 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:06.431 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:06.432 DEBUG:boto:Method: HEAD 2022-09-09T22:38:06.433 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2 2022-09-09T22:38:06.433 DEBUG:boto:Data: 2022-09-09T22:38:06.434 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.434 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:06.435 DEBUG:boto:Port: 8001 2022-09-09T22:38:06.435 DEBUG:boto:Params: {} 2022-09-09T22:38:06.436 DEBUG:boto:Token: None 2022-09-09T22:38:06.437 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/obj-test-zone2 2022-09-09T22:38:06.438 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wCxzLbrwDsql1hmSXsPIrUaGcTM= 2022-09-09T22:38:06.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wCxzLbrwDsql1hmSXsPIrUaGcTM=', 'Content-Length': '0'} 2022-09-09T22:38:06.444 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d4858b85c505e1b-00631bc04e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.445 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:06.446 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:06.447 DEBUG:boto:Method: HEAD 2022-09-09T22:38:06.448 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2 2022-09-09T22:38:06.449 DEBUG:boto:Data: 2022-09-09T22:38:06.450 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.450 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:06.451 DEBUG:boto:Port: 8000 2022-09-09T22:38:06.452 DEBUG:boto:Params: {} 2022-09-09T22:38:06.453 DEBUG:boto:Token: None 2022-09-09T22:38:06.454 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/obj-test-zone2 2022-09-09T22:38:06.455 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wCxzLbrwDsql1hmSXsPIrUaGcTM= 2022-09-09T22:38:06.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wCxzLbrwDsql1hmSXsPIrUaGcTM=', 'Content-Length': '0'} 2022-09-09T22:38:06.461 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dce9fa12169d33f0-00631bc04e-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:38:06.463 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:06.464 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:06.465 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.466 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.467 DEBUG:boto:Method: GET 2022-09-09T22:38:06.468 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.469 DEBUG:boto:Data: 2022-09-09T22:38:06.470 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.471 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:06.472 DEBUG:boto:Port: 8001 2022-09-09T22:38:06.473 DEBUG:boto:Params: {} 2022-09-09T22:38:06.474 DEBUG:boto:Token: None 2022-09-09T22:38:06.475 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.476 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:faqR9rB9qscOWlU5OrDjL0V3g6o= 2022-09-09T22:38:06.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:faqR9rB9qscOWlU5OrDjL0V3g6o=', 'Content-Length': '0'} 2022-09-09T22:38:06.481 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce7815fb21f92fd3-00631bc04e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.522 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:06.523 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:06.524 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.524 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.526 DEBUG:boto:Method: GET 2022-09-09T22:38:06.526 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.527 DEBUG:boto:Data: 2022-09-09T22:38:06.527 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.528 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:06.528 DEBUG:boto:Port: 8000 2022-09-09T22:38:06.529 DEBUG:boto:Params: {} 2022-09-09T22:38:06.530 DEBUG:boto:Token: None 2022-09-09T22:38:06.530 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:06.531 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:faqR9rB9qscOWlU5OrDjL0V3g6o= 2022-09-09T22:38:06.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:faqR9rB9qscOWlU5OrDjL0V3g6o=', 'Content-Length': '0'} 2022-09-09T22:38:06.536 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b445299c004ba131-00631bc04e-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.537 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:06.539 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:06.539 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:06.540 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:06.541 DEBUG:boto:Method: GET 2022-09-09T22:38:06.542 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:06.543 DEBUG:boto:Data: 2022-09-09T22:38:06.544 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.545 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:06.546 DEBUG:boto:Port: 8001 2022-09-09T22:38:06.547 DEBUG:boto:Params: {} 2022-09-09T22:38:06.548 DEBUG:boto:Token: None 2022-09-09T22:38:06.549 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/ 2022-09-09T22:38:06.550 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JA6Ij0lw2WdAvLLLh0jRGEh5b+c= 2022-09-09T22:38:06.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JA6Ij0lw2WdAvLLLh0jRGEh5b+c=', 'Content-Length': '0'} 2022-09-09T22:38:06.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006111862306b65aed-00631bc04e-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.559 DEBUG:boto:b'jkcgbj-16obj-test-zone21000falseobj-test-zone22022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:06.560 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:06.561 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:06.561 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:06.562 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:06.564 DEBUG:boto:Method: GET 2022-09-09T22:38:06.565 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:06.566 DEBUG:boto:Data: 2022-09-09T22:38:06.567 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.568 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:06.568 DEBUG:boto:Port: 8000 2022-09-09T22:38:06.569 DEBUG:boto:Params: {} 2022-09-09T22:38:06.569 DEBUG:boto:Token: None 2022-09-09T22:38:06.570 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/ 2022-09-09T22:38:06.571 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JA6Ij0lw2WdAvLLLh0jRGEh5b+c= 2022-09-09T22:38:06.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JA6Ij0lw2WdAvLLLh0jRGEh5b+c=', 'Content-Length': '0'} 2022-09-09T22:38:06.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048f4bb92d2bb8146-00631bc04e-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.618 DEBUG:boto:b'jkcgbj-16obj-test-zone21000falseobj-test-zone22022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:06.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:38:06.621 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:06.621 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:06.622 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.623 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.624 DEBUG:boto:Method: GET 2022-09-09T22:38:06.624 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.625 DEBUG:boto:Data: 2022-09-09T22:38:06.625 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.626 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:06.627 DEBUG:boto:Port: 8001 2022-09-09T22:38:06.627 DEBUG:boto:Params: {} 2022-09-09T22:38:06.628 DEBUG:boto:Token: None 2022-09-09T22:38:06.629 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.630 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:SsySTU8tlc7/CcTcxFnIXNR7eoA= 2022-09-09T22:38:06.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:SsySTU8tlc7/CcTcxFnIXNR7eoA=', 'Content-Length': '0'} 2022-09-09T22:38:06.634 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5f5231549d390ab-00631bc04e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.635 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:06.636 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:06.636 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.637 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.637 DEBUG:boto:Method: GET 2022-09-09T22:38:06.638 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.639 DEBUG:boto:Data: 2022-09-09T22:38:06.639 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.640 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:06.640 DEBUG:boto:Port: 8000 2022-09-09T22:38:06.641 DEBUG:boto:Params: {} 2022-09-09T22:38:06.641 DEBUG:boto:Token: None 2022-09-09T22:38:06.642 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.642 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:SsySTU8tlc7/CcTcxFnIXNR7eoA= 2022-09-09T22:38:06.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:SsySTU8tlc7/CcTcxFnIXNR7eoA=', 'Content-Length': '0'} 2022-09-09T22:38:06.646 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d82374789edcc285-00631bc04e-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.647 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:06.648 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:06.648 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.649 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.649 DEBUG:boto:Method: HEAD 2022-09-09T22:38:06.650 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.650 DEBUG:boto:Data: 2022-09-09T22:38:06.651 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.651 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:06.652 DEBUG:boto:Port: 8001 2022-09-09T22:38:06.652 DEBUG:boto:Params: {} 2022-09-09T22:38:06.653 DEBUG:boto:Token: None 2022-09-09T22:38:06.653 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.654 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:zoltdt0V7tfPVtWH89pDxnNDktQ= 2022-09-09T22:38:06.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:zoltdt0V7tfPVtWH89pDxnNDktQ=', 'Content-Length': '0'} 2022-09-09T22:38:06.658 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db179e09eab63e1c-00631bc04e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.659 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:06.659 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:06.660 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.660 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.661 DEBUG:boto:Method: HEAD 2022-09-09T22:38:06.662 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.662 DEBUG:boto:Data: 2022-09-09T22:38:06.663 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.663 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:06.664 DEBUG:boto:Port: 8000 2022-09-09T22:38:06.664 DEBUG:boto:Params: {} 2022-09-09T22:38:06.665 DEBUG:boto:Token: None 2022-09-09T22:38:06.666 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.666 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:zoltdt0V7tfPVtWH89pDxnNDktQ= 2022-09-09T22:38:06.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:zoltdt0V7tfPVtWH89pDxnNDktQ=', 'Content-Length': '0'} 2022-09-09T22:38:06.670 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b0fa49b5bc13f1af-00631bc04e-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:38:06.673 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:06.674 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:06.675 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.676 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.677 DEBUG:boto:Method: GET 2022-09-09T22:38:06.678 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.679 DEBUG:boto:Data: 2022-09-09T22:38:06.680 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.680 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:06.681 DEBUG:boto:Port: 8001 2022-09-09T22:38:06.682 DEBUG:boto:Params: {} 2022-09-09T22:38:06.684 DEBUG:boto:Token: None 2022-09-09T22:38:06.685 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.686 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:SsySTU8tlc7/CcTcxFnIXNR7eoA= 2022-09-09T22:38:06.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:SsySTU8tlc7/CcTcxFnIXNR7eoA=', 'Content-Length': '0'} 2022-09-09T22:38:06.690 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005df6b877e9b821aa-00631bc04e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.691 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:06.692 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:06.693 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.693 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.694 DEBUG:boto:Method: GET 2022-09-09T22:38:06.695 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.695 DEBUG:boto:Data: 2022-09-09T22:38:06.696 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.697 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:06.697 DEBUG:boto:Port: 8000 2022-09-09T22:38:06.698 DEBUG:boto:Params: {} 2022-09-09T22:38:06.699 DEBUG:boto:Token: None 2022-09-09T22:38:06.699 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.700 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:SsySTU8tlc7/CcTcxFnIXNR7eoA= 2022-09-09T22:38:06.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:SsySTU8tlc7/CcTcxFnIXNR7eoA=', 'Content-Length': '0'} 2022-09-09T22:38:06.704 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b61b610c2695f973-00631bc04e-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.746 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:06.747 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:06.749 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:06.750 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:06.751 DEBUG:boto:Method: GET 2022-09-09T22:38:06.752 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:06.753 DEBUG:boto:Data: 2022-09-09T22:38:06.754 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.755 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:06.756 DEBUG:boto:Port: 8001 2022-09-09T22:38:06.757 DEBUG:boto:Params: {} 2022-09-09T22:38:06.758 DEBUG:boto:Token: None 2022-09-09T22:38:06.759 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/ 2022-09-09T22:38:06.760 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JA6Ij0lw2WdAvLLLh0jRGEh5b+c= 2022-09-09T22:38:06.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JA6Ij0lw2WdAvLLLh0jRGEh5b+c=', 'Content-Length': '0'} 2022-09-09T22:38:06.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000384ae89f9ab199a4-00631bc04e-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.767 DEBUG:boto:b'jkcgbj-16obj-test-zone31000falseobj-test-zone32022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:06.768 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:06.769 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:06.770 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:06.770 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:06.771 DEBUG:boto:Method: GET 2022-09-09T22:38:06.771 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:06.772 DEBUG:boto:Data: 2022-09-09T22:38:06.772 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.773 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:06.774 DEBUG:boto:Port: 8000 2022-09-09T22:38:06.774 DEBUG:boto:Params: {} 2022-09-09T22:38:06.775 DEBUG:boto:Token: None 2022-09-09T22:38:06.776 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/ 2022-09-09T22:38:06.777 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JA6Ij0lw2WdAvLLLh0jRGEh5b+c= 2022-09-09T22:38:06.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JA6Ij0lw2WdAvLLLh0jRGEh5b+c=', 'Content-Length': '0'} 2022-09-09T22:38:06.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006809dc0e1e767901-00631bc04e-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.785 DEBUG:boto:b'jkcgbj-16obj-test-zone31000falseobj-test-zone32022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:06.786 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:06.786 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:06.787 DEBUG:boto:Method: HEAD 2022-09-09T22:38:06.787 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3 2022-09-09T22:38:06.788 DEBUG:boto:Data: 2022-09-09T22:38:06.788 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.789 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:06.789 DEBUG:boto:Port: 8001 2022-09-09T22:38:06.790 DEBUG:boto:Params: {} 2022-09-09T22:38:06.790 DEBUG:boto:Token: None 2022-09-09T22:38:06.791 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/obj-test-zone3 2022-09-09T22:38:06.792 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dTwZ4qwAFBC39q/ncMNgTcgJ5Bk= 2022-09-09T22:38:06.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dTwZ4qwAFBC39q/ncMNgTcgJ5Bk=', 'Content-Length': '0'} 2022-09-09T22:38:06.797 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082722a16208bbdb4-00631bc04e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.798 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:06.799 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:06.799 DEBUG:boto:Method: HEAD 2022-09-09T22:38:06.800 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3 2022-09-09T22:38:06.801 DEBUG:boto:Data: 2022-09-09T22:38:06.802 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.803 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:06.804 DEBUG:boto:Port: 8000 2022-09-09T22:38:06.804 DEBUG:boto:Params: {} 2022-09-09T22:38:06.805 DEBUG:boto:Token: None 2022-09-09T22:38:06.805 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/obj-test-zone3 2022-09-09T22:38:06.806 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dTwZ4qwAFBC39q/ncMNgTcgJ5Bk= 2022-09-09T22:38:06.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dTwZ4qwAFBC39q/ncMNgTcgJ5Bk=', 'Content-Length': '0'} 2022-09-09T22:38:06.811 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b420f380ee361fa-00631bc04e-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:38:06.812 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:06.813 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:06.813 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.814 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.814 DEBUG:boto:Method: GET 2022-09-09T22:38:06.815 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.815 DEBUG:boto:Data: 2022-09-09T22:38:06.816 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.816 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:06.817 DEBUG:boto:Port: 8001 2022-09-09T22:38:06.817 DEBUG:boto:Params: {} 2022-09-09T22:38:06.818 DEBUG:boto:Token: None 2022-09-09T22:38:06.819 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.820 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:SsySTU8tlc7/CcTcxFnIXNR7eoA= 2022-09-09T22:38:06.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:SsySTU8tlc7/CcTcxFnIXNR7eoA=', 'Content-Length': '0'} 2022-09-09T22:38:06.824 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b302ec96bdec6d75-00631bc04e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.866 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:06.867 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:06.868 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.868 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.869 DEBUG:boto:Method: GET 2022-09-09T22:38:06.869 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.870 DEBUG:boto:Data: 2022-09-09T22:38:06.871 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.871 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:06.872 DEBUG:boto:Port: 8000 2022-09-09T22:38:06.873 DEBUG:boto:Params: {} 2022-09-09T22:38:06.873 DEBUG:boto:Token: None 2022-09-09T22:38:06.874 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:06.875 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:SsySTU8tlc7/CcTcxFnIXNR7eoA= 2022-09-09T22:38:06.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:SsySTU8tlc7/CcTcxFnIXNR7eoA=', 'Content-Length': '0'} 2022-09-09T22:38:06.878 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2e8a75ec4e9c9c2-00631bc04e-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.879 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:06.880 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:06.881 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:06.881 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:06.882 DEBUG:boto:Method: GET 2022-09-09T22:38:06.882 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:06.883 DEBUG:boto:Data: 2022-09-09T22:38:06.883 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.884 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:06.884 DEBUG:boto:Port: 8001 2022-09-09T22:38:06.885 DEBUG:boto:Params: {} 2022-09-09T22:38:06.886 DEBUG:boto:Token: None 2022-09-09T22:38:06.886 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/ 2022-09-09T22:38:06.887 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JA6Ij0lw2WdAvLLLh0jRGEh5b+c= 2022-09-09T22:38:06.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JA6Ij0lw2WdAvLLLh0jRGEh5b+c=', 'Content-Length': '0'} 2022-09-09T22:38:06.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077233e1278f52b4b-00631bc04e-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.895 DEBUG:boto:b'jkcgbj-16obj-test-zone31000falseobj-test-zone32022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:06.896 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:06.897 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:06.897 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:06.898 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:06.899 DEBUG:boto:Method: GET 2022-09-09T22:38:06.899 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:06.900 DEBUG:boto:Data: 2022-09-09T22:38:06.901 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.901 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:06.902 DEBUG:boto:Port: 8000 2022-09-09T22:38:06.902 DEBUG:boto:Params: {} 2022-09-09T22:38:06.903 DEBUG:boto:Token: None 2022-09-09T22:38:06.903 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/ 2022-09-09T22:38:06.904 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JA6Ij0lw2WdAvLLLh0jRGEh5b+c= 2022-09-09T22:38:06.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JA6Ij0lw2WdAvLLLh0jRGEh5b+c=', 'Content-Length': '0'} 2022-09-09T22:38:06.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002635f8a829d8830d-00631bc04e-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.954 DEBUG:boto:b'jkcgbj-16obj-test-zone31000falseobj-test-zone32022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:06.955 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-16 zones={test-zone3, test-zone1} 2022-09-09T22:38:06.956 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-16 zones={test-zone3, test-zone2} 2022-09-09T22:38:06.956 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:06.957 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:06.957 DEBUG:boto:Method: HEAD 2022-09-09T22:38:06.959 DEBUG:boto:Path: /jkcgbj-16/ 2022-09-09T22:38:06.959 DEBUG:boto:Data: 2022-09-09T22:38:06.960 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.960 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:06.961 DEBUG:boto:Port: 8001 2022-09-09T22:38:06.961 DEBUG:boto:Params: {} 2022-09-09T22:38:06.962 DEBUG:boto:Token: None 2022-09-09T22:38:06.962 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/ 2022-09-09T22:38:06.963 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CgFJ/16mTyrcHSqvNqyZhtt6rAE= 2022-09-09T22:38:06.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CgFJ/16mTyrcHSqvNqyZhtt6rAE=', 'Content-Length': '0'} 2022-09-09T22:38:06.969 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b45ca51b8b5e655-00631bc04e-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.970 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:06.971 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:06.971 DEBUG:boto:Method: HEAD 2022-09-09T22:38:06.972 DEBUG:boto:Path: /jkcgbj-16/ 2022-09-09T22:38:06.972 DEBUG:boto:Data: 2022-09-09T22:38:06.973 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.973 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:06.974 DEBUG:boto:Port: 8000 2022-09-09T22:38:06.974 DEBUG:boto:Params: {} 2022-09-09T22:38:06.975 DEBUG:boto:Token: None 2022-09-09T22:38:06.976 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/ 2022-09-09T22:38:06.976 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CgFJ/16mTyrcHSqvNqyZhtt6rAE= 2022-09-09T22:38:06.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CgFJ/16mTyrcHSqvNqyZhtt6rAE=', 'Content-Length': '0'} 2022-09-09T22:38:06.983 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000660426649cda376f-00631bc04e-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.983 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:38:06.984 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:06.985 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:06.985 DEBUG:boto:path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:06.986 DEBUG:boto:auth_path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:06.986 DEBUG:boto:Method: GET 2022-09-09T22:38:06.987 DEBUG:boto:Path: /jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:06.987 DEBUG:boto:Data: 2022-09-09T22:38:06.988 DEBUG:boto:Headers: {} 2022-09-09T22:38:06.988 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:06.989 DEBUG:boto:Port: 8001 2022-09-09T22:38:06.989 DEBUG:boto:Params: {} 2022-09-09T22:38:06.990 DEBUG:boto:Token: None 2022-09-09T22:38:06.991 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:06 GMT /jkcgbj-16/?versions 2022-09-09T22:38:06.991 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:hmINqK4go1/WhQMFBEkl4TtMVxg= 2022-09-09T22:38:06.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:06 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:hmINqK4go1/WhQMFBEkl4TtMVxg=', 'Content-Length': '0'} 2022-09-09T22:38:06.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084d70e21f267cfa1-00631bc04e-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:06.999 DEBUG:boto:b'jkcgbj-16999falsedummyJ2BEYVUeYOtJqYkdQX3amzkINxiyxtstrue2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1123zMLKlbUHrE3mBcpUCJblKnWDeBXDtrue2022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2MZ.b57oN5HI9SlBmYt.8.2016m6aPMftrue2022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3AesdIyOhh8xrq24fWFi6fMM39PxhXM8true2022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:07.000 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:38:07.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:38:07.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:38:07.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:38:07.002 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:38:07.003 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:07.003 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:07.004 DEBUG:boto:path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:07.004 DEBUG:boto:auth_path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:07.005 DEBUG:boto:Method: GET 2022-09-09T22:38:07.005 DEBUG:boto:Path: /jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:07.006 DEBUG:boto:Data: 2022-09-09T22:38:07.006 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.007 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:07.008 DEBUG:boto:Port: 8000 2022-09-09T22:38:07.008 DEBUG:boto:Params: {} 2022-09-09T22:38:07.009 DEBUG:boto:Token: None 2022-09-09T22:38:07.009 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/?versions 2022-09-09T22:38:07.010 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+bnoX3VM+oiWYC7A0PQ4ghNhP4Q= 2022-09-09T22:38:07.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+bnoX3VM+oiWYC7A0PQ4ghNhP4Q=', 'Content-Length': '0'} 2022-09-09T22:38:07.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cba0938687ebd9bc-00631bc04f-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.019 DEBUG:boto:b'jkcgbj-16999falsedummyJ2BEYVUeYOtJqYkdQX3amzkINxiyxtstrue2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1123zMLKlbUHrE3mBcpUCJblKnWDeBXDtrue2022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2MZ.b57oN5HI9SlBmYt.8.2016m6aPMftrue2022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3AesdIyOhh8xrq24fWFi6fMM39PxhXM8true2022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:07.020 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:38:07.020 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:38:07.020 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:38:07.021 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:38:07.022 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:07.022 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:07.023 DEBUG:boto:path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:07.023 DEBUG:boto:auth_path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:07.024 DEBUG:boto:Method: GET 2022-09-09T22:38:07.024 DEBUG:boto:Path: /jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:07.025 DEBUG:boto:Data: 2022-09-09T22:38:07.025 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.026 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:07.026 DEBUG:boto:Port: 8001 2022-09-09T22:38:07.027 DEBUG:boto:Params: {} 2022-09-09T22:38:07.028 DEBUG:boto:Token: None 2022-09-09T22:38:07.028 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/?versions 2022-09-09T22:38:07.029 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+bnoX3VM+oiWYC7A0PQ4ghNhP4Q= 2022-09-09T22:38:07.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+bnoX3VM+oiWYC7A0PQ4ghNhP4Q=', 'Content-Length': '0'} 2022-09-09T22:38:07.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cc9543c384e62b3-00631bc04f-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.036 DEBUG:boto:b'jkcgbj-16999falsedummyJ2BEYVUeYOtJqYkdQX3amzkINxiyxtstrue2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1123zMLKlbUHrE3mBcpUCJblKnWDeBXDtrue2022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2MZ.b57oN5HI9SlBmYt.8.2016m6aPMftrue2022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3AesdIyOhh8xrq24fWFi6fMM39PxhXM8true2022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:07.037 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:07.038 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:07.038 DEBUG:boto:path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:07.039 DEBUG:boto:auth_path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:07.040 DEBUG:boto:Method: GET 2022-09-09T22:38:07.041 DEBUG:boto:Path: /jkcgbj-16/?versions&max-keys=999 2022-09-09T22:38:07.042 DEBUG:boto:Data: 2022-09-09T22:38:07.043 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.044 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:07.045 DEBUG:boto:Port: 8000 2022-09-09T22:38:07.045 DEBUG:boto:Params: {} 2022-09-09T22:38:07.046 DEBUG:boto:Token: None 2022-09-09T22:38:07.047 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/?versions 2022-09-09T22:38:07.048 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+bnoX3VM+oiWYC7A0PQ4ghNhP4Q= 2022-09-09T22:38:07.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+bnoX3VM+oiWYC7A0PQ4ghNhP4Q=', 'Content-Length': '0'} 2022-09-09T22:38:07.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006aa59ad8121d0012-00631bc04f-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.056 DEBUG:boto:b'jkcgbj-16999falsedummyJ2BEYVUeYOtJqYkdQX3amzkINxiyxtstrue2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1123zMLKlbUHrE3mBcpUCJblKnWDeBXDtrue2022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2MZ.b57oN5HI9SlBmYt.8.2016m6aPMftrue2022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3AesdIyOhh8xrq24fWFi6fMM39PxhXM8true2022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:07.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:38:07.058 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:38:07.059 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:38:07.062 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.063 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.064 DEBUG:boto:Method: GET 2022-09-09T22:38:07.065 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.066 DEBUG:boto:Data: 2022-09-09T22:38:07.067 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.069 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:07.070 DEBUG:boto:Port: 8001 2022-09-09T22:38:07.071 DEBUG:boto:Params: {} 2022-09-09T22:38:07.072 DEBUG:boto:Token: None 2022-09-09T22:38:07.073 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.074 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sLik36/0HwtNK0K0Q7Ad8nHwuYU= 2022-09-09T22:38:07.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sLik36/0HwtNK0K0Q7Ad8nHwuYU=', 'Content-Length': '0'} 2022-09-09T22:38:07.078 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c36483daea8a4591-00631bc04f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.079 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:38:07.080 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:38:07.081 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.082 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.083 DEBUG:boto:Method: GET 2022-09-09T22:38:07.083 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.084 DEBUG:boto:Data: 2022-09-09T22:38:07.085 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.085 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:07.086 DEBUG:boto:Port: 8000 2022-09-09T22:38:07.087 DEBUG:boto:Params: {} 2022-09-09T22:38:07.089 DEBUG:boto:Token: None 2022-09-09T22:38:07.089 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.090 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sLik36/0HwtNK0K0Q7Ad8nHwuYU= 2022-09-09T22:38:07.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sLik36/0HwtNK0K0Q7Ad8nHwuYU=', 'Content-Length': '0'} 2022-09-09T22:38:07.094 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006bc217b2783ce7df-00631bc04f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.094 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:38:07.095 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:38:07.096 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.096 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.097 DEBUG:boto:Method: HEAD 2022-09-09T22:38:07.097 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.098 DEBUG:boto:Data: 2022-09-09T22:38:07.098 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.099 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:07.099 DEBUG:boto:Port: 8001 2022-09-09T22:38:07.100 DEBUG:boto:Params: {} 2022-09-09T22:38:07.100 DEBUG:boto:Token: None 2022-09-09T22:38:07.101 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.101 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:SDXaj5MplbEry+PzoHInXry1gGw= 2022-09-09T22:38:07.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:SDXaj5MplbEry+PzoHInXry1gGw=', 'Content-Length': '0'} 2022-09-09T22:38:07.105 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d25d58af136097da-00631bc04f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.106 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:38:07.107 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:38:07.107 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.108 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.108 DEBUG:boto:Method: HEAD 2022-09-09T22:38:07.109 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.109 DEBUG:boto:Data: 2022-09-09T22:38:07.110 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.110 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:07.111 DEBUG:boto:Port: 8000 2022-09-09T22:38:07.111 DEBUG:boto:Params: {} 2022-09-09T22:38:07.112 DEBUG:boto:Token: None 2022-09-09T22:38:07.112 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.113 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:SDXaj5MplbEry+PzoHInXry1gGw= 2022-09-09T22:38:07.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:SDXaj5MplbEry+PzoHInXry1gGw=', 'Content-Length': '0'} 2022-09-09T22:38:07.117 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d750c55612577a34-00631bc04f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:38:07.118 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:38:07.119 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:38:07.119 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.120 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.120 DEBUG:boto:Method: GET 2022-09-09T22:38:07.121 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.121 DEBUG:boto:Data: 2022-09-09T22:38:07.122 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.123 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:07.123 DEBUG:boto:Port: 8001 2022-09-09T22:38:07.124 DEBUG:boto:Params: {} 2022-09-09T22:38:07.124 DEBUG:boto:Token: None 2022-09-09T22:38:07.125 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.126 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sLik36/0HwtNK0K0Q7Ad8nHwuYU= 2022-09-09T22:38:07.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sLik36/0HwtNK0K0Q7Ad8nHwuYU=', 'Content-Length': '0'} 2022-09-09T22:38:07.130 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003c6137e133b34cea-00631bc04f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.130 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:38:07.131 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:38:07.132 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.132 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.133 DEBUG:boto:Method: GET 2022-09-09T22:38:07.133 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.134 DEBUG:boto:Data: 2022-09-09T22:38:07.135 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.136 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:07.137 DEBUG:boto:Port: 8000 2022-09-09T22:38:07.137 DEBUG:boto:Params: {} 2022-09-09T22:38:07.139 DEBUG:boto:Token: None 2022-09-09T22:38:07.140 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.141 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sLik36/0HwtNK0K0Q7Ad8nHwuYU= 2022-09-09T22:38:07.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sLik36/0HwtNK0K0Q7Ad8nHwuYU=', 'Content-Length': '0'} 2022-09-09T22:38:07.146 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000be58a36861408624-00631bc04f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.147 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:07.148 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:07.148 DEBUG:boto:path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:38:07.149 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:38:07.149 DEBUG:boto:Method: GET 2022-09-09T22:38:07.150 DEBUG:boto:Path: /jkcgbj-16/?prefix=dummy 2022-09-09T22:38:07.150 DEBUG:boto:Data: 2022-09-09T22:38:07.151 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.151 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:07.152 DEBUG:boto:Port: 8001 2022-09-09T22:38:07.152 DEBUG:boto:Params: {} 2022-09-09T22:38:07.153 DEBUG:boto:Token: None 2022-09-09T22:38:07.154 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/ 2022-09-09T22:38:07.154 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:oA5xCW+RIyGVULrHNZXsGFncjBA= 2022-09-09T22:38:07.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:oA5xCW+RIyGVULrHNZXsGFncjBA=', 'Content-Length': '0'} 2022-09-09T22:38:07.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb79731b570525f0-00631bc04f-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.202 DEBUG:boto:b'jkcgbj-16dummy1000falsedummy2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:07.203 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:07.204 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:07.204 DEBUG:boto:path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:38:07.205 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:38:07.205 DEBUG:boto:Method: GET 2022-09-09T22:38:07.206 DEBUG:boto:Path: /jkcgbj-16/?prefix=dummy 2022-09-09T22:38:07.206 DEBUG:boto:Data: 2022-09-09T22:38:07.207 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.209 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:07.210 DEBUG:boto:Port: 8000 2022-09-09T22:38:07.210 DEBUG:boto:Params: {} 2022-09-09T22:38:07.211 DEBUG:boto:Token: None 2022-09-09T22:38:07.212 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/ 2022-09-09T22:38:07.212 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:oA5xCW+RIyGVULrHNZXsGFncjBA= 2022-09-09T22:38:07.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:oA5xCW+RIyGVULrHNZXsGFncjBA=', 'Content-Length': '0'} 2022-09-09T22:38:07.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb87bfe2cc71c03a-00631bc04f-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.219 DEBUG:boto:b'jkcgbj-16dummy1000falsedummy2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:07.220 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:38:07.220 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:38:07.221 DEBUG:boto:Method: HEAD 2022-09-09T22:38:07.221 DEBUG:boto:Path: /jkcgbj-16/dummy 2022-09-09T22:38:07.222 DEBUG:boto:Data: 2022-09-09T22:38:07.222 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.223 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:07.223 DEBUG:boto:Port: 8001 2022-09-09T22:38:07.224 DEBUG:boto:Params: {} 2022-09-09T22:38:07.224 DEBUG:boto:Token: None 2022-09-09T22:38:07.225 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/dummy 2022-09-09T22:38:07.225 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:SAY5yGCQjEG77VzehDLOOt6irwQ= 2022-09-09T22:38:07.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:SAY5yGCQjEG77VzehDLOOt6irwQ=', 'Content-Length': '0'} 2022-09-09T22:38:07.230 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009018b684cc8d30cb-00631bc04f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.231 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:38:07.231 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:38:07.232 DEBUG:boto:Method: HEAD 2022-09-09T22:38:07.232 DEBUG:boto:Path: /jkcgbj-16/dummy 2022-09-09T22:38:07.233 DEBUG:boto:Data: 2022-09-09T22:38:07.233 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.234 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:07.234 DEBUG:boto:Port: 8000 2022-09-09T22:38:07.235 DEBUG:boto:Params: {} 2022-09-09T22:38:07.235 DEBUG:boto:Token: None 2022-09-09T22:38:07.236 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/dummy 2022-09-09T22:38:07.236 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:SAY5yGCQjEG77VzehDLOOt6irwQ= 2022-09-09T22:38:07.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:SAY5yGCQjEG77VzehDLOOt6irwQ=', 'Content-Length': '0'} 2022-09-09T22:38:07.241 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002df14f8643734127-00631bc04f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:38:07.242 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:38:07.242 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:38:07.243 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.243 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.244 DEBUG:boto:Method: GET 2022-09-09T22:38:07.244 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.245 DEBUG:boto:Data: 2022-09-09T22:38:07.245 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.246 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:07.246 DEBUG:boto:Port: 8001 2022-09-09T22:38:07.247 DEBUG:boto:Params: {} 2022-09-09T22:38:07.248 DEBUG:boto:Token: None 2022-09-09T22:38:07.248 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.249 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sLik36/0HwtNK0K0Q7Ad8nHwuYU= 2022-09-09T22:38:07.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sLik36/0HwtNK0K0Q7Ad8nHwuYU=', 'Content-Length': '0'} 2022-09-09T22:38:07.253 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000080eb67bdba24450e-00631bc04f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.254 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:38:07.254 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:38:07.255 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.255 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.256 DEBUG:boto:Method: GET 2022-09-09T22:38:07.256 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.257 DEBUG:boto:Data: 2022-09-09T22:38:07.257 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.258 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:07.258 DEBUG:boto:Port: 8000 2022-09-09T22:38:07.259 DEBUG:boto:Params: {} 2022-09-09T22:38:07.259 DEBUG:boto:Token: None 2022-09-09T22:38:07.260 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:38:07.261 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sLik36/0HwtNK0K0Q7Ad8nHwuYU= 2022-09-09T22:38:07.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sLik36/0HwtNK0K0Q7Ad8nHwuYU=', 'Content-Length': '0'} 2022-09-09T22:38:07.264 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002cc889b69070e37c-00631bc04f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.265 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:07.266 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:07.266 DEBUG:boto:path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:38:07.267 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:38:07.267 DEBUG:boto:Method: GET 2022-09-09T22:38:07.268 DEBUG:boto:Path: /jkcgbj-16/?prefix=dummy 2022-09-09T22:38:07.268 DEBUG:boto:Data: 2022-09-09T22:38:07.269 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.269 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:07.270 DEBUG:boto:Port: 8001 2022-09-09T22:38:07.270 DEBUG:boto:Params: {} 2022-09-09T22:38:07.271 DEBUG:boto:Token: None 2022-09-09T22:38:07.271 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/ 2022-09-09T22:38:07.272 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:oA5xCW+RIyGVULrHNZXsGFncjBA= 2022-09-09T22:38:07.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:oA5xCW+RIyGVULrHNZXsGFncjBA=', 'Content-Length': '0'} 2022-09-09T22:38:07.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be543f3720b67cf6-00631bc04f-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.279 DEBUG:boto:b'jkcgbj-16dummy1000falsedummy2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:07.280 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:07.281 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:07.281 DEBUG:boto:path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:38:07.282 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:38:07.282 DEBUG:boto:Method: GET 2022-09-09T22:38:07.283 DEBUG:boto:Path: /jkcgbj-16/?prefix=dummy 2022-09-09T22:38:07.283 DEBUG:boto:Data: 2022-09-09T22:38:07.284 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.284 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:07.285 DEBUG:boto:Port: 8000 2022-09-09T22:38:07.285 DEBUG:boto:Params: {} 2022-09-09T22:38:07.286 DEBUG:boto:Token: None 2022-09-09T22:38:07.286 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/ 2022-09-09T22:38:07.287 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:oA5xCW+RIyGVULrHNZXsGFncjBA= 2022-09-09T22:38:07.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:oA5xCW+RIyGVULrHNZXsGFncjBA=', 'Content-Length': '0'} 2022-09-09T22:38:07.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a11786e59afb49b-00631bc04f-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.294 DEBUG:boto:b'jkcgbj-16dummy1000falsedummy2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:07.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:38:07.295 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:07.296 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:07.296 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.297 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.297 DEBUG:boto:Method: GET 2022-09-09T22:38:07.298 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.298 DEBUG:boto:Data: 2022-09-09T22:38:07.299 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.299 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:07.300 DEBUG:boto:Port: 8001 2022-09-09T22:38:07.300 DEBUG:boto:Params: {} 2022-09-09T22:38:07.301 DEBUG:boto:Token: None 2022-09-09T22:38:07.301 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.302 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:z9pPKF4AL/PPKCWcHf8EjjHQ12s= 2022-09-09T22:38:07.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:z9pPKF4AL/PPKCWcHf8EjjHQ12s=', 'Content-Length': '0'} 2022-09-09T22:38:07.306 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad91ce6a48c8a7dd-00631bc04f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.307 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:07.307 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:07.308 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.308 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.309 DEBUG:boto:Method: GET 2022-09-09T22:38:07.309 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.310 DEBUG:boto:Data: 2022-09-09T22:38:07.310 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.311 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:07.311 DEBUG:boto:Port: 8000 2022-09-09T22:38:07.312 DEBUG:boto:Params: {} 2022-09-09T22:38:07.312 DEBUG:boto:Token: None 2022-09-09T22:38:07.313 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.313 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:z9pPKF4AL/PPKCWcHf8EjjHQ12s= 2022-09-09T22:38:07.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:z9pPKF4AL/PPKCWcHf8EjjHQ12s=', 'Content-Length': '0'} 2022-09-09T22:38:07.317 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009114168437566e82-00631bc04f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.362 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:07.363 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:07.364 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.364 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.365 DEBUG:boto:Method: HEAD 2022-09-09T22:38:07.365 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.366 DEBUG:boto:Data: 2022-09-09T22:38:07.366 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.367 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:07.367 DEBUG:boto:Port: 8001 2022-09-09T22:38:07.368 DEBUG:boto:Params: {} 2022-09-09T22:38:07.368 DEBUG:boto:Token: None 2022-09-09T22:38:07.369 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.370 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4a9XyBXAxU6FVeGtqlPKkyoHu9I= 2022-09-09T22:38:07.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4a9XyBXAxU6FVeGtqlPKkyoHu9I=', 'Content-Length': '0'} 2022-09-09T22:38:07.373 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fac79e3ca3b0c801-00631bc04f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.374 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:07.375 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:07.375 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.376 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.376 DEBUG:boto:Method: HEAD 2022-09-09T22:38:07.377 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.377 DEBUG:boto:Data: 2022-09-09T22:38:07.378 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.379 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:07.379 DEBUG:boto:Port: 8000 2022-09-09T22:38:07.380 DEBUG:boto:Params: {} 2022-09-09T22:38:07.380 DEBUG:boto:Token: None 2022-09-09T22:38:07.381 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.382 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4a9XyBXAxU6FVeGtqlPKkyoHu9I= 2022-09-09T22:38:07.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4a9XyBXAxU6FVeGtqlPKkyoHu9I=', 'Content-Length': '0'} 2022-09-09T22:38:07.386 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b69ff924a6c4607-00631bc04f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:38:07.388 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:07.388 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:07.389 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.389 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.390 DEBUG:boto:Method: GET 2022-09-09T22:38:07.391 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.391 DEBUG:boto:Data: 2022-09-09T22:38:07.392 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.392 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:07.393 DEBUG:boto:Port: 8001 2022-09-09T22:38:07.393 DEBUG:boto:Params: {} 2022-09-09T22:38:07.394 DEBUG:boto:Token: None 2022-09-09T22:38:07.395 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.395 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:z9pPKF4AL/PPKCWcHf8EjjHQ12s= 2022-09-09T22:38:07.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:z9pPKF4AL/PPKCWcHf8EjjHQ12s=', 'Content-Length': '0'} 2022-09-09T22:38:07.399 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033e33248c2689c9f-00631bc04f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.400 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:07.400 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:07.401 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.401 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.402 DEBUG:boto:Method: GET 2022-09-09T22:38:07.403 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.403 DEBUG:boto:Data: 2022-09-09T22:38:07.404 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.404 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:07.405 DEBUG:boto:Port: 8000 2022-09-09T22:38:07.406 DEBUG:boto:Params: {} 2022-09-09T22:38:07.407 DEBUG:boto:Token: None 2022-09-09T22:38:07.407 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.408 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:z9pPKF4AL/PPKCWcHf8EjjHQ12s= 2022-09-09T22:38:07.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:z9pPKF4AL/PPKCWcHf8EjjHQ12s=', 'Content-Length': '0'} 2022-09-09T22:38:07.412 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eb544d5ca9f875c2-00631bc04f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.413 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:07.414 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:07.414 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:07.415 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:07.415 DEBUG:boto:Method: GET 2022-09-09T22:38:07.416 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:07.417 DEBUG:boto:Data: 2022-09-09T22:38:07.417 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.418 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:07.419 DEBUG:boto:Port: 8001 2022-09-09T22:38:07.419 DEBUG:boto:Params: {} 2022-09-09T22:38:07.420 DEBUG:boto:Token: None 2022-09-09T22:38:07.420 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/ 2022-09-09T22:38:07.421 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:oA5xCW+RIyGVULrHNZXsGFncjBA= 2022-09-09T22:38:07.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:oA5xCW+RIyGVULrHNZXsGFncjBA=', 'Content-Length': '0'} 2022-09-09T22:38:07.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7df4e7d4bba671b-00631bc04f-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.474 DEBUG:boto:b'jkcgbj-16obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:07.475 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:07.476 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:07.476 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:07.477 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:07.477 DEBUG:boto:Method: GET 2022-09-09T22:38:07.478 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:07.478 DEBUG:boto:Data: 2022-09-09T22:38:07.479 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.479 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:07.480 DEBUG:boto:Port: 8000 2022-09-09T22:38:07.481 DEBUG:boto:Params: {} 2022-09-09T22:38:07.482 DEBUG:boto:Token: None 2022-09-09T22:38:07.482 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/ 2022-09-09T22:38:07.483 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:oA5xCW+RIyGVULrHNZXsGFncjBA= 2022-09-09T22:38:07.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:oA5xCW+RIyGVULrHNZXsGFncjBA=', 'Content-Length': '0'} 2022-09-09T22:38:07.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086e35120d03b6acb-00631bc04f-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.489 DEBUG:boto:b'jkcgbj-16obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:07.490 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:07.491 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:07.492 DEBUG:boto:Method: HEAD 2022-09-09T22:38:07.492 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1 2022-09-09T22:38:07.493 DEBUG:boto:Data: 2022-09-09T22:38:07.493 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.494 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:07.494 DEBUG:boto:Port: 8001 2022-09-09T22:38:07.495 DEBUG:boto:Params: {} 2022-09-09T22:38:07.495 DEBUG:boto:Token: None 2022-09-09T22:38:07.496 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/obj-test-zone1 2022-09-09T22:38:07.496 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:kwd8edp1ZYtbnKSRvOGJRFZlhIM= 2022-09-09T22:38:07.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:kwd8edp1ZYtbnKSRvOGJRFZlhIM=', 'Content-Length': '0'} 2022-09-09T22:38:07.504 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ab69c9acaee09dc-00631bc04f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.506 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:07.506 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:07.507 DEBUG:boto:Method: HEAD 2022-09-09T22:38:07.508 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1 2022-09-09T22:38:07.510 DEBUG:boto:Data: 2022-09-09T22:38:07.511 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.513 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:07.514 DEBUG:boto:Port: 8000 2022-09-09T22:38:07.514 DEBUG:boto:Params: {} 2022-09-09T22:38:07.515 DEBUG:boto:Token: None 2022-09-09T22:38:07.516 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/obj-test-zone1 2022-09-09T22:38:07.516 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:kwd8edp1ZYtbnKSRvOGJRFZlhIM= 2022-09-09T22:38:07.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:kwd8edp1ZYtbnKSRvOGJRFZlhIM=', 'Content-Length': '0'} 2022-09-09T22:38:07.520 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037ae94166e3f0e6a-00631bc04f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:38:07.522 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:07.522 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:07.523 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.524 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.524 DEBUG:boto:Method: GET 2022-09-09T22:38:07.525 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.525 DEBUG:boto:Data: 2022-09-09T22:38:07.526 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.526 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:07.527 DEBUG:boto:Port: 8001 2022-09-09T22:38:07.528 DEBUG:boto:Params: {} 2022-09-09T22:38:07.528 DEBUG:boto:Token: None 2022-09-09T22:38:07.529 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.529 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:z9pPKF4AL/PPKCWcHf8EjjHQ12s= 2022-09-09T22:38:07.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:z9pPKF4AL/PPKCWcHf8EjjHQ12s=', 'Content-Length': '0'} 2022-09-09T22:38:07.534 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000083c041eef752aec5-00631bc04f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.536 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:07.536 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:07.537 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.538 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.538 DEBUG:boto:Method: GET 2022-09-09T22:38:07.539 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.540 DEBUG:boto:Data: 2022-09-09T22:38:07.540 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.541 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:07.542 DEBUG:boto:Port: 8000 2022-09-09T22:38:07.543 DEBUG:boto:Params: {} 2022-09-09T22:38:07.544 DEBUG:boto:Token: None 2022-09-09T22:38:07.545 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:38:07.546 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:z9pPKF4AL/PPKCWcHf8EjjHQ12s= 2022-09-09T22:38:07.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:z9pPKF4AL/PPKCWcHf8EjjHQ12s=', 'Content-Length': '0'} 2022-09-09T22:38:07.550 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b122d83808b9eb12-00631bc04f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.599 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:07.600 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:07.601 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:07.601 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:07.602 DEBUG:boto:Method: GET 2022-09-09T22:38:07.602 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:07.603 DEBUG:boto:Data: 2022-09-09T22:38:07.604 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.604 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:07.605 DEBUG:boto:Port: 8001 2022-09-09T22:38:07.606 DEBUG:boto:Params: {} 2022-09-09T22:38:07.607 DEBUG:boto:Token: None 2022-09-09T22:38:07.607 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/ 2022-09-09T22:38:07.608 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:oA5xCW+RIyGVULrHNZXsGFncjBA= 2022-09-09T22:38:07.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:oA5xCW+RIyGVULrHNZXsGFncjBA=', 'Content-Length': '0'} 2022-09-09T22:38:07.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ba98c7f21e8edea-00631bc04f-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.614 DEBUG:boto:b'jkcgbj-16obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:07.615 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:07.616 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:07.616 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:07.617 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:07.617 DEBUG:boto:Method: GET 2022-09-09T22:38:07.618 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:38:07.618 DEBUG:boto:Data: 2022-09-09T22:38:07.619 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.619 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:07.620 DEBUG:boto:Port: 8000 2022-09-09T22:38:07.620 DEBUG:boto:Params: {} 2022-09-09T22:38:07.621 DEBUG:boto:Token: None 2022-09-09T22:38:07.621 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/ 2022-09-09T22:38:07.622 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:oA5xCW+RIyGVULrHNZXsGFncjBA= 2022-09-09T22:38:07.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:oA5xCW+RIyGVULrHNZXsGFncjBA=', 'Content-Length': '0'} 2022-09-09T22:38:07.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096ce81c1c41a2e4f-00631bc04f-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.627 DEBUG:boto:b'jkcgbj-16obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:07.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:38:07.629 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:07.630 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:07.631 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.631 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.632 DEBUG:boto:Method: GET 2022-09-09T22:38:07.632 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.633 DEBUG:boto:Data: 2022-09-09T22:38:07.634 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.635 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:07.635 DEBUG:boto:Port: 8001 2022-09-09T22:38:07.636 DEBUG:boto:Params: {} 2022-09-09T22:38:07.636 DEBUG:boto:Token: None 2022-09-09T22:38:07.637 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.637 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:AnD4RORF8utrnjKvPT15dTEZAkA= 2022-09-09T22:38:07.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:AnD4RORF8utrnjKvPT15dTEZAkA=', 'Content-Length': '0'} 2022-09-09T22:38:07.642 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003f260f46bad28398-00631bc04f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.690 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:07.691 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:07.692 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.694 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.695 DEBUG:boto:Method: GET 2022-09-09T22:38:07.695 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.696 DEBUG:boto:Data: 2022-09-09T22:38:07.696 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.697 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:07.698 DEBUG:boto:Port: 8000 2022-09-09T22:38:07.698 DEBUG:boto:Params: {} 2022-09-09T22:38:07.699 DEBUG:boto:Token: None 2022-09-09T22:38:07.700 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.700 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:AnD4RORF8utrnjKvPT15dTEZAkA= 2022-09-09T22:38:07.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:AnD4RORF8utrnjKvPT15dTEZAkA=', 'Content-Length': '0'} 2022-09-09T22:38:07.705 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000020ec57f67554f42c-00631bc04f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.707 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:07.708 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:07.709 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.710 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.711 DEBUG:boto:Method: HEAD 2022-09-09T22:38:07.712 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.713 DEBUG:boto:Data: 2022-09-09T22:38:07.713 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.714 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:07.714 DEBUG:boto:Port: 8001 2022-09-09T22:38:07.715 DEBUG:boto:Params: {} 2022-09-09T22:38:07.716 DEBUG:boto:Token: None 2022-09-09T22:38:07.716 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.717 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:e8qtbGEBPMwKfESDDjuaFQtMi7Y= 2022-09-09T22:38:07.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:e8qtbGEBPMwKfESDDjuaFQtMi7Y=', 'Content-Length': '0'} 2022-09-09T22:38:07.721 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a878a9657192a1d6-00631bc04f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.722 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:07.723 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:07.723 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.724 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.724 DEBUG:boto:Method: HEAD 2022-09-09T22:38:07.725 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.725 DEBUG:boto:Data: 2022-09-09T22:38:07.726 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.726 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:07.727 DEBUG:boto:Port: 8000 2022-09-09T22:38:07.728 DEBUG:boto:Params: {} 2022-09-09T22:38:07.729 DEBUG:boto:Token: None 2022-09-09T22:38:07.730 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.731 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:e8qtbGEBPMwKfESDDjuaFQtMi7Y= 2022-09-09T22:38:07.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:e8qtbGEBPMwKfESDDjuaFQtMi7Y=', 'Content-Length': '0'} 2022-09-09T22:38:07.734 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005114a26dff16978f-00631bc04f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:38:07.735 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:07.736 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:07.737 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.737 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.738 DEBUG:boto:Method: GET 2022-09-09T22:38:07.739 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.740 DEBUG:boto:Data: 2022-09-09T22:38:07.741 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.742 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:07.743 DEBUG:boto:Port: 8001 2022-09-09T22:38:07.743 DEBUG:boto:Params: {} 2022-09-09T22:38:07.745 DEBUG:boto:Token: None 2022-09-09T22:38:07.746 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.747 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:AnD4RORF8utrnjKvPT15dTEZAkA= 2022-09-09T22:38:07.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:AnD4RORF8utrnjKvPT15dTEZAkA=', 'Content-Length': '0'} 2022-09-09T22:38:07.751 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000078c2a6fc83b93e99-00631bc04f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.752 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:07.752 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:07.753 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.753 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.754 DEBUG:boto:Method: GET 2022-09-09T22:38:07.755 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.755 DEBUG:boto:Data: 2022-09-09T22:38:07.756 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.756 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:07.757 DEBUG:boto:Port: 8000 2022-09-09T22:38:07.758 DEBUG:boto:Params: {} 2022-09-09T22:38:07.758 DEBUG:boto:Token: None 2022-09-09T22:38:07.759 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.760 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:AnD4RORF8utrnjKvPT15dTEZAkA= 2022-09-09T22:38:07.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:AnD4RORF8utrnjKvPT15dTEZAkA=', 'Content-Length': '0'} 2022-09-09T22:38:07.764 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fda98f6317673397-00631bc04f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.765 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:07.765 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:07.766 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:07.767 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:07.767 DEBUG:boto:Method: GET 2022-09-09T22:38:07.768 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:07.768 DEBUG:boto:Data: 2022-09-09T22:38:07.769 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.769 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:07.770 DEBUG:boto:Port: 8001 2022-09-09T22:38:07.770 DEBUG:boto:Params: {} 2022-09-09T22:38:07.771 DEBUG:boto:Token: None 2022-09-09T22:38:07.772 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/ 2022-09-09T22:38:07.772 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:oA5xCW+RIyGVULrHNZXsGFncjBA= 2022-09-09T22:38:07.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:oA5xCW+RIyGVULrHNZXsGFncjBA=', 'Content-Length': '0'} 2022-09-09T22:38:07.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075b82d86c68aff04-00631bc04f-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.779 DEBUG:boto:b'jkcgbj-16obj-test-zone21000falseobj-test-zone22022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:07.779 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:07.780 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:07.780 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:07.781 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:07.781 DEBUG:boto:Method: GET 2022-09-09T22:38:07.782 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:07.782 DEBUG:boto:Data: 2022-09-09T22:38:07.783 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.784 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:07.784 DEBUG:boto:Port: 8000 2022-09-09T22:38:07.785 DEBUG:boto:Params: {} 2022-09-09T22:38:07.785 DEBUG:boto:Token: None 2022-09-09T22:38:07.786 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/ 2022-09-09T22:38:07.787 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:oA5xCW+RIyGVULrHNZXsGFncjBA= 2022-09-09T22:38:07.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:oA5xCW+RIyGVULrHNZXsGFncjBA=', 'Content-Length': '0'} 2022-09-09T22:38:07.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034f90539bda8ea66-00631bc04f-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.834 DEBUG:boto:b'jkcgbj-16obj-test-zone21000falseobj-test-zone22022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:07.835 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:07.836 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:07.836 DEBUG:boto:Method: HEAD 2022-09-09T22:38:07.837 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2 2022-09-09T22:38:07.837 DEBUG:boto:Data: 2022-09-09T22:38:07.838 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.838 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:07.839 DEBUG:boto:Port: 8001 2022-09-09T22:38:07.840 DEBUG:boto:Params: {} 2022-09-09T22:38:07.841 DEBUG:boto:Token: None 2022-09-09T22:38:07.842 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/obj-test-zone2 2022-09-09T22:38:07.842 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KH64eH2Hl+JkWNuEJWpvZANZUk0= 2022-09-09T22:38:07.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KH64eH2Hl+JkWNuEJWpvZANZUk0=', 'Content-Length': '0'} 2022-09-09T22:38:07.847 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000888b6e90d1e43ede-00631bc04f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.848 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:07.849 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:07.850 DEBUG:boto:Method: HEAD 2022-09-09T22:38:07.851 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2 2022-09-09T22:38:07.851 DEBUG:boto:Data: 2022-09-09T22:38:07.852 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.853 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:07.853 DEBUG:boto:Port: 8000 2022-09-09T22:38:07.854 DEBUG:boto:Params: {} 2022-09-09T22:38:07.855 DEBUG:boto:Token: None 2022-09-09T22:38:07.856 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/obj-test-zone2 2022-09-09T22:38:07.856 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KH64eH2Hl+JkWNuEJWpvZANZUk0= 2022-09-09T22:38:07.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KH64eH2Hl+JkWNuEJWpvZANZUk0=', 'Content-Length': '0'} 2022-09-09T22:38:07.861 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f30e11ba752d826-00631bc04f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:38:07.862 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:07.863 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:07.863 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.864 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.864 DEBUG:boto:Method: GET 2022-09-09T22:38:07.865 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.866 DEBUG:boto:Data: 2022-09-09T22:38:07.866 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.867 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:07.867 DEBUG:boto:Port: 8001 2022-09-09T22:38:07.868 DEBUG:boto:Params: {} 2022-09-09T22:38:07.868 DEBUG:boto:Token: None 2022-09-09T22:38:07.869 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.870 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:AnD4RORF8utrnjKvPT15dTEZAkA= 2022-09-09T22:38:07.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:AnD4RORF8utrnjKvPT15dTEZAkA=', 'Content-Length': '0'} 2022-09-09T22:38:07.876 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082f572707484f7e4-00631bc04f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.918 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:07.919 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:07.920 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.920 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.921 DEBUG:boto:Method: GET 2022-09-09T22:38:07.922 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.922 DEBUG:boto:Data: 2022-09-09T22:38:07.923 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.923 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:07.924 DEBUG:boto:Port: 8000 2022-09-09T22:38:07.924 DEBUG:boto:Params: {} 2022-09-09T22:38:07.925 DEBUG:boto:Token: None 2022-09-09T22:38:07.926 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:38:07.927 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:AnD4RORF8utrnjKvPT15dTEZAkA= 2022-09-09T22:38:07.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:AnD4RORF8utrnjKvPT15dTEZAkA=', 'Content-Length': '0'} 2022-09-09T22:38:07.931 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f0d5bdc6fdf16cc-00631bc04f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.932 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:07.933 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:07.934 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:07.934 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:07.935 DEBUG:boto:Method: GET 2022-09-09T22:38:07.935 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:07.936 DEBUG:boto:Data: 2022-09-09T22:38:07.937 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.937 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:07.938 DEBUG:boto:Port: 8001 2022-09-09T22:38:07.939 DEBUG:boto:Params: {} 2022-09-09T22:38:07.940 DEBUG:boto:Token: None 2022-09-09T22:38:07.941 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/ 2022-09-09T22:38:07.941 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:oA5xCW+RIyGVULrHNZXsGFncjBA= 2022-09-09T22:38:07.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:oA5xCW+RIyGVULrHNZXsGFncjBA=', 'Content-Length': '0'} 2022-09-09T22:38:07.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070dbfa8330c92edf-00631bc04f-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.949 DEBUG:boto:b'jkcgbj-16obj-test-zone21000falseobj-test-zone22022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:07.949 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:07.950 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:07.951 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:07.952 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:07.952 DEBUG:boto:Method: GET 2022-09-09T22:38:07.953 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:38:07.953 DEBUG:boto:Data: 2022-09-09T22:38:07.954 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.954 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:07.955 DEBUG:boto:Port: 8000 2022-09-09T22:38:07.956 DEBUG:boto:Params: {} 2022-09-09T22:38:07.958 DEBUG:boto:Token: None 2022-09-09T22:38:07.958 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/ 2022-09-09T22:38:07.959 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:oA5xCW+RIyGVULrHNZXsGFncjBA= 2022-09-09T22:38:07.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:oA5xCW+RIyGVULrHNZXsGFncjBA=', 'Content-Length': '0'} 2022-09-09T22:38:07.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f247c53ec2b20bdd-00631bc04f-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.970 DEBUG:boto:b'jkcgbj-16obj-test-zone21000falseobj-test-zone22022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:07.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:38:07.971 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:07.972 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:07.973 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:07.973 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:07.974 DEBUG:boto:Method: GET 2022-09-09T22:38:07.974 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:07.975 DEBUG:boto:Data: 2022-09-09T22:38:07.976 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.976 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:07.977 DEBUG:boto:Port: 8001 2022-09-09T22:38:07.977 DEBUG:boto:Params: {} 2022-09-09T22:38:07.978 DEBUG:boto:Token: None 2022-09-09T22:38:07.979 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:07.979 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0EfURDLeawV2nZbG24Q8f63Az2M= 2022-09-09T22:38:07.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0EfURDLeawV2nZbG24Q8f63Az2M=', 'Content-Length': '0'} 2022-09-09T22:38:07.983 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000761fb6a8dfa50942-00631bc04f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:07.984 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:07.984 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:07.986 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:07.986 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:07.987 DEBUG:boto:Method: GET 2022-09-09T22:38:07.987 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:07.988 DEBUG:boto:Data: 2022-09-09T22:38:07.988 DEBUG:boto:Headers: {} 2022-09-09T22:38:07.989 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:07.989 DEBUG:boto:Port: 8000 2022-09-09T22:38:07.990 DEBUG:boto:Params: {} 2022-09-09T22:38:07.991 DEBUG:boto:Token: None 2022-09-09T22:38:07.992 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:07 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:07.992 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0EfURDLeawV2nZbG24Q8f63Az2M= 2022-09-09T22:38:07.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0EfURDLeawV2nZbG24Q8f63Az2M=', 'Content-Length': '0'} 2022-09-09T22:38:07.996 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053c793305803ba0d-00631bc04f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:08.038 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:08.040 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:08.040 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:08.041 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:08.042 DEBUG:boto:Method: HEAD 2022-09-09T22:38:08.043 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:08.044 DEBUG:boto:Data: 2022-09-09T22:38:08.045 DEBUG:boto:Headers: {} 2022-09-09T22:38:08.046 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:08.046 DEBUG:boto:Port: 8001 2022-09-09T22:38:08.047 DEBUG:boto:Params: {} 2022-09-09T22:38:08.047 DEBUG:boto:Token: None 2022-09-09T22:38:08.048 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:08 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:08.048 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LW8Tu1n4n2ho2K819hwz1hmuVSg= 2022-09-09T22:38:08.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:08 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LW8Tu1n4n2ho2K819hwz1hmuVSg=', 'Content-Length': '0'} 2022-09-09T22:38:08.052 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac603c0212891c94-00631bc050-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:08.053 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:08.053 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:08.054 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:08.054 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:08.055 DEBUG:boto:Method: HEAD 2022-09-09T22:38:08.056 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:08.056 DEBUG:boto:Data: 2022-09-09T22:38:08.057 DEBUG:boto:Headers: {} 2022-09-09T22:38:08.057 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:08.058 DEBUG:boto:Port: 8000 2022-09-09T22:38:08.058 DEBUG:boto:Params: {} 2022-09-09T22:38:08.059 DEBUG:boto:Token: None 2022-09-09T22:38:08.059 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:08 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:08.060 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LW8Tu1n4n2ho2K819hwz1hmuVSg= 2022-09-09T22:38:08.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:08 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LW8Tu1n4n2ho2K819hwz1hmuVSg=', 'Content-Length': '0'} 2022-09-09T22:38:08.064 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000acfb04a373f1c062-00631bc050-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:08.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:38:08.066 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:08.067 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:08.068 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:08.068 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:08.069 DEBUG:boto:Method: GET 2022-09-09T22:38:08.069 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:08.070 DEBUG:boto:Data: 2022-09-09T22:38:08.070 DEBUG:boto:Headers: {} 2022-09-09T22:38:08.071 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:08.072 DEBUG:boto:Port: 8001 2022-09-09T22:38:08.073 DEBUG:boto:Params: {} 2022-09-09T22:38:08.074 DEBUG:boto:Token: None 2022-09-09T22:38:08.075 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:08 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:08.077 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ybCaQHDMx5GlEunOlCxyjd1ie3o= 2022-09-09T22:38:08.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:08 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ybCaQHDMx5GlEunOlCxyjd1ie3o=', 'Content-Length': '0'} 2022-09-09T22:38:08.082 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000065f6ae03345b86b2-00631bc050-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:08.126 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:08.127 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:08.128 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:08.129 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:08.130 DEBUG:boto:Method: GET 2022-09-09T22:38:08.131 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:08.132 DEBUG:boto:Data: 2022-09-09T22:38:08.133 DEBUG:boto:Headers: {} 2022-09-09T22:38:08.134 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:08.134 DEBUG:boto:Port: 8000 2022-09-09T22:38:08.135 DEBUG:boto:Params: {} 2022-09-09T22:38:08.136 DEBUG:boto:Token: None 2022-09-09T22:38:08.137 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:08 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:08.138 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ybCaQHDMx5GlEunOlCxyjd1ie3o= 2022-09-09T22:38:08.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:08 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ybCaQHDMx5GlEunOlCxyjd1ie3o=', 'Content-Length': '0'} 2022-09-09T22:38:08.142 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e9398c9b5d99f8d0-00631bc050-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:08.143 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:08.144 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:08.144 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:08.145 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:08.145 DEBUG:boto:Method: GET 2022-09-09T22:38:08.146 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:08.146 DEBUG:boto:Data: 2022-09-09T22:38:08.147 DEBUG:boto:Headers: {} 2022-09-09T22:38:08.148 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:08.149 DEBUG:boto:Port: 8001 2022-09-09T22:38:08.149 DEBUG:boto:Params: {} 2022-09-09T22:38:08.150 DEBUG:boto:Token: None 2022-09-09T22:38:08.151 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:08 GMT /jkcgbj-16/ 2022-09-09T22:38:08.152 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CZlXb1uR9ZButzc8eMYBAEy7tJo= 2022-09-09T22:38:08.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:08 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CZlXb1uR9ZButzc8eMYBAEy7tJo=', 'Content-Length': '0'} 2022-09-09T22:38:08.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023e60c9418f006e6-00631bc050-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:08.159 DEBUG:boto:b'jkcgbj-16obj-test-zone31000falseobj-test-zone32022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:08.160 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:08.161 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:08.161 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:08.162 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:08.163 DEBUG:boto:Method: GET 2022-09-09T22:38:08.163 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:08.164 DEBUG:boto:Data: 2022-09-09T22:38:08.164 DEBUG:boto:Headers: {} 2022-09-09T22:38:08.165 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:08.166 DEBUG:boto:Port: 8000 2022-09-09T22:38:08.166 DEBUG:boto:Params: {} 2022-09-09T22:38:08.167 DEBUG:boto:Token: None 2022-09-09T22:38:08.168 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:08 GMT /jkcgbj-16/ 2022-09-09T22:38:08.169 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CZlXb1uR9ZButzc8eMYBAEy7tJo= 2022-09-09T22:38:08.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:08 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CZlXb1uR9ZButzc8eMYBAEy7tJo=', 'Content-Length': '0'} 2022-09-09T22:38:08.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6511c1794d4dd14-00631bc050-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:08.176 DEBUG:boto:b'jkcgbj-16obj-test-zone31000falseobj-test-zone32022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:08.177 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:08.178 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:08.178 DEBUG:boto:Method: HEAD 2022-09-09T22:38:08.179 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3 2022-09-09T22:38:08.179 DEBUG:boto:Data: 2022-09-09T22:38:08.180 DEBUG:boto:Headers: {} 2022-09-09T22:38:08.180 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:08.181 DEBUG:boto:Port: 8001 2022-09-09T22:38:08.181 DEBUG:boto:Params: {} 2022-09-09T22:38:08.182 DEBUG:boto:Token: None 2022-09-09T22:38:08.183 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:08 GMT /jkcgbj-16/obj-test-zone3 2022-09-09T22:38:08.183 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:tqBJvWaISo5mj5naOEsXsUg34yw= 2022-09-09T22:38:08.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:08 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:tqBJvWaISo5mj5naOEsXsUg34yw=', 'Content-Length': '0'} 2022-09-09T22:38:08.188 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084e951174ff28985-00631bc050-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:08.189 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:08.189 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:08.190 DEBUG:boto:Method: HEAD 2022-09-09T22:38:08.190 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3 2022-09-09T22:38:08.191 DEBUG:boto:Data: 2022-09-09T22:38:08.192 DEBUG:boto:Headers: {} 2022-09-09T22:38:08.192 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:08.193 DEBUG:boto:Port: 8000 2022-09-09T22:38:08.194 DEBUG:boto:Params: {} 2022-09-09T22:38:08.195 DEBUG:boto:Token: None 2022-09-09T22:38:08.196 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:08 GMT /jkcgbj-16/obj-test-zone3 2022-09-09T22:38:08.197 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:tqBJvWaISo5mj5naOEsXsUg34yw= 2022-09-09T22:38:08.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:08 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:tqBJvWaISo5mj5naOEsXsUg34yw=', 'Content-Length': '0'} 2022-09-09T22:38:08.202 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007bdde7f05bd6e451-00631bc050-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:08.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:38:08.204 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:08.204 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:08.205 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:08.205 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:08.206 DEBUG:boto:Method: GET 2022-09-09T22:38:08.207 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:08.207 DEBUG:boto:Data: 2022-09-09T22:38:08.208 DEBUG:boto:Headers: {} 2022-09-09T22:38:08.208 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:08.209 DEBUG:boto:Port: 8001 2022-09-09T22:38:08.210 DEBUG:boto:Params: {} 2022-09-09T22:38:08.210 DEBUG:boto:Token: None 2022-09-09T22:38:08.211 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:08 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:08.212 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ybCaQHDMx5GlEunOlCxyjd1ie3o= 2022-09-09T22:38:08.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:08 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ybCaQHDMx5GlEunOlCxyjd1ie3o=', 'Content-Length': '0'} 2022-09-09T22:38:08.215 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027194c2bbf4af279-00631bc050-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:08.262 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:08.263 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:08.264 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:08.264 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:08.265 DEBUG:boto:Method: GET 2022-09-09T22:38:08.265 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:08.266 DEBUG:boto:Data: 2022-09-09T22:38:08.267 DEBUG:boto:Headers: {} 2022-09-09T22:38:08.267 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:08.268 DEBUG:boto:Port: 8000 2022-09-09T22:38:08.268 DEBUG:boto:Params: {} 2022-09-09T22:38:08.269 DEBUG:boto:Token: None 2022-09-09T22:38:08.270 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:08 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:38:08.270 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ybCaQHDMx5GlEunOlCxyjd1ie3o= 2022-09-09T22:38:08.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:08 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ybCaQHDMx5GlEunOlCxyjd1ie3o=', 'Content-Length': '0'} 2022-09-09T22:38:08.274 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d96b33eab4b7a66-00631bc050-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:08.275 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:08.276 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:08.276 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:08.277 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:08.277 DEBUG:boto:Method: GET 2022-09-09T22:38:08.278 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:08.278 DEBUG:boto:Data: 2022-09-09T22:38:08.279 DEBUG:boto:Headers: {} 2022-09-09T22:38:08.280 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:08.280 DEBUG:boto:Port: 8001 2022-09-09T22:38:08.281 DEBUG:boto:Params: {} 2022-09-09T22:38:08.282 DEBUG:boto:Token: None 2022-09-09T22:38:08.283 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:08 GMT /jkcgbj-16/ 2022-09-09T22:38:08.284 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CZlXb1uR9ZButzc8eMYBAEy7tJo= 2022-09-09T22:38:08.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:08 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CZlXb1uR9ZButzc8eMYBAEy7tJo=', 'Content-Length': '0'} 2022-09-09T22:38:08.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004527b7a75076d8cd-00631bc050-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:08.291 DEBUG:boto:b'jkcgbj-16obj-test-zone31000falseobj-test-zone32022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:08.291 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:08.292 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:08.293 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:08.293 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:08.294 DEBUG:boto:Method: GET 2022-09-09T22:38:08.294 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:38:08.295 DEBUG:boto:Data: 2022-09-09T22:38:08.296 DEBUG:boto:Headers: {} 2022-09-09T22:38:08.296 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:08.297 DEBUG:boto:Port: 8000 2022-09-09T22:38:08.297 DEBUG:boto:Params: {} 2022-09-09T22:38:08.298 DEBUG:boto:Token: None 2022-09-09T22:38:08.298 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:08 GMT /jkcgbj-16/ 2022-09-09T22:38:08.299 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CZlXb1uR9ZButzc8eMYBAEy7tJo= 2022-09-09T22:38:08.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:08 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CZlXb1uR9ZButzc8eMYBAEy7tJo=', 'Content-Length': '0'} 2022-09-09T22:38:08.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7694c85e69c80a6-00631bc050-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:08.306 DEBUG:boto:b'jkcgbj-16obj-test-zone31000falseobj-test-zone32022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:08.307 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-16 zones={test-zone3, test-zone2} 2022-09-09T22:38:08.308 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jkcgbj-17 2022-09-09T22:38:08.308 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:08.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:08.310 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:38:08.365 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:38:08.366 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:38:08.435 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:38:08.428+0000 7f281e0dedc0 1 bucket sync caught up with source: 2022-09-09T22:38:08.435 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000010.31.5, , 00000000003.10.9, , 00000000007.37.9, , , , , ] 2022-09-09T22:38:08.436 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000010.31.5, , 00000000003.10.9, , 00000000007.37.9, , , , , ] 2022-09-09T22:38:08.436 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:38:08.428+0000 7f281e0dedc0 0 bucket checkpoint complete 2022-09-09T22:38:08.463 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jkcgbj-17 2022-09-09T22:38:08.464 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:08.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:08.466 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:38:08.521 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:38:08.521 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:38:08.587 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:38:08.581+0000 7fa59e7c1dc0 1 bucket sync caught up with source: 2022-09-09T22:38:08.587 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000010.31.5, , 00000000003.10.9, , 00000000007.37.9, , , , , ] 2022-09-09T22:38:08.587 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000010.31.5, , 00000000003.10.9, , 00000000007.37.9, , , , , ] 2022-09-09T22:38:08.588 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:38:08.581+0000 7fa59e7c1dc0 0 bucket checkpoint complete 2022-09-09T22:38:08.612 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jkcgbj-17 2022-09-09T22:38:08.613 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:08.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:08.614 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:38:08.671 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:38:08.671 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:38:08.744 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:38:08.740+0000 7f13e673fdc0 1 bucket sync caught up with source: 2022-09-09T22:38:08.745 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000010.31.5, , 00000000003.10.9, , 00000000007.37.9, , , , , ] 2022-09-09T22:38:08.745 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000010.31.5, , 00000000003.10.9, , 00000000007.37.9, , , , , ] 2022-09-09T22:38:08.745 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:38:08.740+0000 7f13e673fdc0 0 bucket checkpoint complete 2022-09-09T22:38:08.771 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jkcgbj-17 2022-09-09T22:38:08.772 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:08.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:08.773 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:38:08.827 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:38:08.827 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:38:08.897 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:38:08.893+0000 7f72c015adc0 1 bucket sync caught up with source: 2022-09-09T22:38:08.897 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000003.36.9, , 00000000010.39.5, , 00000000008.38.9, , , , , ] 2022-09-09T22:38:08.897 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000003.36.9, , 00000000010.39.5, , 00000000008.38.9, , , , , ] 2022-09-09T22:38:08.898 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:38:08.893+0000 7f72c015adc0 0 bucket checkpoint complete 2022-09-09T22:38:08.924 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jkcgbj-17 2022-09-09T22:38:08.925 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:08.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:08.926 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:38:08.979 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:38:08.979 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:38:09.049 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:38:09.045+0000 7fb952583dc0 1 bucket sync caught up with source: 2022-09-09T22:38:09.049 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000003.36.9, , 00000000010.39.5, , 00000000008.38.9, , , , , ] 2022-09-09T22:38:09.049 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000003.36.9, , 00000000010.39.5, , 00000000008.38.9, , , , , ] 2022-09-09T22:38:09.050 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:38:09.045+0000 7fb952583dc0 0 bucket checkpoint complete 2022-09-09T22:38:09.078 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jkcgbj-17 2022-09-09T22:38:09.079 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jkcgbj-17 2022-09-09T22:38:09.079 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:09.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:09.080 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:38:09.134 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:38:09.134 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:38:09.200 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:38:09.193+0000 7f1518341dc0 1 bucket sync caught up with source: 2022-09-09T22:38:09.200 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000003.50.9, , 00000000003.23.9, , 00000000014.45.5, , , , , ] 2022-09-09T22:38:09.200 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000003.50.9, , 00000000003.23.9, , 00000000014.45.5, , , , , ] 2022-09-09T22:38:09.201 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:38:09.193+0000 7f1518341dc0 0 bucket checkpoint complete 2022-09-09T22:38:09.227 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jkcgbj-17 2022-09-09T22:38:09.228 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:09.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:09.230 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:38:09.285 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:38:09.286 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:38:09.365 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:38:09.360+0000 7f0a5dc6edc0 1 bucket sync caught up with source: 2022-09-09T22:38:09.365 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000003.50.9, , 00000000003.23.9, , 00000000014.45.5, , , , , ] 2022-09-09T22:38:09.365 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000003.50.9, , 00000000003.23.9, , 00000000014.45.5, , , , , ] 2022-09-09T22:38:09.366 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:38:09.360+0000 7f0a5dc6edc0 0 bucket checkpoint complete 2022-09-09T22:38:09.396 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jkcgbj-17 2022-09-09T22:38:09.396 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-17 zones={test-zone2, test-zone1} 2022-09-09T22:38:09.397 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:09.397 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:09.398 DEBUG:boto:Method: HEAD 2022-09-09T22:38:09.399 DEBUG:boto:Path: /jkcgbj-17/ 2022-09-09T22:38:09.399 DEBUG:boto:Data: 2022-09-09T22:38:09.400 DEBUG:boto:Headers: {} 2022-09-09T22:38:09.400 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:09.401 DEBUG:boto:Port: 8000 2022-09-09T22:38:09.401 DEBUG:boto:Params: {} 2022-09-09T22:38:09.402 DEBUG:boto:Token: None 2022-09-09T22:38:09.402 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:09 GMT /jkcgbj-17/ 2022-09-09T22:38:09.403 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:b15rtDdyOfqftv6DN0YXmXrmsxs= 2022-09-09T22:38:09.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:b15rtDdyOfqftv6DN0YXmXrmsxs=', 'Content-Length': '0'} 2022-09-09T22:38:09.408 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002af3c8f2aa4cdcf3-00631bc051-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:09.409 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:09.409 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:09.410 DEBUG:boto:Method: HEAD 2022-09-09T22:38:09.410 DEBUG:boto:Path: /jkcgbj-17/ 2022-09-09T22:38:09.411 DEBUG:boto:Data: 2022-09-09T22:38:09.411 DEBUG:boto:Headers: {} 2022-09-09T22:38:09.412 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:09.412 DEBUG:boto:Port: 8000 2022-09-09T22:38:09.413 DEBUG:boto:Params: {} 2022-09-09T22:38:09.413 DEBUG:boto:Token: None 2022-09-09T22:38:09.414 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:09 GMT /jkcgbj-17/ 2022-09-09T22:38:09.415 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:b15rtDdyOfqftv6DN0YXmXrmsxs= 2022-09-09T22:38:09.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:b15rtDdyOfqftv6DN0YXmXrmsxs=', 'Content-Length': '0'} 2022-09-09T22:38:09.422 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ec35166571f6985-00631bc051-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:09.423 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:38:09.423 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:09.424 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:09.424 DEBUG:boto:path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:09.425 DEBUG:boto:auth_path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:09.425 DEBUG:boto:Method: GET 2022-09-09T22:38:09.426 DEBUG:boto:Path: /jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:09.427 DEBUG:boto:Data: 2022-09-09T22:38:09.427 DEBUG:boto:Headers: {} 2022-09-09T22:38:09.428 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:09.429 DEBUG:boto:Port: 8000 2022-09-09T22:38:09.430 DEBUG:boto:Params: {} 2022-09-09T22:38:09.430 DEBUG:boto:Token: None 2022-09-09T22:38:09.431 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:09 GMT /jkcgbj-17/?versions 2022-09-09T22:38:09.432 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:hrxKCrFNE115mMDVO+MEFoDH2U0= 2022-09-09T22:38:09.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:hrxKCrFNE115mMDVO+MEFoDH2U0=', 'Content-Length': '0'} 2022-09-09T22:38:09.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000153f838524a96ae3-00631bc051-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:09.439 DEBUG:boto:b'jkcgbj-17999falsedummyrcXzglYDyJxE.ArBmS3k944dyN-s9ZTtrue2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1uPhFM-4pEeqszoz0jfwkNTqs81x9txytrue2022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone217lUOlN45NLK2DSLxrYnqbOAg6jm2Tgtrue2022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone31RXW239HrAmSBD6KjNRB3qRTyY29OqZtrue2022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:09.440 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:38:09.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:38:09.441 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:38:09.441 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:38:09.442 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:38:09.442 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:09.443 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:09.443 DEBUG:boto:path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:09.444 DEBUG:boto:auth_path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:09.444 DEBUG:boto:Method: GET 2022-09-09T22:38:09.445 DEBUG:boto:Path: /jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:09.445 DEBUG:boto:Data: 2022-09-09T22:38:09.446 DEBUG:boto:Headers: {} 2022-09-09T22:38:09.446 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:09.447 DEBUG:boto:Port: 8000 2022-09-09T22:38:09.448 DEBUG:boto:Params: {} 2022-09-09T22:38:09.448 DEBUG:boto:Token: None 2022-09-09T22:38:09.449 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:09 GMT /jkcgbj-17/?versions 2022-09-09T22:38:09.449 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:hrxKCrFNE115mMDVO+MEFoDH2U0= 2022-09-09T22:38:09.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:hrxKCrFNE115mMDVO+MEFoDH2U0=', 'Content-Length': '0'} 2022-09-09T22:38:09.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b8ffc8425d83051-00631bc051-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:09.457 DEBUG:boto:b'jkcgbj-17999falsedummyrcXzglYDyJxE.ArBmS3k944dyN-s9ZTtrue2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1uPhFM-4pEeqszoz0jfwkNTqs81x9txytrue2022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone217lUOlN45NLK2DSLxrYnqbOAg6jm2Tgtrue2022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone31RXW239HrAmSBD6KjNRB3qRTyY29OqZtrue2022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:09.458 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:38:09.459 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:38:09.459 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:38:09.460 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:38:09.460 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:09.461 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:09.461 DEBUG:boto:path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:09.462 DEBUG:boto:auth_path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:09.462 DEBUG:boto:Method: GET 2022-09-09T22:38:09.463 DEBUG:boto:Path: /jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:09.463 DEBUG:boto:Data: 2022-09-09T22:38:09.464 DEBUG:boto:Headers: {} 2022-09-09T22:38:09.464 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:09.465 DEBUG:boto:Port: 8000 2022-09-09T22:38:09.465 DEBUG:boto:Params: {} 2022-09-09T22:38:09.466 DEBUG:boto:Token: None 2022-09-09T22:38:09.466 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:09 GMT /jkcgbj-17/?versions 2022-09-09T22:38:09.467 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:hrxKCrFNE115mMDVO+MEFoDH2U0= 2022-09-09T22:38:09.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:hrxKCrFNE115mMDVO+MEFoDH2U0=', 'Content-Length': '0'} 2022-09-09T22:38:09.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050feff6f666d3e68-00631bc051-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:09.474 DEBUG:boto:b'jkcgbj-17999falsedummyrcXzglYDyJxE.ArBmS3k944dyN-s9ZTtrue2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1uPhFM-4pEeqszoz0jfwkNTqs81x9txytrue2022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone217lUOlN45NLK2DSLxrYnqbOAg6jm2Tgtrue2022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone31RXW239HrAmSBD6KjNRB3qRTyY29OqZtrue2022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:09.475 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:09.476 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:09.477 DEBUG:boto:path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:09.477 DEBUG:boto:auth_path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:09.478 DEBUG:boto:Method: GET 2022-09-09T22:38:09.479 DEBUG:boto:Path: /jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:09.479 DEBUG:boto:Data: 2022-09-09T22:38:09.480 DEBUG:boto:Headers: {} 2022-09-09T22:38:09.480 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:09.481 DEBUG:boto:Port: 8000 2022-09-09T22:38:09.482 DEBUG:boto:Params: {} 2022-09-09T22:38:09.482 DEBUG:boto:Token: None 2022-09-09T22:38:09.483 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:09 GMT /jkcgbj-17/?versions 2022-09-09T22:38:09.484 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:hrxKCrFNE115mMDVO+MEFoDH2U0= 2022-09-09T22:38:09.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:hrxKCrFNE115mMDVO+MEFoDH2U0=', 'Content-Length': '0'} 2022-09-09T22:38:09.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b17358235179ca9-00631bc051-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:09.491 DEBUG:boto:b'jkcgbj-17999falsedummyrcXzglYDyJxE.ArBmS3k944dyN-s9ZTtrue2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1uPhFM-4pEeqszoz0jfwkNTqs81x9txytrue2022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone217lUOlN45NLK2DSLxrYnqbOAg6jm2Tgtrue2022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone31RXW239HrAmSBD6KjNRB3qRTyY29OqZtrue2022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:09.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:38:09.493 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:38:09.494 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:38:09.495 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.496 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.497 DEBUG:boto:Method: GET 2022-09-09T22:38:09.498 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.499 DEBUG:boto:Data: 2022-09-09T22:38:09.500 DEBUG:boto:Headers: {} 2022-09-09T22:38:09.501 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:09.502 DEBUG:boto:Port: 8000 2022-09-09T22:38:09.503 DEBUG:boto:Params: {} 2022-09-09T22:38:09.505 DEBUG:boto:Token: None 2022-09-09T22:38:09.506 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:09 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.507 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0FpsjjMGPmsuzGLbAVNpAvag4To= 2022-09-09T22:38:09.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0FpsjjMGPmsuzGLbAVNpAvag4To=', 'Content-Length': '0'} 2022-09-09T22:38:09.512 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000062816e67b4ea8db3-00631bc051-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:09.513 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:38:09.514 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:38:09.515 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.516 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.517 DEBUG:boto:Method: GET 2022-09-09T22:38:09.518 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.519 DEBUG:boto:Data: 2022-09-09T22:38:09.520 DEBUG:boto:Headers: {} 2022-09-09T22:38:09.521 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:09.523 DEBUG:boto:Port: 8000 2022-09-09T22:38:09.524 DEBUG:boto:Params: {} 2022-09-09T22:38:09.525 DEBUG:boto:Token: None 2022-09-09T22:38:09.525 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:09 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.527 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0FpsjjMGPmsuzGLbAVNpAvag4To= 2022-09-09T22:38:09.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0FpsjjMGPmsuzGLbAVNpAvag4To=', 'Content-Length': '0'} 2022-09-09T22:38:09.531 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000156b022c730bbdfb-00631bc051-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:09.532 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:38:09.533 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:38:09.534 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.535 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.536 DEBUG:boto:Method: HEAD 2022-09-09T22:38:09.537 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.537 DEBUG:boto:Data: 2022-09-09T22:38:09.538 DEBUG:boto:Headers: {} 2022-09-09T22:38:09.539 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:09.540 DEBUG:boto:Port: 8000 2022-09-09T22:38:09.541 DEBUG:boto:Params: {} 2022-09-09T22:38:09.541 DEBUG:boto:Token: None 2022-09-09T22:38:09.542 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:09 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.542 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R4C/z2ZpvNWeyjnlphpnzp4DWqM= 2022-09-09T22:38:09.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R4C/z2ZpvNWeyjnlphpnzp4DWqM=', 'Content-Length': '0'} 2022-09-09T22:38:09.546 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cfd66d644df9723a-00631bc051-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:09.547 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:38:09.548 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:38:09.549 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.549 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.550 DEBUG:boto:Method: HEAD 2022-09-09T22:38:09.550 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.551 DEBUG:boto:Data: 2022-09-09T22:38:09.551 DEBUG:boto:Headers: {} 2022-09-09T22:38:09.552 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:09.552 DEBUG:boto:Port: 8000 2022-09-09T22:38:09.553 DEBUG:boto:Params: {} 2022-09-09T22:38:09.553 DEBUG:boto:Token: None 2022-09-09T22:38:09.554 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:09 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.555 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R4C/z2ZpvNWeyjnlphpnzp4DWqM= 2022-09-09T22:38:09.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R4C/z2ZpvNWeyjnlphpnzp4DWqM=', 'Content-Length': '0'} 2022-09-09T22:38:09.559 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000040ae7d2ffc80c72b-00631bc051-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:09.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:38:09.561 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:38:09.561 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:38:09.562 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.563 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.563 DEBUG:boto:Method: GET 2022-09-09T22:38:09.564 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.564 DEBUG:boto:Data: 2022-09-09T22:38:09.565 DEBUG:boto:Headers: {} 2022-09-09T22:38:09.566 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:09.566 DEBUG:boto:Port: 8000 2022-09-09T22:38:09.567 DEBUG:boto:Params: {} 2022-09-09T22:38:09.567 DEBUG:boto:Token: None 2022-09-09T22:38:09.568 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:09 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.569 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0FpsjjMGPmsuzGLbAVNpAvag4To= 2022-09-09T22:38:09.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0FpsjjMGPmsuzGLbAVNpAvag4To=', 'Content-Length': '0'} 2022-09-09T22:38:09.573 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000097ce1b3cd9867de3-00631bc051-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:09.573 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:38:09.574 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:38:09.575 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.575 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.576 DEBUG:boto:Method: GET 2022-09-09T22:38:09.576 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.577 DEBUG:boto:Data: 2022-09-09T22:38:09.577 DEBUG:boto:Headers: {} 2022-09-09T22:38:09.578 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:09.579 DEBUG:boto:Port: 8000 2022-09-09T22:38:09.579 DEBUG:boto:Params: {} 2022-09-09T22:38:09.580 DEBUG:boto:Token: None 2022-09-09T22:38:09.580 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:09 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.581 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0FpsjjMGPmsuzGLbAVNpAvag4To= 2022-09-09T22:38:09.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0FpsjjMGPmsuzGLbAVNpAvag4To=', 'Content-Length': '0'} 2022-09-09T22:38:09.585 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000073deafcc725635b5-00631bc051-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:09.585 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:09.586 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:09.587 DEBUG:boto:path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:38:09.587 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:38:09.588 DEBUG:boto:Method: GET 2022-09-09T22:38:09.588 DEBUG:boto:Path: /jkcgbj-17/?prefix=dummy 2022-09-09T22:38:09.589 DEBUG:boto:Data: 2022-09-09T22:38:09.589 DEBUG:boto:Headers: {} 2022-09-09T22:38:09.590 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:09.590 DEBUG:boto:Port: 8000 2022-09-09T22:38:09.591 DEBUG:boto:Params: {} 2022-09-09T22:38:09.592 DEBUG:boto:Token: None 2022-09-09T22:38:09.592 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:09 GMT /jkcgbj-17/ 2022-09-09T22:38:09.593 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:5yNMZLGVMusi1Efi9ULQlLHZj3Y= 2022-09-09T22:38:09.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:5yNMZLGVMusi1Efi9ULQlLHZj3Y=', 'Content-Length': '0'} 2022-09-09T22:38:09.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b949b33ec609d875-00631bc051-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:09.646 DEBUG:boto:b'jkcgbj-17dummy1000falsedummy2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:09.647 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:09.648 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:09.648 DEBUG:boto:path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:38:09.649 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:38:09.649 DEBUG:boto:Method: GET 2022-09-09T22:38:09.650 DEBUG:boto:Path: /jkcgbj-17/?prefix=dummy 2022-09-09T22:38:09.650 DEBUG:boto:Data: 2022-09-09T22:38:09.651 DEBUG:boto:Headers: {} 2022-09-09T22:38:09.652 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:09.652 DEBUG:boto:Port: 8000 2022-09-09T22:38:09.653 DEBUG:boto:Params: {} 2022-09-09T22:38:09.653 DEBUG:boto:Token: None 2022-09-09T22:38:09.654 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:09 GMT /jkcgbj-17/ 2022-09-09T22:38:09.655 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:5yNMZLGVMusi1Efi9ULQlLHZj3Y= 2022-09-09T22:38:09.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:5yNMZLGVMusi1Efi9ULQlLHZj3Y=', 'Content-Length': '0'} 2022-09-09T22:38:09.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acc7ccabaf7faa7d-00631bc051-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:09.662 DEBUG:boto:b'jkcgbj-17dummy1000falsedummy2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:09.662 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:38:09.663 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:38:09.663 DEBUG:boto:Method: HEAD 2022-09-09T22:38:09.664 DEBUG:boto:Path: /jkcgbj-17/dummy 2022-09-09T22:38:09.664 DEBUG:boto:Data: 2022-09-09T22:38:09.665 DEBUG:boto:Headers: {} 2022-09-09T22:38:09.665 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:09.666 DEBUG:boto:Port: 8000 2022-09-09T22:38:09.666 DEBUG:boto:Params: {} 2022-09-09T22:38:09.667 DEBUG:boto:Token: None 2022-09-09T22:38:09.668 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:09 GMT /jkcgbj-17/dummy 2022-09-09T22:38:09.669 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:38gg83SbqYzkF2XLQv4HYy2jtwc= 2022-09-09T22:38:09.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:38gg83SbqYzkF2XLQv4HYy2jtwc=', 'Content-Length': '0'} 2022-09-09T22:38:09.673 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007c9a1faae2c3707b-00631bc051-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:09.674 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:38:09.675 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:38:09.676 DEBUG:boto:Method: HEAD 2022-09-09T22:38:09.676 DEBUG:boto:Path: /jkcgbj-17/dummy 2022-09-09T22:38:09.677 DEBUG:boto:Data: 2022-09-09T22:38:09.678 DEBUG:boto:Headers: {} 2022-09-09T22:38:09.679 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:09.680 DEBUG:boto:Port: 8000 2022-09-09T22:38:09.681 DEBUG:boto:Params: {} 2022-09-09T22:38:09.681 DEBUG:boto:Token: None 2022-09-09T22:38:09.682 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:09 GMT /jkcgbj-17/dummy 2022-09-09T22:38:09.683 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:38gg83SbqYzkF2XLQv4HYy2jtwc= 2022-09-09T22:38:09.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:38gg83SbqYzkF2XLQv4HYy2jtwc=', 'Content-Length': '0'} 2022-09-09T22:38:09.687 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000069cfcf4d6d2849ae-00631bc051-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:09.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:38:09.688 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:38:09.689 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:38:09.689 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.690 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.691 DEBUG:boto:Method: GET 2022-09-09T22:38:09.691 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.692 DEBUG:boto:Data: 2022-09-09T22:38:09.692 DEBUG:boto:Headers: {} 2022-09-09T22:38:09.693 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:09.693 DEBUG:boto:Port: 8000 2022-09-09T22:38:09.694 DEBUG:boto:Params: {} 2022-09-09T22:38:09.695 DEBUG:boto:Token: None 2022-09-09T22:38:09.695 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:09 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.696 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0FpsjjMGPmsuzGLbAVNpAvag4To= 2022-09-09T22:38:09.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0FpsjjMGPmsuzGLbAVNpAvag4To=', 'Content-Length': '0'} 2022-09-09T22:38:09.699 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008f8b9e4e70604cf9-00631bc051-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:09.700 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:38:09.701 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:38:09.701 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.702 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.702 DEBUG:boto:Method: GET 2022-09-09T22:38:09.703 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.704 DEBUG:boto:Data: 2022-09-09T22:38:09.704 DEBUG:boto:Headers: {} 2022-09-09T22:38:09.705 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:09.705 DEBUG:boto:Port: 8000 2022-09-09T22:38:09.706 DEBUG:boto:Params: {} 2022-09-09T22:38:09.706 DEBUG:boto:Token: None 2022-09-09T22:38:09.707 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:09 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:09.707 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0FpsjjMGPmsuzGLbAVNpAvag4To= 2022-09-09T22:38:09.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0FpsjjMGPmsuzGLbAVNpAvag4To=', 'Content-Length': '0'} 2022-09-09T22:38:09.711 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003e1d7a918a3ee737-00631bc051-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:09.712 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:09.712 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:09.713 DEBUG:boto:path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:38:09.713 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:38:09.714 DEBUG:boto:Method: GET 2022-09-09T22:38:09.714 DEBUG:boto:Path: /jkcgbj-17/?prefix=dummy 2022-09-09T22:38:09.715 DEBUG:boto:Data: 2022-09-09T22:38:09.715 DEBUG:boto:Headers: {} 2022-09-09T22:38:09.716 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:09.716 DEBUG:boto:Port: 8000 2022-09-09T22:38:09.717 DEBUG:boto:Params: {} 2022-09-09T22:38:09.717 DEBUG:boto:Token: None 2022-09-09T22:38:09.718 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:09 GMT /jkcgbj-17/ 2022-09-09T22:38:09.719 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:5yNMZLGVMusi1Efi9ULQlLHZj3Y= 2022-09-09T22:38:09.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:5yNMZLGVMusi1Efi9ULQlLHZj3Y=', 'Content-Length': '0'} 2022-09-09T22:38:09.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c673089a3ab1516-00631bc051-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:09.725 DEBUG:boto:b'jkcgbj-17dummy1000falsedummy2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:09.726 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:09.727 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:09.727 DEBUG:boto:path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:38:09.728 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:38:09.728 DEBUG:boto:Method: GET 2022-09-09T22:38:09.729 DEBUG:boto:Path: /jkcgbj-17/?prefix=dummy 2022-09-09T22:38:09.730 DEBUG:boto:Data: 2022-09-09T22:38:09.730 DEBUG:boto:Headers: {} 2022-09-09T22:38:09.731 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:09.731 DEBUG:boto:Port: 8000 2022-09-09T22:38:09.732 DEBUG:boto:Params: {} 2022-09-09T22:38:09.732 DEBUG:boto:Token: None 2022-09-09T22:38:09.733 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:09 GMT /jkcgbj-17/ 2022-09-09T22:38:09.733 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:5yNMZLGVMusi1Efi9ULQlLHZj3Y= 2022-09-09T22:38:09.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:5yNMZLGVMusi1Efi9ULQlLHZj3Y=', 'Content-Length': '0'} 2022-09-09T22:38:09.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000249906a462a2803d-00631bc051-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:09.786 DEBUG:boto:b'jkcgbj-17dummy1000falsedummy2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:09.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:38:09.789 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:09.789 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:09.790 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:09.791 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:09.792 DEBUG:boto:Method: GET 2022-09-09T22:38:09.793 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:09.793 DEBUG:boto:Data: 2022-09-09T22:38:09.794 DEBUG:boto:Headers: {} 2022-09-09T22:38:09.795 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:09.795 DEBUG:boto:Port: 8000 2022-09-09T22:38:09.796 DEBUG:boto:Params: {} 2022-09-09T22:38:09.797 DEBUG:boto:Token: None 2022-09-09T22:38:09.798 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:09 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:09.798 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:xHeQXwkkGRtQEcrBBz//tvR5Nmw= 2022-09-09T22:38:09.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:xHeQXwkkGRtQEcrBBz//tvR5Nmw=', 'Content-Length': '0'} 2022-09-09T22:38:09.802 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c51446c358159801-00631bc051-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:09.803 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:09.804 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:09.804 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:09.805 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:09.805 DEBUG:boto:Method: GET 2022-09-09T22:38:09.806 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:09.806 DEBUG:boto:Data: 2022-09-09T22:38:09.807 DEBUG:boto:Headers: {} 2022-09-09T22:38:09.807 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:09.808 DEBUG:boto:Port: 8000 2022-09-09T22:38:09.808 DEBUG:boto:Params: {} 2022-09-09T22:38:09.809 DEBUG:boto:Token: None 2022-09-09T22:38:09.810 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:09 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:09.810 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:xHeQXwkkGRtQEcrBBz//tvR5Nmw= 2022-09-09T22:38:09.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:xHeQXwkkGRtQEcrBBz//tvR5Nmw=', 'Content-Length': '0'} 2022-09-09T22:38:09.814 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000fde3162c980418b-00631bc051-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:09.815 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:09.816 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:09.816 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:09.817 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:09.817 DEBUG:boto:Method: HEAD 2022-09-09T22:38:09.818 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:09.818 DEBUG:boto:Data: 2022-09-09T22:38:09.819 DEBUG:boto:Headers: {} 2022-09-09T22:38:09.819 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:09.820 DEBUG:boto:Port: 8000 2022-09-09T22:38:09.820 DEBUG:boto:Params: {} 2022-09-09T22:38:09.821 DEBUG:boto:Token: None 2022-09-09T22:38:09.821 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:09 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:09.822 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jWlGb9/tUeGBLpLOkNjf2JJ8HWM= 2022-09-09T22:38:09.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jWlGb9/tUeGBLpLOkNjf2JJ8HWM=', 'Content-Length': '0'} 2022-09-09T22:38:09.827 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b45dcb6e22431fe-00631bc051-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:09.828 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:09.828 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:09.829 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:09.829 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:09.830 DEBUG:boto:Method: HEAD 2022-09-09T22:38:09.831 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:09.831 DEBUG:boto:Data: 2022-09-09T22:38:09.832 DEBUG:boto:Headers: {} 2022-09-09T22:38:09.832 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:09.833 DEBUG:boto:Port: 8000 2022-09-09T22:38:09.833 DEBUG:boto:Params: {} 2022-09-09T22:38:09.834 DEBUG:boto:Token: None 2022-09-09T22:38:09.834 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:09 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:09.835 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jWlGb9/tUeGBLpLOkNjf2JJ8HWM= 2022-09-09T22:38:09.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jWlGb9/tUeGBLpLOkNjf2JJ8HWM=', 'Content-Length': '0'} 2022-09-09T22:38:09.838 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000061f0d44a8c4129bd-00631bc051-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:09.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:38:09.840 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:09.841 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:09.841 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:09.842 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:09.843 DEBUG:boto:Method: GET 2022-09-09T22:38:09.843 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:09.844 DEBUG:boto:Data: 2022-09-09T22:38:09.844 DEBUG:boto:Headers: {} 2022-09-09T22:38:09.845 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:09.845 DEBUG:boto:Port: 8000 2022-09-09T22:38:09.846 DEBUG:boto:Params: {} 2022-09-09T22:38:09.846 DEBUG:boto:Token: None 2022-09-09T22:38:09.847 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:09 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:09.848 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:xHeQXwkkGRtQEcrBBz//tvR5Nmw= 2022-09-09T22:38:09.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:xHeQXwkkGRtQEcrBBz//tvR5Nmw=', 'Content-Length': '0'} 2022-09-09T22:38:09.851 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c8fbb05964022504-00631bc051-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:09.898 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:09.899 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:09.899 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:09.900 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:09.900 DEBUG:boto:Method: GET 2022-09-09T22:38:09.901 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:09.901 DEBUG:boto:Data: 2022-09-09T22:38:09.902 DEBUG:boto:Headers: {} 2022-09-09T22:38:09.903 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:09.903 DEBUG:boto:Port: 8000 2022-09-09T22:38:09.904 DEBUG:boto:Params: {} 2022-09-09T22:38:09.904 DEBUG:boto:Token: None 2022-09-09T22:38:09.905 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:09 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:09.905 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:xHeQXwkkGRtQEcrBBz//tvR5Nmw= 2022-09-09T22:38:09.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:xHeQXwkkGRtQEcrBBz//tvR5Nmw=', 'Content-Length': '0'} 2022-09-09T22:38:09.909 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b7bc72628b25d99-00631bc051-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:09.910 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:09.911 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:09.911 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:09.912 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:09.912 DEBUG:boto:Method: GET 2022-09-09T22:38:09.913 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:09.913 DEBUG:boto:Data: 2022-09-09T22:38:09.914 DEBUG:boto:Headers: {} 2022-09-09T22:38:09.914 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:09.915 DEBUG:boto:Port: 8000 2022-09-09T22:38:09.916 DEBUG:boto:Params: {} 2022-09-09T22:38:09.916 DEBUG:boto:Token: None 2022-09-09T22:38:09.917 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:09 GMT /jkcgbj-17/ 2022-09-09T22:38:09.918 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:5yNMZLGVMusi1Efi9ULQlLHZj3Y= 2022-09-09T22:38:09.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:5yNMZLGVMusi1Efi9ULQlLHZj3Y=', 'Content-Length': '0'} 2022-09-09T22:38:09.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a286080d10dfaea7-00631bc051-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:09.925 DEBUG:boto:b'jkcgbj-17obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:09.926 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:09.927 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:09.927 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:09.928 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:09.928 DEBUG:boto:Method: GET 2022-09-09T22:38:09.929 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:09.930 DEBUG:boto:Data: 2022-09-09T22:38:09.930 DEBUG:boto:Headers: {} 2022-09-09T22:38:09.931 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:09.931 DEBUG:boto:Port: 8000 2022-09-09T22:38:09.932 DEBUG:boto:Params: {} 2022-09-09T22:38:09.932 DEBUG:boto:Token: None 2022-09-09T22:38:09.933 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:09 GMT /jkcgbj-17/ 2022-09-09T22:38:09.934 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:5yNMZLGVMusi1Efi9ULQlLHZj3Y= 2022-09-09T22:38:09.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:5yNMZLGVMusi1Efi9ULQlLHZj3Y=', 'Content-Length': '0'} 2022-09-09T22:38:09.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0a8581635c08158-00631bc051-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:09.942 DEBUG:boto:b'jkcgbj-17obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:09.942 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:09.943 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:09.943 DEBUG:boto:Method: HEAD 2022-09-09T22:38:09.944 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1 2022-09-09T22:38:09.944 DEBUG:boto:Data: 2022-09-09T22:38:09.945 DEBUG:boto:Headers: {} 2022-09-09T22:38:09.945 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:09.946 DEBUG:boto:Port: 8000 2022-09-09T22:38:09.946 DEBUG:boto:Params: {} 2022-09-09T22:38:09.947 DEBUG:boto:Token: None 2022-09-09T22:38:09.948 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:09 GMT /jkcgbj-17/obj-test-zone1 2022-09-09T22:38:09.948 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:QN0XNKmZ5OzclAgUQnUzEhvhlVo= 2022-09-09T22:38:09.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:QN0XNKmZ5OzclAgUQnUzEhvhlVo=', 'Content-Length': '0'} 2022-09-09T22:38:09.952 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f7a42b1a979c5fc-00631bc051-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:09.953 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:09.954 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:09.954 DEBUG:boto:Method: HEAD 2022-09-09T22:38:09.955 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1 2022-09-09T22:38:09.955 DEBUG:boto:Data: 2022-09-09T22:38:09.956 DEBUG:boto:Headers: {} 2022-09-09T22:38:09.956 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:09.957 DEBUG:boto:Port: 8000 2022-09-09T22:38:09.957 DEBUG:boto:Params: {} 2022-09-09T22:38:09.958 DEBUG:boto:Token: None 2022-09-09T22:38:09.958 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:09 GMT /jkcgbj-17/obj-test-zone1 2022-09-09T22:38:09.959 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:QN0XNKmZ5OzclAgUQnUzEhvhlVo= 2022-09-09T22:38:09.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:QN0XNKmZ5OzclAgUQnUzEhvhlVo=', 'Content-Length': '0'} 2022-09-09T22:38:09.963 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bcac83c3d14e3160-00631bc051-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:09.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:38:09.965 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:09.965 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:09.966 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:09.966 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:09.967 DEBUG:boto:Method: GET 2022-09-09T22:38:09.967 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:09.968 DEBUG:boto:Data: 2022-09-09T22:38:09.968 DEBUG:boto:Headers: {} 2022-09-09T22:38:09.969 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:09.970 DEBUG:boto:Port: 8000 2022-09-09T22:38:09.970 DEBUG:boto:Params: {} 2022-09-09T22:38:09.971 DEBUG:boto:Token: None 2022-09-09T22:38:09.971 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:09 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:09.972 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:xHeQXwkkGRtQEcrBBz//tvR5Nmw= 2022-09-09T22:38:09.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:xHeQXwkkGRtQEcrBBz//tvR5Nmw=', 'Content-Length': '0'} 2022-09-09T22:38:09.976 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000640841cb240f9487-00631bc051-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.018 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:10.019 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:10.019 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:10.020 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:10.020 DEBUG:boto:Method: GET 2022-09-09T22:38:10.021 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:10.022 DEBUG:boto:Data: 2022-09-09T22:38:10.022 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.023 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:10.023 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.024 DEBUG:boto:Params: {} 2022-09-09T22:38:10.024 DEBUG:boto:Token: None 2022-09-09T22:38:10.025 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:10.025 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:amjMZQ2W3huFt8/ihFVSjeYgHzg= 2022-09-09T22:38:10.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:amjMZQ2W3huFt8/ihFVSjeYgHzg=', 'Content-Length': '0'} 2022-09-09T22:38:10.029 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dea8bda5b5d638dd-00631bc052-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.030 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:10.031 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:10.031 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:10.032 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:10.033 DEBUG:boto:Method: GET 2022-09-09T22:38:10.033 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:10.034 DEBUG:boto:Data: 2022-09-09T22:38:10.034 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.035 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:10.036 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.036 DEBUG:boto:Params: {} 2022-09-09T22:38:10.037 DEBUG:boto:Token: None 2022-09-09T22:38:10.038 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/ 2022-09-09T22:38:10.038 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rs982FgnTmgvOd8OB31USKMSYIA= 2022-09-09T22:38:10.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rs982FgnTmgvOd8OB31USKMSYIA=', 'Content-Length': '0'} 2022-09-09T22:38:10.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004996f015827dfa7-00631bc052-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.046 DEBUG:boto:b'jkcgbj-17obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:10.047 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:10.047 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:10.048 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:10.048 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:10.049 DEBUG:boto:Method: GET 2022-09-09T22:38:10.049 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:10.050 DEBUG:boto:Data: 2022-09-09T22:38:10.050 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.051 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:10.051 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.052 DEBUG:boto:Params: {} 2022-09-09T22:38:10.053 DEBUG:boto:Token: None 2022-09-09T22:38:10.053 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/ 2022-09-09T22:38:10.054 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rs982FgnTmgvOd8OB31USKMSYIA= 2022-09-09T22:38:10.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rs982FgnTmgvOd8OB31USKMSYIA=', 'Content-Length': '0'} 2022-09-09T22:38:10.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdcc7c4f00bc324d-00631bc052-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.061 DEBUG:boto:b'jkcgbj-17obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:10.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:38:10.063 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:10.063 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:10.064 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.064 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.065 DEBUG:boto:Method: GET 2022-09-09T22:38:10.065 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.066 DEBUG:boto:Data: 2022-09-09T22:38:10.066 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.067 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:10.068 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.068 DEBUG:boto:Params: {} 2022-09-09T22:38:10.069 DEBUG:boto:Token: None 2022-09-09T22:38:10.070 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.070 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+y9AvzdUKFMIOlcZImQwSYqHwVs= 2022-09-09T22:38:10.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+y9AvzdUKFMIOlcZImQwSYqHwVs=', 'Content-Length': '0'} 2022-09-09T22:38:10.074 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f75be6918790ea6b-00631bc052-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.075 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:10.075 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:10.076 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.077 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.078 DEBUG:boto:Method: GET 2022-09-09T22:38:10.079 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.080 DEBUG:boto:Data: 2022-09-09T22:38:10.080 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.081 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:10.081 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.082 DEBUG:boto:Params: {} 2022-09-09T22:38:10.083 DEBUG:boto:Token: None 2022-09-09T22:38:10.084 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.084 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+y9AvzdUKFMIOlcZImQwSYqHwVs= 2022-09-09T22:38:10.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+y9AvzdUKFMIOlcZImQwSYqHwVs=', 'Content-Length': '0'} 2022-09-09T22:38:10.089 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049d8c02c42ff630f-00631bc052-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.090 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:10.091 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:10.092 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.092 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.093 DEBUG:boto:Method: HEAD 2022-09-09T22:38:10.093 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.094 DEBUG:boto:Data: 2022-09-09T22:38:10.094 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.095 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:10.095 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.096 DEBUG:boto:Params: {} 2022-09-09T22:38:10.096 DEBUG:boto:Token: None 2022-09-09T22:38:10.097 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.098 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:VJCxhAyUpw1q3C+pEm50Vxjw/UA= 2022-09-09T22:38:10.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:VJCxhAyUpw1q3C+pEm50Vxjw/UA=', 'Content-Length': '0'} 2022-09-09T22:38:10.101 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097fa59c98a52782d-00631bc052-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.102 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:10.102 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:10.103 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.103 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.104 DEBUG:boto:Method: HEAD 2022-09-09T22:38:10.104 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.105 DEBUG:boto:Data: 2022-09-09T22:38:10.106 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.106 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:10.107 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.107 DEBUG:boto:Params: {} 2022-09-09T22:38:10.108 DEBUG:boto:Token: None 2022-09-09T22:38:10.108 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.109 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:VJCxhAyUpw1q3C+pEm50Vxjw/UA= 2022-09-09T22:38:10.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:VJCxhAyUpw1q3C+pEm50Vxjw/UA=', 'Content-Length': '0'} 2022-09-09T22:38:10.113 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000278f06599f9b206-00631bc052-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:38:10.114 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:10.115 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:10.115 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.116 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.116 DEBUG:boto:Method: GET 2022-09-09T22:38:10.117 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.117 DEBUG:boto:Data: 2022-09-09T22:38:10.118 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.118 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:10.119 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.119 DEBUG:boto:Params: {} 2022-09-09T22:38:10.120 DEBUG:boto:Token: None 2022-09-09T22:38:10.121 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.121 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+y9AvzdUKFMIOlcZImQwSYqHwVs= 2022-09-09T22:38:10.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+y9AvzdUKFMIOlcZImQwSYqHwVs=', 'Content-Length': '0'} 2022-09-09T22:38:10.125 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000075fbd19c770f6e00-00631bc052-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.166 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:10.167 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:10.167 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.168 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.168 DEBUG:boto:Method: GET 2022-09-09T22:38:10.169 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.170 DEBUG:boto:Data: 2022-09-09T22:38:10.170 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.171 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:10.171 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.172 DEBUG:boto:Params: {} 2022-09-09T22:38:10.173 DEBUG:boto:Token: None 2022-09-09T22:38:10.173 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.174 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+y9AvzdUKFMIOlcZImQwSYqHwVs= 2022-09-09T22:38:10.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+y9AvzdUKFMIOlcZImQwSYqHwVs=', 'Content-Length': '0'} 2022-09-09T22:38:10.178 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a65efd673323ac9d-00631bc052-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.179 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:10.179 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:10.180 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:10.180 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:10.181 DEBUG:boto:Method: GET 2022-09-09T22:38:10.182 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:10.182 DEBUG:boto:Data: 2022-09-09T22:38:10.183 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.183 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:10.184 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.184 DEBUG:boto:Params: {} 2022-09-09T22:38:10.185 DEBUG:boto:Token: None 2022-09-09T22:38:10.185 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/ 2022-09-09T22:38:10.186 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rs982FgnTmgvOd8OB31USKMSYIA= 2022-09-09T22:38:10.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rs982FgnTmgvOd8OB31USKMSYIA=', 'Content-Length': '0'} 2022-09-09T22:38:10.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026b8e2464d3be53f-00631bc052-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.193 DEBUG:boto:b'jkcgbj-17obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:10.194 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:10.195 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:10.195 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:10.196 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:10.196 DEBUG:boto:Method: GET 2022-09-09T22:38:10.197 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:10.197 DEBUG:boto:Data: 2022-09-09T22:38:10.198 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.198 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:10.199 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.199 DEBUG:boto:Params: {} 2022-09-09T22:38:10.200 DEBUG:boto:Token: None 2022-09-09T22:38:10.201 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/ 2022-09-09T22:38:10.201 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rs982FgnTmgvOd8OB31USKMSYIA= 2022-09-09T22:38:10.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rs982FgnTmgvOd8OB31USKMSYIA=', 'Content-Length': '0'} 2022-09-09T22:38:10.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000968860104c969c38-00631bc052-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.209 DEBUG:boto:b'jkcgbj-17obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:10.210 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:10.210 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:10.211 DEBUG:boto:Method: HEAD 2022-09-09T22:38:10.211 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2 2022-09-09T22:38:10.212 DEBUG:boto:Data: 2022-09-09T22:38:10.213 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.213 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:10.214 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.214 DEBUG:boto:Params: {} 2022-09-09T22:38:10.215 DEBUG:boto:Token: None 2022-09-09T22:38:10.216 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/obj-test-zone2 2022-09-09T22:38:10.216 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:hp3TDy0jeIX39DheAQQb/qjaODA= 2022-09-09T22:38:10.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:hp3TDy0jeIX39DheAQQb/qjaODA=', 'Content-Length': '0'} 2022-09-09T22:38:10.221 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a11ca13138c55d9c-00631bc052-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.222 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:10.223 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:10.223 DEBUG:boto:Method: HEAD 2022-09-09T22:38:10.224 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2 2022-09-09T22:38:10.224 DEBUG:boto:Data: 2022-09-09T22:38:10.225 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.225 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:10.226 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.226 DEBUG:boto:Params: {} 2022-09-09T22:38:10.227 DEBUG:boto:Token: None 2022-09-09T22:38:10.228 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/obj-test-zone2 2022-09-09T22:38:10.229 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:hp3TDy0jeIX39DheAQQb/qjaODA= 2022-09-09T22:38:10.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:hp3TDy0jeIX39DheAQQb/qjaODA=', 'Content-Length': '0'} 2022-09-09T22:38:10.233 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002da3da7e0a2c5488-00631bc052-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:38:10.235 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:10.235 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:10.236 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.236 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.237 DEBUG:boto:Method: GET 2022-09-09T22:38:10.237 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.238 DEBUG:boto:Data: 2022-09-09T22:38:10.239 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.240 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:10.241 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.242 DEBUG:boto:Params: {} 2022-09-09T22:38:10.242 DEBUG:boto:Token: None 2022-09-09T22:38:10.244 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.245 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+y9AvzdUKFMIOlcZImQwSYqHwVs= 2022-09-09T22:38:10.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+y9AvzdUKFMIOlcZImQwSYqHwVs=', 'Content-Length': '0'} 2022-09-09T22:38:10.249 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef22add55efe47a4-00631bc052-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.294 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:10.295 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:10.295 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.296 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.296 DEBUG:boto:Method: GET 2022-09-09T22:38:10.297 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.297 DEBUG:boto:Data: 2022-09-09T22:38:10.298 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.299 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:10.299 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.300 DEBUG:boto:Params: {} 2022-09-09T22:38:10.300 DEBUG:boto:Token: None 2022-09-09T22:38:10.301 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:10.301 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+y9AvzdUKFMIOlcZImQwSYqHwVs= 2022-09-09T22:38:10.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+y9AvzdUKFMIOlcZImQwSYqHwVs=', 'Content-Length': '0'} 2022-09-09T22:38:10.305 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9fc8a0ddda7cd99-00631bc052-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.306 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:10.307 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:10.308 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:10.308 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:10.309 DEBUG:boto:Method: GET 2022-09-09T22:38:10.309 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:10.310 DEBUG:boto:Data: 2022-09-09T22:38:10.310 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.311 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:10.311 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.312 DEBUG:boto:Params: {} 2022-09-09T22:38:10.312 DEBUG:boto:Token: None 2022-09-09T22:38:10.313 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/ 2022-09-09T22:38:10.313 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rs982FgnTmgvOd8OB31USKMSYIA= 2022-09-09T22:38:10.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rs982FgnTmgvOd8OB31USKMSYIA=', 'Content-Length': '0'} 2022-09-09T22:38:10.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b14f4f41d67e7224-00631bc052-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.321 DEBUG:boto:b'jkcgbj-17obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:10.323 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:10.324 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:10.325 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:10.326 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:10.327 DEBUG:boto:Method: GET 2022-09-09T22:38:10.328 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:10.329 DEBUG:boto:Data: 2022-09-09T22:38:10.330 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.331 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:10.332 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.333 DEBUG:boto:Params: {} 2022-09-09T22:38:10.334 DEBUG:boto:Token: None 2022-09-09T22:38:10.335 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/ 2022-09-09T22:38:10.336 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rs982FgnTmgvOd8OB31USKMSYIA= 2022-09-09T22:38:10.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rs982FgnTmgvOd8OB31USKMSYIA=', 'Content-Length': '0'} 2022-09-09T22:38:10.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000477fe6209dbc76d4-00631bc052-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.345 DEBUG:boto:b'jkcgbj-17obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:10.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:38:10.347 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:10.348 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:10.348 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.349 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.350 DEBUG:boto:Method: GET 2022-09-09T22:38:10.351 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.352 DEBUG:boto:Data: 2022-09-09T22:38:10.353 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.354 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:10.355 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.356 DEBUG:boto:Params: {} 2022-09-09T22:38:10.357 DEBUG:boto:Token: None 2022-09-09T22:38:10.359 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.360 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:88x049uqEf0HPhIFKR2GANeOcX0= 2022-09-09T22:38:10.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:88x049uqEf0HPhIFKR2GANeOcX0=', 'Content-Length': '0'} 2022-09-09T22:38:10.365 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000478cea61caacf4ab-00631bc052-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.366 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:10.367 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:10.368 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.369 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.370 DEBUG:boto:Method: GET 2022-09-09T22:38:10.371 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.372 DEBUG:boto:Data: 2022-09-09T22:38:10.373 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.374 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:10.375 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.376 DEBUG:boto:Params: {} 2022-09-09T22:38:10.377 DEBUG:boto:Token: None 2022-09-09T22:38:10.378 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.379 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:88x049uqEf0HPhIFKR2GANeOcX0= 2022-09-09T22:38:10.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:88x049uqEf0HPhIFKR2GANeOcX0=', 'Content-Length': '0'} 2022-09-09T22:38:10.382 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000688bf96e591b1746-00631bc052-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.430 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:10.431 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:10.431 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.432 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.432 DEBUG:boto:Method: HEAD 2022-09-09T22:38:10.433 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.433 DEBUG:boto:Data: 2022-09-09T22:38:10.434 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.434 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:10.435 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.435 DEBUG:boto:Params: {} 2022-09-09T22:38:10.436 DEBUG:boto:Token: None 2022-09-09T22:38:10.437 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.438 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:yehpJnGjQsIf1KmE2keliYCKU4g= 2022-09-09T22:38:10.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:yehpJnGjQsIf1KmE2keliYCKU4g=', 'Content-Length': '0'} 2022-09-09T22:38:10.442 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006bca9b4dd60b855b-00631bc052-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.442 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:10.443 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:10.443 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.444 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.445 DEBUG:boto:Method: HEAD 2022-09-09T22:38:10.445 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.446 DEBUG:boto:Data: 2022-09-09T22:38:10.446 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.447 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:10.447 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.448 DEBUG:boto:Params: {} 2022-09-09T22:38:10.448 DEBUG:boto:Token: None 2022-09-09T22:38:10.449 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.449 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:yehpJnGjQsIf1KmE2keliYCKU4g= 2022-09-09T22:38:10.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:yehpJnGjQsIf1KmE2keliYCKU4g=', 'Content-Length': '0'} 2022-09-09T22:38:10.453 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d0f3f5233d1f1fe-00631bc052-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:38:10.455 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:10.455 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:10.456 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.457 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.457 DEBUG:boto:Method: GET 2022-09-09T22:38:10.458 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.458 DEBUG:boto:Data: 2022-09-09T22:38:10.459 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.459 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:10.460 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.460 DEBUG:boto:Params: {} 2022-09-09T22:38:10.461 DEBUG:boto:Token: None 2022-09-09T22:38:10.462 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.463 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:88x049uqEf0HPhIFKR2GANeOcX0= 2022-09-09T22:38:10.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:88x049uqEf0HPhIFKR2GANeOcX0=', 'Content-Length': '0'} 2022-09-09T22:38:10.467 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054d443e9241d39f5-00631bc052-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.468 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:10.469 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:10.469 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.470 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.471 DEBUG:boto:Method: GET 2022-09-09T22:38:10.472 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.472 DEBUG:boto:Data: 2022-09-09T22:38:10.473 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.474 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:10.474 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.475 DEBUG:boto:Params: {} 2022-09-09T22:38:10.475 DEBUG:boto:Token: None 2022-09-09T22:38:10.476 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.477 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:88x049uqEf0HPhIFKR2GANeOcX0= 2022-09-09T22:38:10.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:88x049uqEf0HPhIFKR2GANeOcX0=', 'Content-Length': '0'} 2022-09-09T22:38:10.481 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000032ac69895fd89585-00631bc052-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.482 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:10.482 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:10.483 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:10.484 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:10.484 DEBUG:boto:Method: GET 2022-09-09T22:38:10.485 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:10.485 DEBUG:boto:Data: 2022-09-09T22:38:10.486 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.486 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:10.487 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.487 DEBUG:boto:Params: {} 2022-09-09T22:38:10.488 DEBUG:boto:Token: None 2022-09-09T22:38:10.489 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/ 2022-09-09T22:38:10.489 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rs982FgnTmgvOd8OB31USKMSYIA= 2022-09-09T22:38:10.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rs982FgnTmgvOd8OB31USKMSYIA=', 'Content-Length': '0'} 2022-09-09T22:38:10.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff2792e901a18a35-00631bc052-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.495 DEBUG:boto:b'jkcgbj-17obj-test-zone31000falseobj-test-zone32022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:10.496 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:10.496 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:10.497 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:10.498 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:10.498 DEBUG:boto:Method: GET 2022-09-09T22:38:10.499 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:10.499 DEBUG:boto:Data: 2022-09-09T22:38:10.500 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.500 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:10.501 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.502 DEBUG:boto:Params: {} 2022-09-09T22:38:10.502 DEBUG:boto:Token: None 2022-09-09T22:38:10.503 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/ 2022-09-09T22:38:10.503 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rs982FgnTmgvOd8OB31USKMSYIA= 2022-09-09T22:38:10.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rs982FgnTmgvOd8OB31USKMSYIA=', 'Content-Length': '0'} 2022-09-09T22:38:10.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5777ffb7b0c6eb5-00631bc052-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.510 DEBUG:boto:b'jkcgbj-17obj-test-zone31000falseobj-test-zone32022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:10.511 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:10.512 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:10.512 DEBUG:boto:Method: HEAD 2022-09-09T22:38:10.513 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3 2022-09-09T22:38:10.513 DEBUG:boto:Data: 2022-09-09T22:38:10.514 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.514 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:10.515 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.515 DEBUG:boto:Params: {} 2022-09-09T22:38:10.516 DEBUG:boto:Token: None 2022-09-09T22:38:10.516 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/obj-test-zone3 2022-09-09T22:38:10.517 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vn7XN53U9K0N6eHojNuAjBYMg5I= 2022-09-09T22:38:10.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vn7XN53U9K0N6eHojNuAjBYMg5I=', 'Content-Length': '0'} 2022-09-09T22:38:10.521 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7dc037e3b6587d9-00631bc052-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.522 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:10.522 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:10.523 DEBUG:boto:Method: HEAD 2022-09-09T22:38:10.523 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3 2022-09-09T22:38:10.524 DEBUG:boto:Data: 2022-09-09T22:38:10.524 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.525 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:10.525 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.526 DEBUG:boto:Params: {} 2022-09-09T22:38:10.526 DEBUG:boto:Token: None 2022-09-09T22:38:10.527 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/obj-test-zone3 2022-09-09T22:38:10.527 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vn7XN53U9K0N6eHojNuAjBYMg5I= 2022-09-09T22:38:10.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vn7XN53U9K0N6eHojNuAjBYMg5I=', 'Content-Length': '0'} 2022-09-09T22:38:10.532 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000023aaad4d26052723-00631bc052-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:38:10.533 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:10.534 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:10.534 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.535 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.535 DEBUG:boto:Method: GET 2022-09-09T22:38:10.536 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.536 DEBUG:boto:Data: 2022-09-09T22:38:10.537 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.537 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:10.538 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.538 DEBUG:boto:Params: {} 2022-09-09T22:38:10.539 DEBUG:boto:Token: None 2022-09-09T22:38:10.540 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.540 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:88x049uqEf0HPhIFKR2GANeOcX0= 2022-09-09T22:38:10.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:88x049uqEf0HPhIFKR2GANeOcX0=', 'Content-Length': '0'} 2022-09-09T22:38:10.544 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000155aa7c5900ee724-00631bc052-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.586 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:10.587 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:10.587 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.588 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.588 DEBUG:boto:Method: GET 2022-09-09T22:38:10.589 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.590 DEBUG:boto:Data: 2022-09-09T22:38:10.590 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.591 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:10.592 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.592 DEBUG:boto:Params: {} 2022-09-09T22:38:10.593 DEBUG:boto:Token: None 2022-09-09T22:38:10.594 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:10.594 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:88x049uqEf0HPhIFKR2GANeOcX0= 2022-09-09T22:38:10.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:88x049uqEf0HPhIFKR2GANeOcX0=', 'Content-Length': '0'} 2022-09-09T22:38:10.598 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e93ce0719276b6af-00631bc052-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.599 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:10.600 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:10.601 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:10.601 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:10.602 DEBUG:boto:Method: GET 2022-09-09T22:38:10.602 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:10.603 DEBUG:boto:Data: 2022-09-09T22:38:10.603 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.604 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:10.604 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.605 DEBUG:boto:Params: {} 2022-09-09T22:38:10.605 DEBUG:boto:Token: None 2022-09-09T22:38:10.606 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/ 2022-09-09T22:38:10.607 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rs982FgnTmgvOd8OB31USKMSYIA= 2022-09-09T22:38:10.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rs982FgnTmgvOd8OB31USKMSYIA=', 'Content-Length': '0'} 2022-09-09T22:38:10.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0d9752c65d56d8c-00631bc052-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.613 DEBUG:boto:b'jkcgbj-17obj-test-zone31000falseobj-test-zone32022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:10.614 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:10.615 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:10.615 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:10.616 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:10.616 DEBUG:boto:Method: GET 2022-09-09T22:38:10.617 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:10.618 DEBUG:boto:Data: 2022-09-09T22:38:10.618 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.619 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:10.619 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.620 DEBUG:boto:Params: {} 2022-09-09T22:38:10.620 DEBUG:boto:Token: None 2022-09-09T22:38:10.621 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/ 2022-09-09T22:38:10.622 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rs982FgnTmgvOd8OB31USKMSYIA= 2022-09-09T22:38:10.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rs982FgnTmgvOd8OB31USKMSYIA=', 'Content-Length': '0'} 2022-09-09T22:38:10.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bfa0f45cc1b03a5-00631bc052-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.670 DEBUG:boto:b'jkcgbj-17obj-test-zone31000falseobj-test-zone32022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:10.671 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-17 zones={test-zone2, test-zone1} 2022-09-09T22:38:10.672 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-17 zones={test-zone3, test-zone1} 2022-09-09T22:38:10.672 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:10.673 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:10.673 DEBUG:boto:Method: HEAD 2022-09-09T22:38:10.674 DEBUG:boto:Path: /jkcgbj-17/ 2022-09-09T22:38:10.674 DEBUG:boto:Data: 2022-09-09T22:38:10.675 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.675 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:10.676 DEBUG:boto:Port: 8001 2022-09-09T22:38:10.676 DEBUG:boto:Params: {} 2022-09-09T22:38:10.677 DEBUG:boto:Token: None 2022-09-09T22:38:10.677 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/ 2022-09-09T22:38:10.678 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ovUHF4S+tAUyZHRBT4KJVF5FQY4= 2022-09-09T22:38:10.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ovUHF4S+tAUyZHRBT4KJVF5FQY4=', 'Content-Length': '0'} 2022-09-09T22:38:10.683 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001d770ad6eef58705-00631bc052-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.684 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:10.684 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:10.685 DEBUG:boto:Method: HEAD 2022-09-09T22:38:10.685 DEBUG:boto:Path: /jkcgbj-17/ 2022-09-09T22:38:10.686 DEBUG:boto:Data: 2022-09-09T22:38:10.686 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.687 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:10.687 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.688 DEBUG:boto:Params: {} 2022-09-09T22:38:10.689 DEBUG:boto:Token: None 2022-09-09T22:38:10.689 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/ 2022-09-09T22:38:10.690 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ovUHF4S+tAUyZHRBT4KJVF5FQY4= 2022-09-09T22:38:10.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ovUHF4S+tAUyZHRBT4KJVF5FQY4=', 'Content-Length': '0'} 2022-09-09T22:38:10.696 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000853e5e3dee434c23-00631bc052-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.696 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:38:10.697 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:10.698 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:10.698 DEBUG:boto:path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:10.699 DEBUG:boto:auth_path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:10.699 DEBUG:boto:Method: GET 2022-09-09T22:38:10.700 DEBUG:boto:Path: /jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:10.700 DEBUG:boto:Data: 2022-09-09T22:38:10.701 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.701 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:10.702 DEBUG:boto:Port: 8001 2022-09-09T22:38:10.702 DEBUG:boto:Params: {} 2022-09-09T22:38:10.703 DEBUG:boto:Token: None 2022-09-09T22:38:10.704 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/?versions 2022-09-09T22:38:10.704 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:HZbiBlR7Ec7BpPsgYlj0sYDAYmU= 2022-09-09T22:38:10.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:HZbiBlR7Ec7BpPsgYlj0sYDAYmU=', 'Content-Length': '0'} 2022-09-09T22:38:10.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006927e811514b7c04-00631bc052-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.711 DEBUG:boto:b'jkcgbj-17999falsedummyrcXzglYDyJxE.ArBmS3k944dyN-s9ZTtrue2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1uPhFM-4pEeqszoz0jfwkNTqs81x9txytrue2022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone217lUOlN45NLK2DSLxrYnqbOAg6jm2Tgtrue2022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone31RXW239HrAmSBD6KjNRB3qRTyY29OqZtrue2022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:10.712 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:38:10.713 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:38:10.713 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:38:10.714 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:38:10.714 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:38:10.715 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:10.715 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:10.716 DEBUG:boto:path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:10.716 DEBUG:boto:auth_path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:10.717 DEBUG:boto:Method: GET 2022-09-09T22:38:10.717 DEBUG:boto:Path: /jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:10.718 DEBUG:boto:Data: 2022-09-09T22:38:10.718 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.719 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:10.719 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.720 DEBUG:boto:Params: {} 2022-09-09T22:38:10.720 DEBUG:boto:Token: None 2022-09-09T22:38:10.721 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/?versions 2022-09-09T22:38:10.721 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:HZbiBlR7Ec7BpPsgYlj0sYDAYmU= 2022-09-09T22:38:10.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:HZbiBlR7Ec7BpPsgYlj0sYDAYmU=', 'Content-Length': '0'} 2022-09-09T22:38:10.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000831720583a3eb371-00631bc052-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.729 DEBUG:boto:b'jkcgbj-17999falsedummyrcXzglYDyJxE.ArBmS3k944dyN-s9ZTtrue2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1uPhFM-4pEeqszoz0jfwkNTqs81x9txytrue2022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone217lUOlN45NLK2DSLxrYnqbOAg6jm2Tgtrue2022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone31RXW239HrAmSBD6KjNRB3qRTyY29OqZtrue2022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:10.730 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:38:10.731 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:38:10.731 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:38:10.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:38:10.732 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:10.733 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:10.733 DEBUG:boto:path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:10.734 DEBUG:boto:auth_path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:10.734 DEBUG:boto:Method: GET 2022-09-09T22:38:10.735 DEBUG:boto:Path: /jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:10.735 DEBUG:boto:Data: 2022-09-09T22:38:10.736 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.736 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:10.737 DEBUG:boto:Port: 8001 2022-09-09T22:38:10.737 DEBUG:boto:Params: {} 2022-09-09T22:38:10.738 DEBUG:boto:Token: None 2022-09-09T22:38:10.738 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/?versions 2022-09-09T22:38:10.739 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:HZbiBlR7Ec7BpPsgYlj0sYDAYmU= 2022-09-09T22:38:10.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:HZbiBlR7Ec7BpPsgYlj0sYDAYmU=', 'Content-Length': '0'} 2022-09-09T22:38:10.745 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000821a4dcc116c31f6-00631bc052-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.757 DEBUG:boto:b'jkcgbj-17999falsedummyrcXzglYDyJxE.ArBmS3k944dyN-s9ZTtrue2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1uPhFM-4pEeqszoz0jfwkNTqs81x9txytrue2022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone217lUOlN45NLK2DSLxrYnqbOAg6jm2Tgtrue2022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone31RXW239HrAmSBD6KjNRB3qRTyY29OqZtrue2022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:10.758 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:10.759 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:10.759 DEBUG:boto:path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:10.759 DEBUG:boto:auth_path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:10.760 DEBUG:boto:Method: GET 2022-09-09T22:38:10.760 DEBUG:boto:Path: /jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:10.761 DEBUG:boto:Data: 2022-09-09T22:38:10.762 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.762 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:10.763 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.763 DEBUG:boto:Params: {} 2022-09-09T22:38:10.764 DEBUG:boto:Token: None 2022-09-09T22:38:10.764 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/?versions 2022-09-09T22:38:10.765 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:HZbiBlR7Ec7BpPsgYlj0sYDAYmU= 2022-09-09T22:38:10.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:HZbiBlR7Ec7BpPsgYlj0sYDAYmU=', 'Content-Length': '0'} 2022-09-09T22:38:10.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007040b930917b14e7-00631bc052-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.773 DEBUG:boto:b'jkcgbj-17999falsedummyrcXzglYDyJxE.ArBmS3k944dyN-s9ZTtrue2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1uPhFM-4pEeqszoz0jfwkNTqs81x9txytrue2022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone217lUOlN45NLK2DSLxrYnqbOAg6jm2Tgtrue2022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone31RXW239HrAmSBD6KjNRB3qRTyY29OqZtrue2022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:10.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:38:10.775 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:38:10.775 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:38:10.776 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.776 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.777 DEBUG:boto:Method: GET 2022-09-09T22:38:10.777 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.778 DEBUG:boto:Data: 2022-09-09T22:38:10.779 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.779 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:10.780 DEBUG:boto:Port: 8001 2022-09-09T22:38:10.780 DEBUG:boto:Params: {} 2022-09-09T22:38:10.781 DEBUG:boto:Token: None 2022-09-09T22:38:10.782 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.783 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:blQUefej0vNl9sv5vLNCEPPwBlo= 2022-09-09T22:38:10.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:blQUefej0vNl9sv5vLNCEPPwBlo=', 'Content-Length': '0'} 2022-09-09T22:38:10.786 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000074ad6bdbf8548184-00631bc052-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.787 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:38:10.788 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:38:10.789 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.789 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.790 DEBUG:boto:Method: GET 2022-09-09T22:38:10.791 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.792 DEBUG:boto:Data: 2022-09-09T22:38:10.792 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.793 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:10.793 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.794 DEBUG:boto:Params: {} 2022-09-09T22:38:10.794 DEBUG:boto:Token: None 2022-09-09T22:38:10.795 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.795 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:blQUefej0vNl9sv5vLNCEPPwBlo= 2022-09-09T22:38:10.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:blQUefej0vNl9sv5vLNCEPPwBlo=', 'Content-Length': '0'} 2022-09-09T22:38:10.799 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000093cf48854c09aa3f-00631bc052-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.800 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:38:10.800 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:38:10.801 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.801 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.802 DEBUG:boto:Method: HEAD 2022-09-09T22:38:10.802 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.803 DEBUG:boto:Data: 2022-09-09T22:38:10.803 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.804 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:10.804 DEBUG:boto:Port: 8001 2022-09-09T22:38:10.805 DEBUG:boto:Params: {} 2022-09-09T22:38:10.805 DEBUG:boto:Token: None 2022-09-09T22:38:10.806 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.807 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YIIA2sKM6Qzy20e5SFe00YvYplI= 2022-09-09T22:38:10.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YIIA2sKM6Qzy20e5SFe00YvYplI=', 'Content-Length': '0'} 2022-09-09T22:38:10.810 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004dd73cfda46eb037-00631bc052-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.811 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:38:10.811 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:38:10.812 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.813 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.813 DEBUG:boto:Method: HEAD 2022-09-09T22:38:10.814 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.814 DEBUG:boto:Data: 2022-09-09T22:38:10.815 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.815 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:10.816 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.816 DEBUG:boto:Params: {} 2022-09-09T22:38:10.817 DEBUG:boto:Token: None 2022-09-09T22:38:10.817 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.818 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YIIA2sKM6Qzy20e5SFe00YvYplI= 2022-09-09T22:38:10.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YIIA2sKM6Qzy20e5SFe00YvYplI=', 'Content-Length': '0'} 2022-09-09T22:38:10.821 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005ef01ffd51c95fc1-00631bc052-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:38:10.823 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:38:10.823 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:38:10.824 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.824 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.825 DEBUG:boto:Method: GET 2022-09-09T22:38:10.826 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.826 DEBUG:boto:Data: 2022-09-09T22:38:10.827 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.827 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:10.828 DEBUG:boto:Port: 8001 2022-09-09T22:38:10.828 DEBUG:boto:Params: {} 2022-09-09T22:38:10.829 DEBUG:boto:Token: None 2022-09-09T22:38:10.829 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.830 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:blQUefej0vNl9sv5vLNCEPPwBlo= 2022-09-09T22:38:10.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:blQUefej0vNl9sv5vLNCEPPwBlo=', 'Content-Length': '0'} 2022-09-09T22:38:10.834 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000c8f8fea9824e7d1-00631bc052-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.835 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:38:10.835 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:38:10.836 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.836 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.837 DEBUG:boto:Method: GET 2022-09-09T22:38:10.837 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.838 DEBUG:boto:Data: 2022-09-09T22:38:10.838 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.839 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:10.839 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.840 DEBUG:boto:Params: {} 2022-09-09T22:38:10.840 DEBUG:boto:Token: None 2022-09-09T22:38:10.841 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.842 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:blQUefej0vNl9sv5vLNCEPPwBlo= 2022-09-09T22:38:10.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:blQUefej0vNl9sv5vLNCEPPwBlo=', 'Content-Length': '0'} 2022-09-09T22:38:10.845 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ca0cf19a07630441-00631bc052-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.846 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:10.847 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:10.847 DEBUG:boto:path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:38:10.848 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:38:10.848 DEBUG:boto:Method: GET 2022-09-09T22:38:10.849 DEBUG:boto:Path: /jkcgbj-17/?prefix=dummy 2022-09-09T22:38:10.849 DEBUG:boto:Data: 2022-09-09T22:38:10.850 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.851 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:10.852 DEBUG:boto:Port: 8001 2022-09-09T22:38:10.852 DEBUG:boto:Params: {} 2022-09-09T22:38:10.853 DEBUG:boto:Token: None 2022-09-09T22:38:10.854 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/ 2022-09-09T22:38:10.854 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rs982FgnTmgvOd8OB31USKMSYIA= 2022-09-09T22:38:10.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rs982FgnTmgvOd8OB31USKMSYIA=', 'Content-Length': '0'} 2022-09-09T22:38:10.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000992454e65bf4e6c1-00631bc052-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.902 DEBUG:boto:b'jkcgbj-17dummy1000falsedummy2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:10.903 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:10.904 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:10.904 DEBUG:boto:path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:38:10.905 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:38:10.905 DEBUG:boto:Method: GET 2022-09-09T22:38:10.906 DEBUG:boto:Path: /jkcgbj-17/?prefix=dummy 2022-09-09T22:38:10.907 DEBUG:boto:Data: 2022-09-09T22:38:10.907 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.908 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:10.908 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.909 DEBUG:boto:Params: {} 2022-09-09T22:38:10.909 DEBUG:boto:Token: None 2022-09-09T22:38:10.910 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/ 2022-09-09T22:38:10.911 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rs982FgnTmgvOd8OB31USKMSYIA= 2022-09-09T22:38:10.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rs982FgnTmgvOd8OB31USKMSYIA=', 'Content-Length': '0'} 2022-09-09T22:38:10.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b89b5028d7050c5-00631bc052-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.918 DEBUG:boto:b'jkcgbj-17dummy1000falsedummy2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:10.919 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:38:10.920 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:38:10.920 DEBUG:boto:Method: HEAD 2022-09-09T22:38:10.921 DEBUG:boto:Path: /jkcgbj-17/dummy 2022-09-09T22:38:10.922 DEBUG:boto:Data: 2022-09-09T22:38:10.922 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.923 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:10.923 DEBUG:boto:Port: 8001 2022-09-09T22:38:10.924 DEBUG:boto:Params: {} 2022-09-09T22:38:10.924 DEBUG:boto:Token: None 2022-09-09T22:38:10.925 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/dummy 2022-09-09T22:38:10.925 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:hXsc6Pu1T1+Sl+jTZMv7318Bx6E= 2022-09-09T22:38:10.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:hXsc6Pu1T1+Sl+jTZMv7318Bx6E=', 'Content-Length': '0'} 2022-09-09T22:38:10.930 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006db7f291d84e8cfd-00631bc052-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.931 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:38:10.931 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:38:10.932 DEBUG:boto:Method: HEAD 2022-09-09T22:38:10.932 DEBUG:boto:Path: /jkcgbj-17/dummy 2022-09-09T22:38:10.933 DEBUG:boto:Data: 2022-09-09T22:38:10.933 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.934 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:10.934 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.935 DEBUG:boto:Params: {} 2022-09-09T22:38:10.935 DEBUG:boto:Token: None 2022-09-09T22:38:10.936 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/dummy 2022-09-09T22:38:10.937 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:hXsc6Pu1T1+Sl+jTZMv7318Bx6E= 2022-09-09T22:38:10.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:hXsc6Pu1T1+Sl+jTZMv7318Bx6E=', 'Content-Length': '0'} 2022-09-09T22:38:10.941 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e9d569e5cce4ced3-00631bc052-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:38:10.943 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:38:10.943 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:38:10.944 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.944 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.945 DEBUG:boto:Method: GET 2022-09-09T22:38:10.945 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.946 DEBUG:boto:Data: 2022-09-09T22:38:10.946 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.947 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:10.947 DEBUG:boto:Port: 8001 2022-09-09T22:38:10.948 DEBUG:boto:Params: {} 2022-09-09T22:38:10.949 DEBUG:boto:Token: None 2022-09-09T22:38:10.949 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.950 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:blQUefej0vNl9sv5vLNCEPPwBlo= 2022-09-09T22:38:10.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:blQUefej0vNl9sv5vLNCEPPwBlo=', 'Content-Length': '0'} 2022-09-09T22:38:10.953 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007b897f7d11fdd9bd-00631bc052-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.954 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:38:10.955 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:38:10.955 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.956 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.957 DEBUG:boto:Method: GET 2022-09-09T22:38:10.958 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.959 DEBUG:boto:Data: 2022-09-09T22:38:10.960 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.961 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:10.962 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.963 DEBUG:boto:Params: {} 2022-09-09T22:38:10.964 DEBUG:boto:Token: None 2022-09-09T22:38:10.964 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:10.965 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:blQUefej0vNl9sv5vLNCEPPwBlo= 2022-09-09T22:38:10.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:blQUefej0vNl9sv5vLNCEPPwBlo=', 'Content-Length': '0'} 2022-09-09T22:38:10.969 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001e28ee5f29d026b1-00631bc052-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.970 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:10.970 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:10.971 DEBUG:boto:path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:38:10.972 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:38:10.972 DEBUG:boto:Method: GET 2022-09-09T22:38:10.973 DEBUG:boto:Path: /jkcgbj-17/?prefix=dummy 2022-09-09T22:38:10.974 DEBUG:boto:Data: 2022-09-09T22:38:10.974 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.975 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:10.975 DEBUG:boto:Port: 8001 2022-09-09T22:38:10.976 DEBUG:boto:Params: {} 2022-09-09T22:38:10.977 DEBUG:boto:Token: None 2022-09-09T22:38:10.977 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/ 2022-09-09T22:38:10.978 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rs982FgnTmgvOd8OB31USKMSYIA= 2022-09-09T22:38:10.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rs982FgnTmgvOd8OB31USKMSYIA=', 'Content-Length': '0'} 2022-09-09T22:38:10.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043bc9cfd18dc2435-00631bc052-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:10.986 DEBUG:boto:b'jkcgbj-17dummy1000falsedummy2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:10.987 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:10.987 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:10.988 DEBUG:boto:path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:38:10.988 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:38:10.989 DEBUG:boto:Method: GET 2022-09-09T22:38:10.989 DEBUG:boto:Path: /jkcgbj-17/?prefix=dummy 2022-09-09T22:38:10.990 DEBUG:boto:Data: 2022-09-09T22:38:10.990 DEBUG:boto:Headers: {} 2022-09-09T22:38:10.991 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:10.992 DEBUG:boto:Port: 8000 2022-09-09T22:38:10.992 DEBUG:boto:Params: {} 2022-09-09T22:38:10.993 DEBUG:boto:Token: None 2022-09-09T22:38:10.993 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:10 GMT /jkcgbj-17/ 2022-09-09T22:38:10.994 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rs982FgnTmgvOd8OB31USKMSYIA= 2022-09-09T22:38:10.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rs982FgnTmgvOd8OB31USKMSYIA=', 'Content-Length': '0'} 2022-09-09T22:38:11.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000694fef53d8949ce6-00631bc052-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.001 DEBUG:boto:b'jkcgbj-17dummy1000falsedummy2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:11.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:38:11.003 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:11.003 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:11.004 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.004 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.005 DEBUG:boto:Method: GET 2022-09-09T22:38:11.005 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.006 DEBUG:boto:Data: 2022-09-09T22:38:11.006 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.007 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:11.007 DEBUG:boto:Port: 8001 2022-09-09T22:38:11.008 DEBUG:boto:Params: {} 2022-09-09T22:38:11.009 DEBUG:boto:Token: None 2022-09-09T22:38:11.009 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.010 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6RcEFKahsUQZELgwg4p8vWA8iYI= 2022-09-09T22:38:11.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6RcEFKahsUQZELgwg4p8vWA8iYI=', 'Content-Length': '0'} 2022-09-09T22:38:11.014 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae734b1b6cc58894-00631bc053-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.015 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:11.015 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:11.016 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.016 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.017 DEBUG:boto:Method: GET 2022-09-09T22:38:11.017 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.018 DEBUG:boto:Data: 2022-09-09T22:38:11.018 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.019 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:11.019 DEBUG:boto:Port: 8000 2022-09-09T22:38:11.020 DEBUG:boto:Params: {} 2022-09-09T22:38:11.021 DEBUG:boto:Token: None 2022-09-09T22:38:11.021 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.022 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6RcEFKahsUQZELgwg4p8vWA8iYI= 2022-09-09T22:38:11.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6RcEFKahsUQZELgwg4p8vWA8iYI=', 'Content-Length': '0'} 2022-09-09T22:38:11.026 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001be786a30ad2b263-00631bc053-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.026 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:11.027 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:11.028 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.028 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.029 DEBUG:boto:Method: HEAD 2022-09-09T22:38:11.029 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.030 DEBUG:boto:Data: 2022-09-09T22:38:11.030 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.031 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:11.031 DEBUG:boto:Port: 8001 2022-09-09T22:38:11.032 DEBUG:boto:Params: {} 2022-09-09T22:38:11.032 DEBUG:boto:Token: None 2022-09-09T22:38:11.033 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.033 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dq5cSi+SaMXFDniOVSyslMaNq6k= 2022-09-09T22:38:11.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dq5cSi+SaMXFDniOVSyslMaNq6k=', 'Content-Length': '0'} 2022-09-09T22:38:11.037 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c7e599b44608e78f-00631bc053-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.038 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:11.038 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:11.039 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.039 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.040 DEBUG:boto:Method: HEAD 2022-09-09T22:38:11.040 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.041 DEBUG:boto:Data: 2022-09-09T22:38:11.041 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.042 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:11.042 DEBUG:boto:Port: 8000 2022-09-09T22:38:11.043 DEBUG:boto:Params: {} 2022-09-09T22:38:11.043 DEBUG:boto:Token: None 2022-09-09T22:38:11.044 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.044 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dq5cSi+SaMXFDniOVSyslMaNq6k= 2022-09-09T22:38:11.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dq5cSi+SaMXFDniOVSyslMaNq6k=', 'Content-Length': '0'} 2022-09-09T22:38:11.048 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e3d6ccb746dc8dbc-00631bc053-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:38:11.049 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:11.050 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:11.050 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.051 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.051 DEBUG:boto:Method: GET 2022-09-09T22:38:11.052 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.052 DEBUG:boto:Data: 2022-09-09T22:38:11.053 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.053 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:11.054 DEBUG:boto:Port: 8001 2022-09-09T22:38:11.054 DEBUG:boto:Params: {} 2022-09-09T22:38:11.055 DEBUG:boto:Token: None 2022-09-09T22:38:11.056 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.056 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6RcEFKahsUQZELgwg4p8vWA8iYI= 2022-09-09T22:38:11.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6RcEFKahsUQZELgwg4p8vWA8iYI=', 'Content-Length': '0'} 2022-09-09T22:38:11.060 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000163ba62a7346203d-00631bc053-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.061 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:11.062 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:11.062 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.063 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.063 DEBUG:boto:Method: GET 2022-09-09T22:38:11.064 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.064 DEBUG:boto:Data: 2022-09-09T22:38:11.065 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.065 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:11.066 DEBUG:boto:Port: 8000 2022-09-09T22:38:11.066 DEBUG:boto:Params: {} 2022-09-09T22:38:11.067 DEBUG:boto:Token: None 2022-09-09T22:38:11.067 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.068 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6RcEFKahsUQZELgwg4p8vWA8iYI= 2022-09-09T22:38:11.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6RcEFKahsUQZELgwg4p8vWA8iYI=', 'Content-Length': '0'} 2022-09-09T22:38:11.072 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079e775182dd527de-00631bc053-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.114 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:11.115 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:11.116 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:11.116 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:11.117 DEBUG:boto:Method: GET 2022-09-09T22:38:11.117 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:11.118 DEBUG:boto:Data: 2022-09-09T22:38:11.118 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.119 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:11.119 DEBUG:boto:Port: 8001 2022-09-09T22:38:11.120 DEBUG:boto:Params: {} 2022-09-09T22:38:11.120 DEBUG:boto:Token: None 2022-09-09T22:38:11.121 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/ 2022-09-09T22:38:11.122 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FBtowTvNdAKB3wWZfPoNpbrqWWg= 2022-09-09T22:38:11.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FBtowTvNdAKB3wWZfPoNpbrqWWg=', 'Content-Length': '0'} 2022-09-09T22:38:11.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5e3ee3696cc7d32-00631bc053-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.130 DEBUG:boto:b'jkcgbj-17obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:11.131 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:11.132 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:11.132 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:11.133 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:11.133 DEBUG:boto:Method: GET 2022-09-09T22:38:11.134 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:11.134 DEBUG:boto:Data: 2022-09-09T22:38:11.135 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.135 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:11.136 DEBUG:boto:Port: 8000 2022-09-09T22:38:11.136 DEBUG:boto:Params: {} 2022-09-09T22:38:11.137 DEBUG:boto:Token: None 2022-09-09T22:38:11.138 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/ 2022-09-09T22:38:11.138 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FBtowTvNdAKB3wWZfPoNpbrqWWg= 2022-09-09T22:38:11.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FBtowTvNdAKB3wWZfPoNpbrqWWg=', 'Content-Length': '0'} 2022-09-09T22:38:11.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a7e0956a7379b91-00631bc053-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.156 DEBUG:boto:b'jkcgbj-17obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:11.157 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:11.158 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:11.159 DEBUG:boto:Method: HEAD 2022-09-09T22:38:11.159 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1 2022-09-09T22:38:11.160 DEBUG:boto:Data: 2022-09-09T22:38:11.161 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.161 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:11.162 DEBUG:boto:Port: 8001 2022-09-09T22:38:11.162 DEBUG:boto:Params: {} 2022-09-09T22:38:11.163 DEBUG:boto:Token: None 2022-09-09T22:38:11.164 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/obj-test-zone1 2022-09-09T22:38:11.164 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R7ZhCyfExms2hNFGNF7kajiN/E8= 2022-09-09T22:38:11.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R7ZhCyfExms2hNFGNF7kajiN/E8=', 'Content-Length': '0'} 2022-09-09T22:38:11.169 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ae158a1357eb0d1-00631bc053-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.170 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:11.171 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:11.171 DEBUG:boto:Method: HEAD 2022-09-09T22:38:11.172 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1 2022-09-09T22:38:11.172 DEBUG:boto:Data: 2022-09-09T22:38:11.173 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.173 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:11.174 DEBUG:boto:Port: 8000 2022-09-09T22:38:11.174 DEBUG:boto:Params: {} 2022-09-09T22:38:11.175 DEBUG:boto:Token: None 2022-09-09T22:38:11.176 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/obj-test-zone1 2022-09-09T22:38:11.176 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R7ZhCyfExms2hNFGNF7kajiN/E8= 2022-09-09T22:38:11.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R7ZhCyfExms2hNFGNF7kajiN/E8=', 'Content-Length': '0'} 2022-09-09T22:38:11.181 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc76a8f672c66d61-00631bc053-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:38:11.183 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:11.184 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:11.184 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.185 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.186 DEBUG:boto:Method: GET 2022-09-09T22:38:11.187 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.188 DEBUG:boto:Data: 2022-09-09T22:38:11.189 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.190 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:11.191 DEBUG:boto:Port: 8001 2022-09-09T22:38:11.192 DEBUG:boto:Params: {} 2022-09-09T22:38:11.193 DEBUG:boto:Token: None 2022-09-09T22:38:11.194 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.195 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6RcEFKahsUQZELgwg4p8vWA8iYI= 2022-09-09T22:38:11.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6RcEFKahsUQZELgwg4p8vWA8iYI=', 'Content-Length': '0'} 2022-09-09T22:38:11.199 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b380c951ca41ed04-00631bc053-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.242 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:11.243 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:11.243 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.244 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.245 DEBUG:boto:Method: GET 2022-09-09T22:38:11.245 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.246 DEBUG:boto:Data: 2022-09-09T22:38:11.247 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.247 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:11.248 DEBUG:boto:Port: 8000 2022-09-09T22:38:11.249 DEBUG:boto:Params: {} 2022-09-09T22:38:11.250 DEBUG:boto:Token: None 2022-09-09T22:38:11.252 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:11.253 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6RcEFKahsUQZELgwg4p8vWA8iYI= 2022-09-09T22:38:11.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6RcEFKahsUQZELgwg4p8vWA8iYI=', 'Content-Length': '0'} 2022-09-09T22:38:11.258 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001586359166354849-00631bc053-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.259 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:11.259 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:11.260 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:11.261 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:11.261 DEBUG:boto:Method: GET 2022-09-09T22:38:11.262 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:11.262 DEBUG:boto:Data: 2022-09-09T22:38:11.263 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.263 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:11.264 DEBUG:boto:Port: 8001 2022-09-09T22:38:11.264 DEBUG:boto:Params: {} 2022-09-09T22:38:11.265 DEBUG:boto:Token: None 2022-09-09T22:38:11.266 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/ 2022-09-09T22:38:11.266 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FBtowTvNdAKB3wWZfPoNpbrqWWg= 2022-09-09T22:38:11.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FBtowTvNdAKB3wWZfPoNpbrqWWg=', 'Content-Length': '0'} 2022-09-09T22:38:11.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000565c5c823a0ae467-00631bc053-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.274 DEBUG:boto:b'jkcgbj-17obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:11.275 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:11.275 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:11.276 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:11.276 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:11.277 DEBUG:boto:Method: GET 2022-09-09T22:38:11.277 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:11.278 DEBUG:boto:Data: 2022-09-09T22:38:11.278 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.279 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:11.279 DEBUG:boto:Port: 8000 2022-09-09T22:38:11.280 DEBUG:boto:Params: {} 2022-09-09T22:38:11.280 DEBUG:boto:Token: None 2022-09-09T22:38:11.281 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/ 2022-09-09T22:38:11.282 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FBtowTvNdAKB3wWZfPoNpbrqWWg= 2022-09-09T22:38:11.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FBtowTvNdAKB3wWZfPoNpbrqWWg=', 'Content-Length': '0'} 2022-09-09T22:38:11.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ffcc4355779e37d-00631bc053-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.330 DEBUG:boto:b'jkcgbj-17obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:11.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:38:11.332 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:11.332 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:11.333 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.333 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.334 DEBUG:boto:Method: GET 2022-09-09T22:38:11.335 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.335 DEBUG:boto:Data: 2022-09-09T22:38:11.336 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.336 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:11.337 DEBUG:boto:Port: 8001 2022-09-09T22:38:11.337 DEBUG:boto:Params: {} 2022-09-09T22:38:11.338 DEBUG:boto:Token: None 2022-09-09T22:38:11.339 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.339 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ZIr5mjL2vXXHGgcQewAAp3hb5FM= 2022-09-09T22:38:11.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ZIr5mjL2vXXHGgcQewAAp3hb5FM=', 'Content-Length': '0'} 2022-09-09T22:38:11.343 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b5c630c77642f7cf-00631bc053-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.344 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:11.345 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:11.345 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.346 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.346 DEBUG:boto:Method: GET 2022-09-09T22:38:11.347 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.347 DEBUG:boto:Data: 2022-09-09T22:38:11.348 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.348 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:11.349 DEBUG:boto:Port: 8000 2022-09-09T22:38:11.349 DEBUG:boto:Params: {} 2022-09-09T22:38:11.350 DEBUG:boto:Token: None 2022-09-09T22:38:11.351 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.352 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ZIr5mjL2vXXHGgcQewAAp3hb5FM= 2022-09-09T22:38:11.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ZIr5mjL2vXXHGgcQewAAp3hb5FM=', 'Content-Length': '0'} 2022-09-09T22:38:11.355 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000272e6cb0150a265c-00631bc053-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.356 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:11.357 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:11.357 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.358 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.358 DEBUG:boto:Method: HEAD 2022-09-09T22:38:11.359 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.359 DEBUG:boto:Data: 2022-09-09T22:38:11.360 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.360 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:11.361 DEBUG:boto:Port: 8001 2022-09-09T22:38:11.361 DEBUG:boto:Params: {} 2022-09-09T22:38:11.362 DEBUG:boto:Token: None 2022-09-09T22:38:11.363 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.363 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:RKoZuDXOygnpZkFCypQMngbkh4k= 2022-09-09T22:38:11.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:RKoZuDXOygnpZkFCypQMngbkh4k=', 'Content-Length': '0'} 2022-09-09T22:38:11.367 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b9c236e4943d9747-00631bc053-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.368 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:11.368 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:11.369 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.369 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.369 DEBUG:boto:Method: HEAD 2022-09-09T22:38:11.370 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.371 DEBUG:boto:Data: 2022-09-09T22:38:11.371 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.372 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:11.372 DEBUG:boto:Port: 8000 2022-09-09T22:38:11.373 DEBUG:boto:Params: {} 2022-09-09T22:38:11.373 DEBUG:boto:Token: None 2022-09-09T22:38:11.374 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.375 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:RKoZuDXOygnpZkFCypQMngbkh4k= 2022-09-09T22:38:11.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:RKoZuDXOygnpZkFCypQMngbkh4k=', 'Content-Length': '0'} 2022-09-09T22:38:11.378 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000940c4ce316e8c604-00631bc053-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:38:11.379 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:11.380 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:11.381 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.381 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.382 DEBUG:boto:Method: GET 2022-09-09T22:38:11.383 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.383 DEBUG:boto:Data: 2022-09-09T22:38:11.384 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.384 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:11.385 DEBUG:boto:Port: 8001 2022-09-09T22:38:11.385 DEBUG:boto:Params: {} 2022-09-09T22:38:11.386 DEBUG:boto:Token: None 2022-09-09T22:38:11.387 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.387 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ZIr5mjL2vXXHGgcQewAAp3hb5FM= 2022-09-09T22:38:11.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ZIr5mjL2vXXHGgcQewAAp3hb5FM=', 'Content-Length': '0'} 2022-09-09T22:38:11.391 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a318006a6d45c9b-00631bc053-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.434 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:11.435 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:11.435 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.436 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.436 DEBUG:boto:Method: GET 2022-09-09T22:38:11.437 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.438 DEBUG:boto:Data: 2022-09-09T22:38:11.438 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.439 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:11.439 DEBUG:boto:Port: 8000 2022-09-09T22:38:11.440 DEBUG:boto:Params: {} 2022-09-09T22:38:11.440 DEBUG:boto:Token: None 2022-09-09T22:38:11.441 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.442 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ZIr5mjL2vXXHGgcQewAAp3hb5FM= 2022-09-09T22:38:11.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ZIr5mjL2vXXHGgcQewAAp3hb5FM=', 'Content-Length': '0'} 2022-09-09T22:38:11.446 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000cba1a48e93f42ce-00631bc053-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.447 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:11.447 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:11.448 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:11.448 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:11.449 DEBUG:boto:Method: GET 2022-09-09T22:38:11.449 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:11.450 DEBUG:boto:Data: 2022-09-09T22:38:11.450 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.451 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:11.451 DEBUG:boto:Port: 8001 2022-09-09T22:38:11.452 DEBUG:boto:Params: {} 2022-09-09T22:38:11.453 DEBUG:boto:Token: None 2022-09-09T22:38:11.453 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/ 2022-09-09T22:38:11.454 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FBtowTvNdAKB3wWZfPoNpbrqWWg= 2022-09-09T22:38:11.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FBtowTvNdAKB3wWZfPoNpbrqWWg=', 'Content-Length': '0'} 2022-09-09T22:38:11.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000190165adf16fc998-00631bc053-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.461 DEBUG:boto:b'jkcgbj-17obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:11.462 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:11.463 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:11.463 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:11.464 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:11.464 DEBUG:boto:Method: GET 2022-09-09T22:38:11.465 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:11.466 DEBUG:boto:Data: 2022-09-09T22:38:11.466 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.467 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:11.467 DEBUG:boto:Port: 8000 2022-09-09T22:38:11.468 DEBUG:boto:Params: {} 2022-09-09T22:38:11.468 DEBUG:boto:Token: None 2022-09-09T22:38:11.469 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/ 2022-09-09T22:38:11.469 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FBtowTvNdAKB3wWZfPoNpbrqWWg= 2022-09-09T22:38:11.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FBtowTvNdAKB3wWZfPoNpbrqWWg=', 'Content-Length': '0'} 2022-09-09T22:38:11.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b065abfe8469b531-00631bc053-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.477 DEBUG:boto:b'jkcgbj-17obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:11.477 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:11.478 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:11.478 DEBUG:boto:Method: HEAD 2022-09-09T22:38:11.479 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2 2022-09-09T22:38:11.479 DEBUG:boto:Data: 2022-09-09T22:38:11.480 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.480 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:11.481 DEBUG:boto:Port: 8001 2022-09-09T22:38:11.481 DEBUG:boto:Params: {} 2022-09-09T22:38:11.482 DEBUG:boto:Token: None 2022-09-09T22:38:11.483 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/obj-test-zone2 2022-09-09T22:38:11.483 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:XpAJ3xDFcWLU05E3+2piEFdFOEQ= 2022-09-09T22:38:11.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:XpAJ3xDFcWLU05E3+2piEFdFOEQ=', 'Content-Length': '0'} 2022-09-09T22:38:11.488 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000534d188107d47454-00631bc053-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.488 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:11.489 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:11.490 DEBUG:boto:Method: HEAD 2022-09-09T22:38:11.490 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2 2022-09-09T22:38:11.491 DEBUG:boto:Data: 2022-09-09T22:38:11.491 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.492 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:11.492 DEBUG:boto:Port: 8000 2022-09-09T22:38:11.493 DEBUG:boto:Params: {} 2022-09-09T22:38:11.493 DEBUG:boto:Token: None 2022-09-09T22:38:11.494 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/obj-test-zone2 2022-09-09T22:38:11.494 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:XpAJ3xDFcWLU05E3+2piEFdFOEQ= 2022-09-09T22:38:11.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:XpAJ3xDFcWLU05E3+2piEFdFOEQ=', 'Content-Length': '0'} 2022-09-09T22:38:11.499 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cac781a43505c5e4-00631bc053-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:38:11.500 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:11.500 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:11.501 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.502 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.502 DEBUG:boto:Method: GET 2022-09-09T22:38:11.503 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.503 DEBUG:boto:Data: 2022-09-09T22:38:11.504 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.504 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:11.505 DEBUG:boto:Port: 8001 2022-09-09T22:38:11.505 DEBUG:boto:Params: {} 2022-09-09T22:38:11.506 DEBUG:boto:Token: None 2022-09-09T22:38:11.507 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.507 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ZIr5mjL2vXXHGgcQewAAp3hb5FM= 2022-09-09T22:38:11.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ZIr5mjL2vXXHGgcQewAAp3hb5FM=', 'Content-Length': '0'} 2022-09-09T22:38:11.511 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e9d12b6b9fb6ef3-00631bc053-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.558 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:11.559 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:11.559 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.560 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.560 DEBUG:boto:Method: GET 2022-09-09T22:38:11.561 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.561 DEBUG:boto:Data: 2022-09-09T22:38:11.562 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.563 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:11.563 DEBUG:boto:Port: 8000 2022-09-09T22:38:11.564 DEBUG:boto:Params: {} 2022-09-09T22:38:11.564 DEBUG:boto:Token: None 2022-09-09T22:38:11.565 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:11.565 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ZIr5mjL2vXXHGgcQewAAp3hb5FM= 2022-09-09T22:38:11.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ZIr5mjL2vXXHGgcQewAAp3hb5FM=', 'Content-Length': '0'} 2022-09-09T22:38:11.569 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ca2cb473515cf32-00631bc053-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.570 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:11.571 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:11.571 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:11.572 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:11.572 DEBUG:boto:Method: GET 2022-09-09T22:38:11.573 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:11.573 DEBUG:boto:Data: 2022-09-09T22:38:11.574 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.574 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:11.575 DEBUG:boto:Port: 8001 2022-09-09T22:38:11.575 DEBUG:boto:Params: {} 2022-09-09T22:38:11.576 DEBUG:boto:Token: None 2022-09-09T22:38:11.576 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/ 2022-09-09T22:38:11.577 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FBtowTvNdAKB3wWZfPoNpbrqWWg= 2022-09-09T22:38:11.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FBtowTvNdAKB3wWZfPoNpbrqWWg=', 'Content-Length': '0'} 2022-09-09T22:38:11.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058f8c93d06e5c47d-00631bc053-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.585 DEBUG:boto:b'jkcgbj-17obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:11.586 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:11.586 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:11.587 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:11.587 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:11.588 DEBUG:boto:Method: GET 2022-09-09T22:38:11.589 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:11.589 DEBUG:boto:Data: 2022-09-09T22:38:11.590 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.590 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:11.591 DEBUG:boto:Port: 8000 2022-09-09T22:38:11.591 DEBUG:boto:Params: {} 2022-09-09T22:38:11.592 DEBUG:boto:Token: None 2022-09-09T22:38:11.592 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/ 2022-09-09T22:38:11.593 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FBtowTvNdAKB3wWZfPoNpbrqWWg= 2022-09-09T22:38:11.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FBtowTvNdAKB3wWZfPoNpbrqWWg=', 'Content-Length': '0'} 2022-09-09T22:38:11.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dea91b08e5bae07b-00631bc053-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.602 DEBUG:boto:b'jkcgbj-17obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:11.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:38:11.604 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:11.604 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:11.605 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.605 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.606 DEBUG:boto:Method: GET 2022-09-09T22:38:11.606 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.607 DEBUG:boto:Data: 2022-09-09T22:38:11.607 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.608 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:11.608 DEBUG:boto:Port: 8001 2022-09-09T22:38:11.609 DEBUG:boto:Params: {} 2022-09-09T22:38:11.609 DEBUG:boto:Token: None 2022-09-09T22:38:11.610 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.611 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:f7VaTPqHiQ74L5nfDJxvhiAAbMY= 2022-09-09T22:38:11.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:f7VaTPqHiQ74L5nfDJxvhiAAbMY=', 'Content-Length': '0'} 2022-09-09T22:38:11.615 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008df90130759886b0-00631bc053-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.616 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:11.616 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:11.617 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.617 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.618 DEBUG:boto:Method: GET 2022-09-09T22:38:11.618 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.619 DEBUG:boto:Data: 2022-09-09T22:38:11.619 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.620 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:11.620 DEBUG:boto:Port: 8000 2022-09-09T22:38:11.621 DEBUG:boto:Params: {} 2022-09-09T22:38:11.621 DEBUG:boto:Token: None 2022-09-09T22:38:11.622 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.623 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:f7VaTPqHiQ74L5nfDJxvhiAAbMY= 2022-09-09T22:38:11.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:f7VaTPqHiQ74L5nfDJxvhiAAbMY=', 'Content-Length': '0'} 2022-09-09T22:38:11.627 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dfc1770caddb86c3-00631bc053-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.627 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:11.628 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:11.628 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.629 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.629 DEBUG:boto:Method: HEAD 2022-09-09T22:38:11.630 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.631 DEBUG:boto:Data: 2022-09-09T22:38:11.631 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.631 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:11.632 DEBUG:boto:Port: 8001 2022-09-09T22:38:11.632 DEBUG:boto:Params: {} 2022-09-09T22:38:11.633 DEBUG:boto:Token: None 2022-09-09T22:38:11.634 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.634 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vSordEYewEIL3hZTfKxCnIki/go= 2022-09-09T22:38:11.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vSordEYewEIL3hZTfKxCnIki/go=', 'Content-Length': '0'} 2022-09-09T22:38:11.638 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac89422d9c8d27d1-00631bc053-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.639 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:11.639 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:11.640 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.640 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.641 DEBUG:boto:Method: HEAD 2022-09-09T22:38:11.641 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.642 DEBUG:boto:Data: 2022-09-09T22:38:11.643 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.643 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:11.644 DEBUG:boto:Port: 8000 2022-09-09T22:38:11.644 DEBUG:boto:Params: {} 2022-09-09T22:38:11.645 DEBUG:boto:Token: None 2022-09-09T22:38:11.645 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.646 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vSordEYewEIL3hZTfKxCnIki/go= 2022-09-09T22:38:11.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vSordEYewEIL3hZTfKxCnIki/go=', 'Content-Length': '0'} 2022-09-09T22:38:11.650 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc2728582cfa6564-00631bc053-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:38:11.651 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:11.652 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:11.652 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.653 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.653 DEBUG:boto:Method: GET 2022-09-09T22:38:11.654 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.654 DEBUG:boto:Data: 2022-09-09T22:38:11.655 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.656 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:11.656 DEBUG:boto:Port: 8001 2022-09-09T22:38:11.657 DEBUG:boto:Params: {} 2022-09-09T22:38:11.657 DEBUG:boto:Token: None 2022-09-09T22:38:11.658 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.659 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:f7VaTPqHiQ74L5nfDJxvhiAAbMY= 2022-09-09T22:38:11.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:f7VaTPqHiQ74L5nfDJxvhiAAbMY=', 'Content-Length': '0'} 2022-09-09T22:38:11.662 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076f6c19ca86910a7-00631bc053-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.706 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:11.707 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:11.707 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.708 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.708 DEBUG:boto:Method: GET 2022-09-09T22:38:11.709 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.710 DEBUG:boto:Data: 2022-09-09T22:38:11.710 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.711 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:11.711 DEBUG:boto:Port: 8000 2022-09-09T22:38:11.712 DEBUG:boto:Params: {} 2022-09-09T22:38:11.713 DEBUG:boto:Token: None 2022-09-09T22:38:11.713 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.714 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:f7VaTPqHiQ74L5nfDJxvhiAAbMY= 2022-09-09T22:38:11.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:f7VaTPqHiQ74L5nfDJxvhiAAbMY=', 'Content-Length': '0'} 2022-09-09T22:38:11.718 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000068406a0298df9f66-00631bc053-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.719 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:11.719 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:11.720 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:11.720 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:11.721 DEBUG:boto:Method: GET 2022-09-09T22:38:11.722 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:11.722 DEBUG:boto:Data: 2022-09-09T22:38:11.723 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.723 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:11.724 DEBUG:boto:Port: 8001 2022-09-09T22:38:11.724 DEBUG:boto:Params: {} 2022-09-09T22:38:11.725 DEBUG:boto:Token: None 2022-09-09T22:38:11.725 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/ 2022-09-09T22:38:11.726 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FBtowTvNdAKB3wWZfPoNpbrqWWg= 2022-09-09T22:38:11.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FBtowTvNdAKB3wWZfPoNpbrqWWg=', 'Content-Length': '0'} 2022-09-09T22:38:11.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000588378445c8d32b0-00631bc053-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.733 DEBUG:boto:b'jkcgbj-17obj-test-zone31000falseobj-test-zone32022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:11.735 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:11.736 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:11.737 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:11.738 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:11.739 DEBUG:boto:Method: GET 2022-09-09T22:38:11.740 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:11.741 DEBUG:boto:Data: 2022-09-09T22:38:11.742 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.743 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:11.744 DEBUG:boto:Port: 8000 2022-09-09T22:38:11.745 DEBUG:boto:Params: {} 2022-09-09T22:38:11.746 DEBUG:boto:Token: None 2022-09-09T22:38:11.747 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/ 2022-09-09T22:38:11.748 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FBtowTvNdAKB3wWZfPoNpbrqWWg= 2022-09-09T22:38:11.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FBtowTvNdAKB3wWZfPoNpbrqWWg=', 'Content-Length': '0'} 2022-09-09T22:38:11.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071de094608ef3d46-00631bc053-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.757 DEBUG:boto:b'jkcgbj-17obj-test-zone31000falseobj-test-zone32022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:11.758 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:11.759 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:11.760 DEBUG:boto:Method: HEAD 2022-09-09T22:38:11.760 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3 2022-09-09T22:38:11.761 DEBUG:boto:Data: 2022-09-09T22:38:11.762 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.763 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:11.764 DEBUG:boto:Port: 8001 2022-09-09T22:38:11.765 DEBUG:boto:Params: {} 2022-09-09T22:38:11.766 DEBUG:boto:Token: None 2022-09-09T22:38:11.767 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/obj-test-zone3 2022-09-09T22:38:11.768 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1frW67VKQmUDwmJMzLhYXRw2vc0= 2022-09-09T22:38:11.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1frW67VKQmUDwmJMzLhYXRw2vc0=', 'Content-Length': '0'} 2022-09-09T22:38:11.774 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001145e4434202d961-00631bc053-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.776 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:11.776 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:11.777 DEBUG:boto:Method: HEAD 2022-09-09T22:38:11.778 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3 2022-09-09T22:38:11.779 DEBUG:boto:Data: 2022-09-09T22:38:11.780 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.781 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:11.782 DEBUG:boto:Port: 8000 2022-09-09T22:38:11.783 DEBUG:boto:Params: {} 2022-09-09T22:38:11.784 DEBUG:boto:Token: None 2022-09-09T22:38:11.785 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/obj-test-zone3 2022-09-09T22:38:11.786 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1frW67VKQmUDwmJMzLhYXRw2vc0= 2022-09-09T22:38:11.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1frW67VKQmUDwmJMzLhYXRw2vc0=', 'Content-Length': '0'} 2022-09-09T22:38:11.792 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2b3f62d1f00e416-00631bc053-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:38:11.794 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:11.795 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:11.796 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.797 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.798 DEBUG:boto:Method: GET 2022-09-09T22:38:11.799 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.800 DEBUG:boto:Data: 2022-09-09T22:38:11.801 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.802 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:11.802 DEBUG:boto:Port: 8001 2022-09-09T22:38:11.803 DEBUG:boto:Params: {} 2022-09-09T22:38:11.803 DEBUG:boto:Token: None 2022-09-09T22:38:11.804 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.805 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:f7VaTPqHiQ74L5nfDJxvhiAAbMY= 2022-09-09T22:38:11.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:f7VaTPqHiQ74L5nfDJxvhiAAbMY=', 'Content-Length': '0'} 2022-09-09T22:38:11.809 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001cce7757fac0e6a2-00631bc053-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.810 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:11.810 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:11.811 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.811 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.812 DEBUG:boto:Method: GET 2022-09-09T22:38:11.813 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.813 DEBUG:boto:Data: 2022-09-09T22:38:11.814 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.814 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:11.815 DEBUG:boto:Port: 8000 2022-09-09T22:38:11.815 DEBUG:boto:Params: {} 2022-09-09T22:38:11.816 DEBUG:boto:Token: None 2022-09-09T22:38:11.816 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:11.817 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:f7VaTPqHiQ74L5nfDJxvhiAAbMY= 2022-09-09T22:38:11.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:f7VaTPqHiQ74L5nfDJxvhiAAbMY=', 'Content-Length': '0'} 2022-09-09T22:38:11.821 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003dcb30dd56be3b35-00631bc053-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.862 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:11.863 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:11.863 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:11.864 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:11.864 DEBUG:boto:Method: GET 2022-09-09T22:38:11.865 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:11.865 DEBUG:boto:Data: 2022-09-09T22:38:11.866 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.867 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:11.867 DEBUG:boto:Port: 8001 2022-09-09T22:38:11.868 DEBUG:boto:Params: {} 2022-09-09T22:38:11.868 DEBUG:boto:Token: None 2022-09-09T22:38:11.869 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/ 2022-09-09T22:38:11.869 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FBtowTvNdAKB3wWZfPoNpbrqWWg= 2022-09-09T22:38:11.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FBtowTvNdAKB3wWZfPoNpbrqWWg=', 'Content-Length': '0'} 2022-09-09T22:38:11.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b59d09acdb5b1a7c-00631bc053-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.876 DEBUG:boto:b'jkcgbj-17obj-test-zone31000falseobj-test-zone32022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:11.877 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:11.877 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:11.878 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:11.878 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:11.879 DEBUG:boto:Method: GET 2022-09-09T22:38:11.879 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:11.880 DEBUG:boto:Data: 2022-09-09T22:38:11.880 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.881 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:11.881 DEBUG:boto:Port: 8000 2022-09-09T22:38:11.882 DEBUG:boto:Params: {} 2022-09-09T22:38:11.882 DEBUG:boto:Token: None 2022-09-09T22:38:11.883 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/ 2022-09-09T22:38:11.884 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FBtowTvNdAKB3wWZfPoNpbrqWWg= 2022-09-09T22:38:11.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FBtowTvNdAKB3wWZfPoNpbrqWWg=', 'Content-Length': '0'} 2022-09-09T22:38:11.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000975707f3c83c8f4f-00631bc053-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.890 DEBUG:boto:b'jkcgbj-17obj-test-zone31000falseobj-test-zone32022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:11.891 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-17 zones={test-zone3, test-zone1} 2022-09-09T22:38:11.892 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-17 zones={test-zone3, test-zone2} 2022-09-09T22:38:11.892 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:11.893 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:11.893 DEBUG:boto:Method: HEAD 2022-09-09T22:38:11.894 DEBUG:boto:Path: /jkcgbj-17/ 2022-09-09T22:38:11.894 DEBUG:boto:Data: 2022-09-09T22:38:11.895 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.895 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:11.896 DEBUG:boto:Port: 8001 2022-09-09T22:38:11.896 DEBUG:boto:Params: {} 2022-09-09T22:38:11.897 DEBUG:boto:Token: None 2022-09-09T22:38:11.897 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/ 2022-09-09T22:38:11.898 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pIQFl90SZwrkDyWZeG/pUfDkCRM= 2022-09-09T22:38:11.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pIQFl90SZwrkDyWZeG/pUfDkCRM=', 'Content-Length': '0'} 2022-09-09T22:38:11.903 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000662cf1e808c9561c-00631bc053-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.904 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:11.904 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:11.905 DEBUG:boto:Method: HEAD 2022-09-09T22:38:11.905 DEBUG:boto:Path: /jkcgbj-17/ 2022-09-09T22:38:11.906 DEBUG:boto:Data: 2022-09-09T22:38:11.906 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.907 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:11.907 DEBUG:boto:Port: 8000 2022-09-09T22:38:11.908 DEBUG:boto:Params: {} 2022-09-09T22:38:11.909 DEBUG:boto:Token: None 2022-09-09T22:38:11.909 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/ 2022-09-09T22:38:11.910 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pIQFl90SZwrkDyWZeG/pUfDkCRM= 2022-09-09T22:38:11.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pIQFl90SZwrkDyWZeG/pUfDkCRM=', 'Content-Length': '0'} 2022-09-09T22:38:11.915 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f5c6f4c7dae0c15-00631bc053-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.916 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:38:11.916 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:11.917 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:11.917 DEBUG:boto:path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:11.918 DEBUG:boto:auth_path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:11.918 DEBUG:boto:Method: GET 2022-09-09T22:38:11.919 DEBUG:boto:Path: /jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:11.919 DEBUG:boto:Data: 2022-09-09T22:38:11.920 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.920 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:11.921 DEBUG:boto:Port: 8001 2022-09-09T22:38:11.921 DEBUG:boto:Params: {} 2022-09-09T22:38:11.922 DEBUG:boto:Token: None 2022-09-09T22:38:11.922 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/?versions 2022-09-09T22:38:11.923 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:BbWtaGFP6UdKwQH7rRzeLMUt2YA= 2022-09-09T22:38:11.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:BbWtaGFP6UdKwQH7rRzeLMUt2YA=', 'Content-Length': '0'} 2022-09-09T22:38:11.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000362d704869984953-00631bc053-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.930 DEBUG:boto:b'jkcgbj-17999falsedummyrcXzglYDyJxE.ArBmS3k944dyN-s9ZTtrue2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1uPhFM-4pEeqszoz0jfwkNTqs81x9txytrue2022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone217lUOlN45NLK2DSLxrYnqbOAg6jm2Tgtrue2022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone31RXW239HrAmSBD6KjNRB3qRTyY29OqZtrue2022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:11.931 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:38:11.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:38:11.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:38:11.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:38:11.933 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:38:11.933 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:11.934 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:11.935 DEBUG:boto:path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:11.935 DEBUG:boto:auth_path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:11.936 DEBUG:boto:Method: GET 2022-09-09T22:38:11.936 DEBUG:boto:Path: /jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:11.936 DEBUG:boto:Data: 2022-09-09T22:38:11.937 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.937 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:11.938 DEBUG:boto:Port: 8000 2022-09-09T22:38:11.939 DEBUG:boto:Params: {} 2022-09-09T22:38:11.939 DEBUG:boto:Token: None 2022-09-09T22:38:11.940 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/?versions 2022-09-09T22:38:11.940 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:BbWtaGFP6UdKwQH7rRzeLMUt2YA= 2022-09-09T22:38:11.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:BbWtaGFP6UdKwQH7rRzeLMUt2YA=', 'Content-Length': '0'} 2022-09-09T22:38:11.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1cf9f8004a54e2a-00631bc053-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.947 DEBUG:boto:b'jkcgbj-17999falsedummyrcXzglYDyJxE.ArBmS3k944dyN-s9ZTtrue2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1uPhFM-4pEeqszoz0jfwkNTqs81x9txytrue2022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone217lUOlN45NLK2DSLxrYnqbOAg6jm2Tgtrue2022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone31RXW239HrAmSBD6KjNRB3qRTyY29OqZtrue2022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:11.948 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:38:11.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:38:11.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:38:11.950 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:38:11.950 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:11.951 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:11.951 DEBUG:boto:path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:11.952 DEBUG:boto:auth_path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:11.952 DEBUG:boto:Method: GET 2022-09-09T22:38:11.953 DEBUG:boto:Path: /jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:11.953 DEBUG:boto:Data: 2022-09-09T22:38:11.954 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.954 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:11.955 DEBUG:boto:Port: 8001 2022-09-09T22:38:11.955 DEBUG:boto:Params: {} 2022-09-09T22:38:11.956 DEBUG:boto:Token: None 2022-09-09T22:38:11.956 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/?versions 2022-09-09T22:38:11.957 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:BbWtaGFP6UdKwQH7rRzeLMUt2YA= 2022-09-09T22:38:11.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:BbWtaGFP6UdKwQH7rRzeLMUt2YA=', 'Content-Length': '0'} 2022-09-09T22:38:11.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bc9019e63eff80f-00631bc053-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.965 DEBUG:boto:b'jkcgbj-17999falsedummyrcXzglYDyJxE.ArBmS3k944dyN-s9ZTtrue2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1uPhFM-4pEeqszoz0jfwkNTqs81x9txytrue2022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone217lUOlN45NLK2DSLxrYnqbOAg6jm2Tgtrue2022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone31RXW239HrAmSBD6KjNRB3qRTyY29OqZtrue2022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:11.966 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:11.967 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:11.967 DEBUG:boto:path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:11.968 DEBUG:boto:auth_path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:11.968 DEBUG:boto:Method: GET 2022-09-09T22:38:11.969 DEBUG:boto:Path: /jkcgbj-17/?versions&max-keys=999 2022-09-09T22:38:11.969 DEBUG:boto:Data: 2022-09-09T22:38:11.970 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.970 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:11.971 DEBUG:boto:Port: 8000 2022-09-09T22:38:11.971 DEBUG:boto:Params: {} 2022-09-09T22:38:11.972 DEBUG:boto:Token: None 2022-09-09T22:38:11.973 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/?versions 2022-09-09T22:38:11.973 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:BbWtaGFP6UdKwQH7rRzeLMUt2YA= 2022-09-09T22:38:11.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:BbWtaGFP6UdKwQH7rRzeLMUt2YA=', 'Content-Length': '0'} 2022-09-09T22:38:11.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056a81abe12485467-00631bc053-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.981 DEBUG:boto:b'jkcgbj-17999falsedummyrcXzglYDyJxE.ArBmS3k944dyN-s9ZTtrue2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1uPhFM-4pEeqszoz0jfwkNTqs81x9txytrue2022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone217lUOlN45NLK2DSLxrYnqbOAg6jm2Tgtrue2022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone31RXW239HrAmSBD6KjNRB3qRTyY29OqZtrue2022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:11.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:38:11.982 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:38:11.983 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:38:11.983 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:11.984 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:11.984 DEBUG:boto:Method: GET 2022-09-09T22:38:11.985 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:11.986 DEBUG:boto:Data: 2022-09-09T22:38:11.986 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.987 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:11.987 DEBUG:boto:Port: 8001 2022-09-09T22:38:11.988 DEBUG:boto:Params: {} 2022-09-09T22:38:11.988 DEBUG:boto:Token: None 2022-09-09T22:38:11.989 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:11 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:11.989 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:938/DHAVtJ1DhSiLvHEN4UoiTQc= 2022-09-09T22:38:11.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:938/DHAVtJ1DhSiLvHEN4UoiTQc=', 'Content-Length': '0'} 2022-09-09T22:38:11.993 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007a799a4ba11d43d7-00631bc053-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:11.994 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:38:11.995 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:38:11.995 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:11.996 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:11.996 DEBUG:boto:Method: GET 2022-09-09T22:38:11.997 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:11.997 DEBUG:boto:Data: 2022-09-09T22:38:11.998 DEBUG:boto:Headers: {} 2022-09-09T22:38:11.999 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:12.000 DEBUG:boto:Port: 8000 2022-09-09T22:38:12.000 DEBUG:boto:Params: {} 2022-09-09T22:38:12.001 DEBUG:boto:Token: None 2022-09-09T22:38:12.001 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:12.002 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4c29PdKEKZZStVplCSHgh+x/3Lw= 2022-09-09T22:38:12.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4c29PdKEKZZStVplCSHgh+x/3Lw=', 'Content-Length': '0'} 2022-09-09T22:38:12.006 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009cf9602e7c50057f-00631bc054-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.006 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:38:12.007 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:38:12.007 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:12.008 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:12.008 DEBUG:boto:Method: HEAD 2022-09-09T22:38:12.009 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:12.009 DEBUG:boto:Data: 2022-09-09T22:38:12.010 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.010 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:12.011 DEBUG:boto:Port: 8001 2022-09-09T22:38:12.012 DEBUG:boto:Params: {} 2022-09-09T22:38:12.012 DEBUG:boto:Token: None 2022-09-09T22:38:12.013 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:12.013 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:xNaBtMobfecOQviuMxwdPdFWFLM= 2022-09-09T22:38:12.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:xNaBtMobfecOQviuMxwdPdFWFLM=', 'Content-Length': '0'} 2022-09-09T22:38:12.017 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000027eaf62745e1c883-00631bc054-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.018 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:38:12.018 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:38:12.019 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:12.019 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:12.020 DEBUG:boto:Method: HEAD 2022-09-09T22:38:12.020 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:12.020 DEBUG:boto:Data: 2022-09-09T22:38:12.021 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.022 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:12.022 DEBUG:boto:Port: 8000 2022-09-09T22:38:12.023 DEBUG:boto:Params: {} 2022-09-09T22:38:12.023 DEBUG:boto:Token: None 2022-09-09T22:38:12.024 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:12.024 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:xNaBtMobfecOQviuMxwdPdFWFLM= 2022-09-09T22:38:12.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:xNaBtMobfecOQviuMxwdPdFWFLM=', 'Content-Length': '0'} 2022-09-09T22:38:12.028 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000084db9df24950d1c5-00631bc054-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:38:12.029 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:38:12.030 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:38:12.030 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:12.031 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:12.032 DEBUG:boto:Method: GET 2022-09-09T22:38:12.032 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:12.033 DEBUG:boto:Data: 2022-09-09T22:38:12.033 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.034 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:12.034 DEBUG:boto:Port: 8001 2022-09-09T22:38:12.035 DEBUG:boto:Params: {} 2022-09-09T22:38:12.036 DEBUG:boto:Token: None 2022-09-09T22:38:12.036 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:12.037 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4c29PdKEKZZStVplCSHgh+x/3Lw= 2022-09-09T22:38:12.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4c29PdKEKZZStVplCSHgh+x/3Lw=', 'Content-Length': '0'} 2022-09-09T22:38:12.041 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006b1f87fb19b475ad-00631bc054-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.042 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:38:12.042 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:38:12.043 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:12.043 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:12.044 DEBUG:boto:Method: GET 2022-09-09T22:38:12.044 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:12.045 DEBUG:boto:Data: 2022-09-09T22:38:12.046 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.046 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:12.047 DEBUG:boto:Port: 8000 2022-09-09T22:38:12.048 DEBUG:boto:Params: {} 2022-09-09T22:38:12.049 DEBUG:boto:Token: None 2022-09-09T22:38:12.049 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:12.050 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4c29PdKEKZZStVplCSHgh+x/3Lw= 2022-09-09T22:38:12.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4c29PdKEKZZStVplCSHgh+x/3Lw=', 'Content-Length': '0'} 2022-09-09T22:38:12.054 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fc1d5d1c6726547a-00631bc054-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.055 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:12.056 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:12.057 DEBUG:boto:path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:38:12.058 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:38:12.059 DEBUG:boto:Method: GET 2022-09-09T22:38:12.060 DEBUG:boto:Path: /jkcgbj-17/?prefix=dummy 2022-09-09T22:38:12.061 DEBUG:boto:Data: 2022-09-09T22:38:12.062 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.063 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:12.064 DEBUG:boto:Port: 8001 2022-09-09T22:38:12.065 DEBUG:boto:Params: {} 2022-09-09T22:38:12.066 DEBUG:boto:Token: None 2022-09-09T22:38:12.067 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/ 2022-09-09T22:38:12.068 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:l84mt4fRZO9GtDJ0oW1tTPZ7yx4= 2022-09-09T22:38:12.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:l84mt4fRZO9GtDJ0oW1tTPZ7yx4=', 'Content-Length': '0'} 2022-09-09T22:38:12.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4265e8135776281-00631bc054-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.118 DEBUG:boto:b'jkcgbj-17dummy1000falsedummy2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:12.119 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:12.120 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:12.120 DEBUG:boto:path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:38:12.121 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:38:12.121 DEBUG:boto:Method: GET 2022-09-09T22:38:12.122 DEBUG:boto:Path: /jkcgbj-17/?prefix=dummy 2022-09-09T22:38:12.122 DEBUG:boto:Data: 2022-09-09T22:38:12.123 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.124 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:12.125 DEBUG:boto:Port: 8000 2022-09-09T22:38:12.126 DEBUG:boto:Params: {} 2022-09-09T22:38:12.127 DEBUG:boto:Token: None 2022-09-09T22:38:12.128 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/ 2022-09-09T22:38:12.129 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:l84mt4fRZO9GtDJ0oW1tTPZ7yx4= 2022-09-09T22:38:12.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:l84mt4fRZO9GtDJ0oW1tTPZ7yx4=', 'Content-Length': '0'} 2022-09-09T22:38:12.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4b969053e5944b8-00631bc054-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.137 DEBUG:boto:b'jkcgbj-17dummy1000falsedummy2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:12.138 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:38:12.138 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:38:12.139 DEBUG:boto:Method: HEAD 2022-09-09T22:38:12.139 DEBUG:boto:Path: /jkcgbj-17/dummy 2022-09-09T22:38:12.140 DEBUG:boto:Data: 2022-09-09T22:38:12.141 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.141 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:12.142 DEBUG:boto:Port: 8001 2022-09-09T22:38:12.142 DEBUG:boto:Params: {} 2022-09-09T22:38:12.143 DEBUG:boto:Token: None 2022-09-09T22:38:12.144 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/dummy 2022-09-09T22:38:12.144 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:p5bEKHbhNuhHMUGkU5BzN6LyK/s= 2022-09-09T22:38:12.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:p5bEKHbhNuhHMUGkU5BzN6LyK/s=', 'Content-Length': '0'} 2022-09-09T22:38:12.149 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000c2c8d57d1353193-00631bc054-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.150 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:38:12.150 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:38:12.151 DEBUG:boto:Method: HEAD 2022-09-09T22:38:12.151 DEBUG:boto:Path: /jkcgbj-17/dummy 2022-09-09T22:38:12.152 DEBUG:boto:Data: 2022-09-09T22:38:12.152 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.153 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:12.153 DEBUG:boto:Port: 8000 2022-09-09T22:38:12.154 DEBUG:boto:Params: {} 2022-09-09T22:38:12.154 DEBUG:boto:Token: None 2022-09-09T22:38:12.155 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/dummy 2022-09-09T22:38:12.155 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:p5bEKHbhNuhHMUGkU5BzN6LyK/s= 2022-09-09T22:38:12.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:p5bEKHbhNuhHMUGkU5BzN6LyK/s=', 'Content-Length': '0'} 2022-09-09T22:38:12.160 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c2693ac60b81a723-00631bc054-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:38:12.161 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:38:12.162 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:38:12.162 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:12.163 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:12.163 DEBUG:boto:Method: GET 2022-09-09T22:38:12.164 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:12.164 DEBUG:boto:Data: 2022-09-09T22:38:12.165 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.166 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:12.167 DEBUG:boto:Port: 8001 2022-09-09T22:38:12.167 DEBUG:boto:Params: {} 2022-09-09T22:38:12.168 DEBUG:boto:Token: None 2022-09-09T22:38:12.168 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:12.169 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4c29PdKEKZZStVplCSHgh+x/3Lw= 2022-09-09T22:38:12.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4c29PdKEKZZStVplCSHgh+x/3Lw=', 'Content-Length': '0'} 2022-09-09T22:38:12.173 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e2d3496e2faa3e98-00631bc054-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.174 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:38:12.174 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:38:12.175 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:12.176 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:12.176 DEBUG:boto:Method: GET 2022-09-09T22:38:12.177 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:12.178 DEBUG:boto:Data: 2022-09-09T22:38:12.178 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.179 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:12.179 DEBUG:boto:Port: 8000 2022-09-09T22:38:12.180 DEBUG:boto:Params: {} 2022-09-09T22:38:12.180 DEBUG:boto:Token: None 2022-09-09T22:38:12.181 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:38:12.181 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4c29PdKEKZZStVplCSHgh+x/3Lw= 2022-09-09T22:38:12.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4c29PdKEKZZStVplCSHgh+x/3Lw=', 'Content-Length': '0'} 2022-09-09T22:38:12.185 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008c3918b3c02759c2-00631bc054-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.186 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:12.186 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:12.187 DEBUG:boto:path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:38:12.187 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:38:12.188 DEBUG:boto:Method: GET 2022-09-09T22:38:12.188 DEBUG:boto:Path: /jkcgbj-17/?prefix=dummy 2022-09-09T22:38:12.189 DEBUG:boto:Data: 2022-09-09T22:38:12.190 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.190 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:12.191 DEBUG:boto:Port: 8001 2022-09-09T22:38:12.191 DEBUG:boto:Params: {} 2022-09-09T22:38:12.192 DEBUG:boto:Token: None 2022-09-09T22:38:12.192 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/ 2022-09-09T22:38:12.193 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:l84mt4fRZO9GtDJ0oW1tTPZ7yx4= 2022-09-09T22:38:12.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:l84mt4fRZO9GtDJ0oW1tTPZ7yx4=', 'Content-Length': '0'} 2022-09-09T22:38:12.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003296b639bed111f4-00631bc054-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.200 DEBUG:boto:b'jkcgbj-17dummy1000falsedummy2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:12.201 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:12.202 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:12.202 DEBUG:boto:path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:38:12.203 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:38:12.204 DEBUG:boto:Method: GET 2022-09-09T22:38:12.204 DEBUG:boto:Path: /jkcgbj-17/?prefix=dummy 2022-09-09T22:38:12.205 DEBUG:boto:Data: 2022-09-09T22:38:12.205 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.206 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:12.206 DEBUG:boto:Port: 8000 2022-09-09T22:38:12.207 DEBUG:boto:Params: {} 2022-09-09T22:38:12.207 DEBUG:boto:Token: None 2022-09-09T22:38:12.208 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/ 2022-09-09T22:38:12.209 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:l84mt4fRZO9GtDJ0oW1tTPZ7yx4= 2022-09-09T22:38:12.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:l84mt4fRZO9GtDJ0oW1tTPZ7yx4=', 'Content-Length': '0'} 2022-09-09T22:38:12.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c96274dc1845baa6-00631bc054-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.217 DEBUG:boto:b'jkcgbj-17dummy1000falsedummy2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:12.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:38:12.219 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:12.220 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:12.221 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.222 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.223 DEBUG:boto:Method: GET 2022-09-09T22:38:12.224 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.225 DEBUG:boto:Data: 2022-09-09T22:38:12.225 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.226 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:12.227 DEBUG:boto:Port: 8001 2022-09-09T22:38:12.228 DEBUG:boto:Params: {} 2022-09-09T22:38:12.229 DEBUG:boto:Token: None 2022-09-09T22:38:12.230 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.231 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:V1T4LJY1mzBJBqEEh2TndsflT0E= 2022-09-09T22:38:12.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:V1T4LJY1mzBJBqEEh2TndsflT0E=', 'Content-Length': '0'} 2022-09-09T22:38:12.236 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016fcfe031b1d4f18-00631bc054-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.278 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:12.279 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:12.280 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.281 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.282 DEBUG:boto:Method: GET 2022-09-09T22:38:12.283 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.284 DEBUG:boto:Data: 2022-09-09T22:38:12.285 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.286 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:12.287 DEBUG:boto:Port: 8000 2022-09-09T22:38:12.288 DEBUG:boto:Params: {} 2022-09-09T22:38:12.288 DEBUG:boto:Token: None 2022-09-09T22:38:12.289 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.290 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:V1T4LJY1mzBJBqEEh2TndsflT0E= 2022-09-09T22:38:12.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:V1T4LJY1mzBJBqEEh2TndsflT0E=', 'Content-Length': '0'} 2022-09-09T22:38:12.294 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000083a330d90f2d7c4b-00631bc054-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.294 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:12.295 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:12.296 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.296 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.297 DEBUG:boto:Method: HEAD 2022-09-09T22:38:12.298 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.299 DEBUG:boto:Data: 2022-09-09T22:38:12.300 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.301 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:12.302 DEBUG:boto:Port: 8001 2022-09-09T22:38:12.303 DEBUG:boto:Params: {} 2022-09-09T22:38:12.304 DEBUG:boto:Token: None 2022-09-09T22:38:12.305 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.306 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:/V6jXGwj0zV52ESpjWq+hDZ7dqA= 2022-09-09T22:38:12.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:/V6jXGwj0zV52ESpjWq+hDZ7dqA=', 'Content-Length': '0'} 2022-09-09T22:38:12.309 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d1e1e95d45e0610-00631bc054-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.310 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:12.311 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:12.311 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.312 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.312 DEBUG:boto:Method: HEAD 2022-09-09T22:38:12.313 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.313 DEBUG:boto:Data: 2022-09-09T22:38:12.314 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.315 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:12.315 DEBUG:boto:Port: 8000 2022-09-09T22:38:12.316 DEBUG:boto:Params: {} 2022-09-09T22:38:12.317 DEBUG:boto:Token: None 2022-09-09T22:38:12.317 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.318 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:/V6jXGwj0zV52ESpjWq+hDZ7dqA= 2022-09-09T22:38:12.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:/V6jXGwj0zV52ESpjWq+hDZ7dqA=', 'Content-Length': '0'} 2022-09-09T22:38:12.322 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000034ee43e8b9a095f1-00631bc054-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:38:12.323 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:12.324 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:12.324 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.325 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.325 DEBUG:boto:Method: GET 2022-09-09T22:38:12.326 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.326 DEBUG:boto:Data: 2022-09-09T22:38:12.327 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.327 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:12.328 DEBUG:boto:Port: 8001 2022-09-09T22:38:12.328 DEBUG:boto:Params: {} 2022-09-09T22:38:12.329 DEBUG:boto:Token: None 2022-09-09T22:38:12.330 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.330 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:V1T4LJY1mzBJBqEEh2TndsflT0E= 2022-09-09T22:38:12.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:V1T4LJY1mzBJBqEEh2TndsflT0E=', 'Content-Length': '0'} 2022-09-09T22:38:12.334 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049a2cebc18db4878-00631bc054-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.335 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:12.335 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:12.336 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.336 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.337 DEBUG:boto:Method: GET 2022-09-09T22:38:12.337 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.338 DEBUG:boto:Data: 2022-09-09T22:38:12.338 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.339 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:12.340 DEBUG:boto:Port: 8000 2022-09-09T22:38:12.340 DEBUG:boto:Params: {} 2022-09-09T22:38:12.341 DEBUG:boto:Token: None 2022-09-09T22:38:12.342 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.342 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:V1T4LJY1mzBJBqEEh2TndsflT0E= 2022-09-09T22:38:12.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:V1T4LJY1mzBJBqEEh2TndsflT0E=', 'Content-Length': '0'} 2022-09-09T22:38:12.346 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004cbfd3b958cc7571-00631bc054-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.347 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:12.348 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:12.348 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:12.349 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:12.349 DEBUG:boto:Method: GET 2022-09-09T22:38:12.350 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:12.350 DEBUG:boto:Data: 2022-09-09T22:38:12.351 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.351 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:12.352 DEBUG:boto:Port: 8001 2022-09-09T22:38:12.352 DEBUG:boto:Params: {} 2022-09-09T22:38:12.353 DEBUG:boto:Token: None 2022-09-09T22:38:12.353 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/ 2022-09-09T22:38:12.354 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:l84mt4fRZO9GtDJ0oW1tTPZ7yx4= 2022-09-09T22:38:12.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:l84mt4fRZO9GtDJ0oW1tTPZ7yx4=', 'Content-Length': '0'} 2022-09-09T22:38:12.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003aa2c3563d0279f2-00631bc054-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.361 DEBUG:boto:b'jkcgbj-17obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:12.362 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:12.362 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:12.363 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:12.363 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:12.364 DEBUG:boto:Method: GET 2022-09-09T22:38:12.364 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:12.365 DEBUG:boto:Data: 2022-09-09T22:38:12.365 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.366 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:12.366 DEBUG:boto:Port: 8000 2022-09-09T22:38:12.367 DEBUG:boto:Params: {} 2022-09-09T22:38:12.367 DEBUG:boto:Token: None 2022-09-09T22:38:12.368 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/ 2022-09-09T22:38:12.369 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:l84mt4fRZO9GtDJ0oW1tTPZ7yx4= 2022-09-09T22:38:12.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:l84mt4fRZO9GtDJ0oW1tTPZ7yx4=', 'Content-Length': '0'} 2022-09-09T22:38:12.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbeebdc0fe207157-00631bc054-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.418 DEBUG:boto:b'jkcgbj-17obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:12.419 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:12.420 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:12.420 DEBUG:boto:Method: HEAD 2022-09-09T22:38:12.421 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1 2022-09-09T22:38:12.421 DEBUG:boto:Data: 2022-09-09T22:38:12.422 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.422 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:12.423 DEBUG:boto:Port: 8001 2022-09-09T22:38:12.423 DEBUG:boto:Params: {} 2022-09-09T22:38:12.424 DEBUG:boto:Token: None 2022-09-09T22:38:12.425 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/obj-test-zone1 2022-09-09T22:38:12.425 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:g5NWe35OWSECapUDjMsQpYhV+Dk= 2022-09-09T22:38:12.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:g5NWe35OWSECapUDjMsQpYhV+Dk=', 'Content-Length': '0'} 2022-09-09T22:38:12.430 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a568bf40ea2800ff-00631bc054-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.431 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:12.432 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:12.433 DEBUG:boto:Method: HEAD 2022-09-09T22:38:12.434 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1 2022-09-09T22:38:12.435 DEBUG:boto:Data: 2022-09-09T22:38:12.435 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.436 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:12.437 DEBUG:boto:Port: 8000 2022-09-09T22:38:12.438 DEBUG:boto:Params: {} 2022-09-09T22:38:12.439 DEBUG:boto:Token: None 2022-09-09T22:38:12.439 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/obj-test-zone1 2022-09-09T22:38:12.440 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:g5NWe35OWSECapUDjMsQpYhV+Dk= 2022-09-09T22:38:12.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:g5NWe35OWSECapUDjMsQpYhV+Dk=', 'Content-Length': '0'} 2022-09-09T22:38:12.445 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af0440edc4774f82-00631bc054-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:38:12.446 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:12.447 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:12.447 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.448 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.448 DEBUG:boto:Method: GET 2022-09-09T22:38:12.449 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.450 DEBUG:boto:Data: 2022-09-09T22:38:12.450 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.451 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:12.451 DEBUG:boto:Port: 8001 2022-09-09T22:38:12.452 DEBUG:boto:Params: {} 2022-09-09T22:38:12.453 DEBUG:boto:Token: None 2022-09-09T22:38:12.453 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.454 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:V1T4LJY1mzBJBqEEh2TndsflT0E= 2022-09-09T22:38:12.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:V1T4LJY1mzBJBqEEh2TndsflT0E=', 'Content-Length': '0'} 2022-09-09T22:38:12.457 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d441bbd97e240394-00631bc054-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.502 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:12.503 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:12.505 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.505 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.506 DEBUG:boto:Method: GET 2022-09-09T22:38:12.506 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.507 DEBUG:boto:Data: 2022-09-09T22:38:12.507 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.508 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:12.508 DEBUG:boto:Port: 8000 2022-09-09T22:38:12.509 DEBUG:boto:Params: {} 2022-09-09T22:38:12.510 DEBUG:boto:Token: None 2022-09-09T22:38:12.510 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:38:12.511 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:V1T4LJY1mzBJBqEEh2TndsflT0E= 2022-09-09T22:38:12.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:V1T4LJY1mzBJBqEEh2TndsflT0E=', 'Content-Length': '0'} 2022-09-09T22:38:12.515 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:01 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e417cd941d3d13e-00631bc054-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.516 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:12.517 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:12.518 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:12.519 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:12.521 DEBUG:boto:Method: GET 2022-09-09T22:38:12.522 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:12.523 DEBUG:boto:Data: 2022-09-09T22:38:12.524 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.525 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:12.526 DEBUG:boto:Port: 8001 2022-09-09T22:38:12.527 DEBUG:boto:Params: {} 2022-09-09T22:38:12.528 DEBUG:boto:Token: None 2022-09-09T22:38:12.529 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/ 2022-09-09T22:38:12.530 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:l84mt4fRZO9GtDJ0oW1tTPZ7yx4= 2022-09-09T22:38:12.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:l84mt4fRZO9GtDJ0oW1tTPZ7yx4=', 'Content-Length': '0'} 2022-09-09T22:38:12.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013c67db4c9b44b41-00631bc054-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.538 DEBUG:boto:b'jkcgbj-17obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:12.539 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:12.540 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:12.541 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:12.542 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:12.543 DEBUG:boto:Method: GET 2022-09-09T22:38:12.544 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:38:12.545 DEBUG:boto:Data: 2022-09-09T22:38:12.545 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.547 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:12.547 DEBUG:boto:Port: 8000 2022-09-09T22:38:12.548 DEBUG:boto:Params: {} 2022-09-09T22:38:12.550 DEBUG:boto:Token: None 2022-09-09T22:38:12.551 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/ 2022-09-09T22:38:12.551 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:l84mt4fRZO9GtDJ0oW1tTPZ7yx4= 2022-09-09T22:38:12.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:l84mt4fRZO9GtDJ0oW1tTPZ7yx4=', 'Content-Length': '0'} 2022-09-09T22:38:12.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a38b5b959628aea8-00631bc054-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.559 DEBUG:boto:b'jkcgbj-17obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:12.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:38:12.561 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:12.562 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:12.563 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.564 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.565 DEBUG:boto:Method: GET 2022-09-09T22:38:12.567 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.568 DEBUG:boto:Data: 2022-09-09T22:38:12.568 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.569 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:12.569 DEBUG:boto:Port: 8001 2022-09-09T22:38:12.570 DEBUG:boto:Params: {} 2022-09-09T22:38:12.570 DEBUG:boto:Token: None 2022-09-09T22:38:12.571 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.572 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:5zdaCEbfgg6agaJhnyFIIV4d6Kk= 2022-09-09T22:38:12.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:5zdaCEbfgg6agaJhnyFIIV4d6Kk=', 'Content-Length': '0'} 2022-09-09T22:38:12.576 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6bda86ee00645b8-00631bc054-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.577 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:12.577 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:12.578 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.579 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.579 DEBUG:boto:Method: GET 2022-09-09T22:38:12.580 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.580 DEBUG:boto:Data: 2022-09-09T22:38:12.581 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.581 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:12.582 DEBUG:boto:Port: 8000 2022-09-09T22:38:12.582 DEBUG:boto:Params: {} 2022-09-09T22:38:12.583 DEBUG:boto:Token: None 2022-09-09T22:38:12.584 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.585 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:5zdaCEbfgg6agaJhnyFIIV4d6Kk= 2022-09-09T22:38:12.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:5zdaCEbfgg6agaJhnyFIIV4d6Kk=', 'Content-Length': '0'} 2022-09-09T22:38:12.590 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000915dc0517e85acb5-00631bc054-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.630 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:12.631 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:12.632 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.633 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.634 DEBUG:boto:Method: HEAD 2022-09-09T22:38:12.635 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.636 DEBUG:boto:Data: 2022-09-09T22:38:12.637 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.638 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:12.639 DEBUG:boto:Port: 8001 2022-09-09T22:38:12.640 DEBUG:boto:Params: {} 2022-09-09T22:38:12.641 DEBUG:boto:Token: None 2022-09-09T22:38:12.642 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.643 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jLZRnEZCSugfz8wejzZT6FeLiyY= 2022-09-09T22:38:12.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jLZRnEZCSugfz8wejzZT6FeLiyY=', 'Content-Length': '0'} 2022-09-09T22:38:12.648 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1749925f6157a32-00631bc054-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.649 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:12.650 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:12.651 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.652 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.653 DEBUG:boto:Method: HEAD 2022-09-09T22:38:12.654 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.655 DEBUG:boto:Data: 2022-09-09T22:38:12.656 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.656 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:12.657 DEBUG:boto:Port: 8000 2022-09-09T22:38:12.657 DEBUG:boto:Params: {} 2022-09-09T22:38:12.658 DEBUG:boto:Token: None 2022-09-09T22:38:12.659 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.659 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jLZRnEZCSugfz8wejzZT6FeLiyY= 2022-09-09T22:38:12.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jLZRnEZCSugfz8wejzZT6FeLiyY=', 'Content-Length': '0'} 2022-09-09T22:38:12.663 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c31844a2d0265713-00631bc054-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:38:12.665 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:12.665 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:12.666 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.667 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.667 DEBUG:boto:Method: GET 2022-09-09T22:38:12.668 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.668 DEBUG:boto:Data: 2022-09-09T22:38:12.669 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.669 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:12.670 DEBUG:boto:Port: 8001 2022-09-09T22:38:12.670 DEBUG:boto:Params: {} 2022-09-09T22:38:12.671 DEBUG:boto:Token: None 2022-09-09T22:38:12.672 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.672 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:5zdaCEbfgg6agaJhnyFIIV4d6Kk= 2022-09-09T22:38:12.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:5zdaCEbfgg6agaJhnyFIIV4d6Kk=', 'Content-Length': '0'} 2022-09-09T22:38:12.676 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086f19ea426081d5d-00631bc054-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.677 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:12.678 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:12.679 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.680 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.680 DEBUG:boto:Method: GET 2022-09-09T22:38:12.681 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.682 DEBUG:boto:Data: 2022-09-09T22:38:12.682 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.683 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:12.684 DEBUG:boto:Port: 8000 2022-09-09T22:38:12.685 DEBUG:boto:Params: {} 2022-09-09T22:38:12.686 DEBUG:boto:Token: None 2022-09-09T22:38:12.688 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.689 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:5zdaCEbfgg6agaJhnyFIIV4d6Kk= 2022-09-09T22:38:12.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:5zdaCEbfgg6agaJhnyFIIV4d6Kk=', 'Content-Length': '0'} 2022-09-09T22:38:12.694 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000593e4ceaad5fdf11-00631bc054-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.695 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:12.696 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:12.697 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:12.698 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:12.699 DEBUG:boto:Method: GET 2022-09-09T22:38:12.700 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:12.701 DEBUG:boto:Data: 2022-09-09T22:38:12.702 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.703 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:12.704 DEBUG:boto:Port: 8001 2022-09-09T22:38:12.705 DEBUG:boto:Params: {} 2022-09-09T22:38:12.706 DEBUG:boto:Token: None 2022-09-09T22:38:12.708 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/ 2022-09-09T22:38:12.709 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:l84mt4fRZO9GtDJ0oW1tTPZ7yx4= 2022-09-09T22:38:12.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:l84mt4fRZO9GtDJ0oW1tTPZ7yx4=', 'Content-Length': '0'} 2022-09-09T22:38:12.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001886813aeb02e6dc-00631bc054-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.719 DEBUG:boto:b'jkcgbj-17obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:12.720 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:12.721 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:12.722 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:12.723 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:12.724 DEBUG:boto:Method: GET 2022-09-09T22:38:12.725 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:12.726 DEBUG:boto:Data: 2022-09-09T22:38:12.727 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.728 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:12.729 DEBUG:boto:Port: 8000 2022-09-09T22:38:12.729 DEBUG:boto:Params: {} 2022-09-09T22:38:12.730 DEBUG:boto:Token: None 2022-09-09T22:38:12.731 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/ 2022-09-09T22:38:12.732 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:l84mt4fRZO9GtDJ0oW1tTPZ7yx4= 2022-09-09T22:38:12.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:l84mt4fRZO9GtDJ0oW1tTPZ7yx4=', 'Content-Length': '0'} 2022-09-09T22:38:12.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e31cc2b55df6a269-00631bc054-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.740 DEBUG:boto:b'jkcgbj-17obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:12.742 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:12.742 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:12.743 DEBUG:boto:Method: HEAD 2022-09-09T22:38:12.743 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2 2022-09-09T22:38:12.744 DEBUG:boto:Data: 2022-09-09T22:38:12.744 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.745 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:12.745 DEBUG:boto:Port: 8001 2022-09-09T22:38:12.746 DEBUG:boto:Params: {} 2022-09-09T22:38:12.747 DEBUG:boto:Token: None 2022-09-09T22:38:12.747 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/obj-test-zone2 2022-09-09T22:38:12.748 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:M1VBmcIHWRotP90wn+sboSaGBAA= 2022-09-09T22:38:12.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:M1VBmcIHWRotP90wn+sboSaGBAA=', 'Content-Length': '0'} 2022-09-09T22:38:12.752 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d7cb368b5dcb260-00631bc054-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.753 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:12.754 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:12.754 DEBUG:boto:Method: HEAD 2022-09-09T22:38:12.755 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2 2022-09-09T22:38:12.755 DEBUG:boto:Data: 2022-09-09T22:38:12.756 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.756 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:12.757 DEBUG:boto:Port: 8000 2022-09-09T22:38:12.757 DEBUG:boto:Params: {} 2022-09-09T22:38:12.758 DEBUG:boto:Token: None 2022-09-09T22:38:12.759 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/obj-test-zone2 2022-09-09T22:38:12.759 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:M1VBmcIHWRotP90wn+sboSaGBAA= 2022-09-09T22:38:12.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:M1VBmcIHWRotP90wn+sboSaGBAA=', 'Content-Length': '0'} 2022-09-09T22:38:12.771 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099858113b1a6f4cc-00631bc054-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:38:12.773 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:12.774 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:12.774 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.775 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.775 DEBUG:boto:Method: GET 2022-09-09T22:38:12.776 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.776 DEBUG:boto:Data: 2022-09-09T22:38:12.777 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.777 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:12.778 DEBUG:boto:Port: 8001 2022-09-09T22:38:12.779 DEBUG:boto:Params: {} 2022-09-09T22:38:12.779 DEBUG:boto:Token: None 2022-09-09T22:38:12.780 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.780 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:5zdaCEbfgg6agaJhnyFIIV4d6Kk= 2022-09-09T22:38:12.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:5zdaCEbfgg6agaJhnyFIIV4d6Kk=', 'Content-Length': '0'} 2022-09-09T22:38:12.785 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6ddcbb33b5d39af-00631bc054-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.827 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:12.827 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:12.828 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.829 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.829 DEBUG:boto:Method: GET 2022-09-09T22:38:12.830 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.831 DEBUG:boto:Data: 2022-09-09T22:38:12.831 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.832 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:12.832 DEBUG:boto:Port: 8000 2022-09-09T22:38:12.833 DEBUG:boto:Params: {} 2022-09-09T22:38:12.833 DEBUG:boto:Token: None 2022-09-09T22:38:12.834 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:38:12.835 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:5zdaCEbfgg6agaJhnyFIIV4d6Kk= 2022-09-09T22:38:12.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:5zdaCEbfgg6agaJhnyFIIV4d6Kk=', 'Content-Length': '0'} 2022-09-09T22:38:12.839 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e08348c59433d204-00631bc054-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.840 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:12.840 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:12.841 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:12.841 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:12.842 DEBUG:boto:Method: GET 2022-09-09T22:38:12.843 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:12.843 DEBUG:boto:Data: 2022-09-09T22:38:12.844 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.844 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:12.845 DEBUG:boto:Port: 8001 2022-09-09T22:38:12.845 DEBUG:boto:Params: {} 2022-09-09T22:38:12.846 DEBUG:boto:Token: None 2022-09-09T22:38:12.847 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/ 2022-09-09T22:38:12.847 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:l84mt4fRZO9GtDJ0oW1tTPZ7yx4= 2022-09-09T22:38:12.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:l84mt4fRZO9GtDJ0oW1tTPZ7yx4=', 'Content-Length': '0'} 2022-09-09T22:38:12.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b70d7ca2d0d6b05-00631bc054-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.855 DEBUG:boto:b'jkcgbj-17obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:12.856 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:12.856 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:12.857 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:12.858 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:12.858 DEBUG:boto:Method: GET 2022-09-09T22:38:12.859 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:38:12.859 DEBUG:boto:Data: 2022-09-09T22:38:12.860 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.860 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:12.861 DEBUG:boto:Port: 8000 2022-09-09T22:38:12.861 DEBUG:boto:Params: {} 2022-09-09T22:38:12.862 DEBUG:boto:Token: None 2022-09-09T22:38:12.863 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/ 2022-09-09T22:38:12.863 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:l84mt4fRZO9GtDJ0oW1tTPZ7yx4= 2022-09-09T22:38:12.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:l84mt4fRZO9GtDJ0oW1tTPZ7yx4=', 'Content-Length': '0'} 2022-09-09T22:38:12.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb6b862132bbbe36-00631bc054-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.870 DEBUG:boto:b'jkcgbj-17obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:12.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:38:12.871 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:12.872 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:12.873 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:12.873 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:12.874 DEBUG:boto:Method: GET 2022-09-09T22:38:12.874 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:12.875 DEBUG:boto:Data: 2022-09-09T22:38:12.875 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.876 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:12.876 DEBUG:boto:Port: 8001 2022-09-09T22:38:12.877 DEBUG:boto:Params: {} 2022-09-09T22:38:12.877 DEBUG:boto:Token: None 2022-09-09T22:38:12.878 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:12.879 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+sjFhHSOMXBpJ7leBxBGsjmXW/w= 2022-09-09T22:38:12.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+sjFhHSOMXBpJ7leBxBGsjmXW/w=', 'Content-Length': '0'} 2022-09-09T22:38:12.882 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000274efa1446cd3165-00631bc054-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.883 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:12.884 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:12.884 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:12.885 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:12.886 DEBUG:boto:Method: GET 2022-09-09T22:38:12.886 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:12.887 DEBUG:boto:Data: 2022-09-09T22:38:12.888 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.889 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:12.889 DEBUG:boto:Port: 8000 2022-09-09T22:38:12.890 DEBUG:boto:Params: {} 2022-09-09T22:38:12.891 DEBUG:boto:Token: None 2022-09-09T22:38:12.891 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:12.892 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+sjFhHSOMXBpJ7leBxBGsjmXW/w= 2022-09-09T22:38:12.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+sjFhHSOMXBpJ7leBxBGsjmXW/w=', 'Content-Length': '0'} 2022-09-09T22:38:12.896 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d06c14d58deaac7d-00631bc054-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.897 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:12.897 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:12.898 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:12.898 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:12.899 DEBUG:boto:Method: HEAD 2022-09-09T22:38:12.899 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:12.900 DEBUG:boto:Data: 2022-09-09T22:38:12.900 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.901 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:12.901 DEBUG:boto:Port: 8001 2022-09-09T22:38:12.902 DEBUG:boto:Params: {} 2022-09-09T22:38:12.902 DEBUG:boto:Token: None 2022-09-09T22:38:12.903 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:12.903 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:uUrDTf8wJZGmcjf0bS1v1aN5cJU= 2022-09-09T22:38:12.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:uUrDTf8wJZGmcjf0bS1v1aN5cJU=', 'Content-Length': '0'} 2022-09-09T22:38:12.907 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba8c507e39c2cc9b-00631bc054-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.908 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:12.908 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:12.909 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:12.909 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:12.910 DEBUG:boto:Method: HEAD 2022-09-09T22:38:12.910 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:12.911 DEBUG:boto:Data: 2022-09-09T22:38:12.911 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.912 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:12.913 DEBUG:boto:Port: 8000 2022-09-09T22:38:12.913 DEBUG:boto:Params: {} 2022-09-09T22:38:12.914 DEBUG:boto:Token: None 2022-09-09T22:38:12.914 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:12.915 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:uUrDTf8wJZGmcjf0bS1v1aN5cJU= 2022-09-09T22:38:12.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:uUrDTf8wJZGmcjf0bS1v1aN5cJU=', 'Content-Length': '0'} 2022-09-09T22:38:12.918 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000370daae1e865c718-00631bc054-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:38:12.920 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:12.920 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:12.921 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:12.921 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:12.922 DEBUG:boto:Method: GET 2022-09-09T22:38:12.923 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:12.923 DEBUG:boto:Data: 2022-09-09T22:38:12.924 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.924 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:12.925 DEBUG:boto:Port: 8001 2022-09-09T22:38:12.925 DEBUG:boto:Params: {} 2022-09-09T22:38:12.926 DEBUG:boto:Token: None 2022-09-09T22:38:12.927 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:12.928 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+sjFhHSOMXBpJ7leBxBGsjmXW/w= 2022-09-09T22:38:12.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+sjFhHSOMXBpJ7leBxBGsjmXW/w=', 'Content-Length': '0'} 2022-09-09T22:38:12.931 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bac22b995872f3a0-00631bc054-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.974 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:12.975 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:12.976 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:12.976 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:12.977 DEBUG:boto:Method: GET 2022-09-09T22:38:12.977 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:12.978 DEBUG:boto:Data: 2022-09-09T22:38:12.979 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.980 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:12.981 DEBUG:boto:Port: 8000 2022-09-09T22:38:12.982 DEBUG:boto:Params: {} 2022-09-09T22:38:12.983 DEBUG:boto:Token: None 2022-09-09T22:38:12.984 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:12.986 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+sjFhHSOMXBpJ7leBxBGsjmXW/w= 2022-09-09T22:38:12.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+sjFhHSOMXBpJ7leBxBGsjmXW/w=', 'Content-Length': '0'} 2022-09-09T22:38:12.991 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001bb447c5c2d76f4d-00631bc054-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:12.992 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:12.992 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:12.993 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:12.993 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:12.994 DEBUG:boto:Method: GET 2022-09-09T22:38:12.995 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:12.995 DEBUG:boto:Data: 2022-09-09T22:38:12.996 DEBUG:boto:Headers: {} 2022-09-09T22:38:12.996 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:12.997 DEBUG:boto:Port: 8001 2022-09-09T22:38:12.997 DEBUG:boto:Params: {} 2022-09-09T22:38:12.998 DEBUG:boto:Token: None 2022-09-09T22:38:12.999 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:12 GMT /jkcgbj-17/ 2022-09-09T22:38:12.999 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:l84mt4fRZO9GtDJ0oW1tTPZ7yx4= 2022-09-09T22:38:13.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:l84mt4fRZO9GtDJ0oW1tTPZ7yx4=', 'Content-Length': '0'} 2022-09-09T22:38:13.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6e7853138174b8d-00631bc054-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:13.007 DEBUG:boto:b'jkcgbj-17obj-test-zone31000falseobj-test-zone32022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:13.008 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:13.009 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:13.010 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:13.011 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:13.012 DEBUG:boto:Method: GET 2022-09-09T22:38:13.013 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:13.014 DEBUG:boto:Data: 2022-09-09T22:38:13.015 DEBUG:boto:Headers: {} 2022-09-09T22:38:13.015 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:13.016 DEBUG:boto:Port: 8000 2022-09-09T22:38:13.017 DEBUG:boto:Params: {} 2022-09-09T22:38:13.018 DEBUG:boto:Token: None 2022-09-09T22:38:13.019 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:13 GMT /jkcgbj-17/ 2022-09-09T22:38:13.020 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:y5+DkCjydc29GY+HA8HIYBAvo6Y= 2022-09-09T22:38:13.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:13 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:y5+DkCjydc29GY+HA8HIYBAvo6Y=', 'Content-Length': '0'} 2022-09-09T22:38:13.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a18fc77346c3897-00631bc055-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:13.028 DEBUG:boto:b'jkcgbj-17obj-test-zone31000falseobj-test-zone32022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:13.029 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:13.030 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:13.031 DEBUG:boto:Method: HEAD 2022-09-09T22:38:13.032 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3 2022-09-09T22:38:13.033 DEBUG:boto:Data: 2022-09-09T22:38:13.034 DEBUG:boto:Headers: {} 2022-09-09T22:38:13.035 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:13.036 DEBUG:boto:Port: 8001 2022-09-09T22:38:13.037 DEBUG:boto:Params: {} 2022-09-09T22:38:13.038 DEBUG:boto:Token: None 2022-09-09T22:38:13.039 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:13 GMT /jkcgbj-17/obj-test-zone3 2022-09-09T22:38:13.039 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:obRN/wPXJw5MkIPmwpsAyTPWTRM= 2022-09-09T22:38:13.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:13 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:obRN/wPXJw5MkIPmwpsAyTPWTRM=', 'Content-Length': '0'} 2022-09-09T22:38:13.044 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a1d7badfacff164d-00631bc055-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:13.045 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:13.045 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:13.046 DEBUG:boto:Method: HEAD 2022-09-09T22:38:13.047 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3 2022-09-09T22:38:13.048 DEBUG:boto:Data: 2022-09-09T22:38:13.048 DEBUG:boto:Headers: {} 2022-09-09T22:38:13.049 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:13.050 DEBUG:boto:Port: 8000 2022-09-09T22:38:13.050 DEBUG:boto:Params: {} 2022-09-09T22:38:13.051 DEBUG:boto:Token: None 2022-09-09T22:38:13.051 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:13 GMT /jkcgbj-17/obj-test-zone3 2022-09-09T22:38:13.052 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:obRN/wPXJw5MkIPmwpsAyTPWTRM= 2022-09-09T22:38:13.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:13 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:obRN/wPXJw5MkIPmwpsAyTPWTRM=', 'Content-Length': '0'} 2022-09-09T22:38:13.056 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a82a67a2cf4cd627-00631bc055-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:13.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:38:13.058 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:13.059 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:13.059 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:13.060 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:13.060 DEBUG:boto:Method: GET 2022-09-09T22:38:13.061 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:13.062 DEBUG:boto:Data: 2022-09-09T22:38:13.062 DEBUG:boto:Headers: {} 2022-09-09T22:38:13.063 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:13.063 DEBUG:boto:Port: 8001 2022-09-09T22:38:13.064 DEBUG:boto:Params: {} 2022-09-09T22:38:13.064 DEBUG:boto:Token: None 2022-09-09T22:38:13.065 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:13 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:13.066 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:i65SgUW/gb37Fz5+A2UmGJ6/EH0= 2022-09-09T22:38:13.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:13 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:i65SgUW/gb37Fz5+A2UmGJ6/EH0=', 'Content-Length': '0'} 2022-09-09T22:38:13.070 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000092f7a40e348b39fc-00631bc055-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:13.110 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:13.111 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:13.111 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:13.112 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:13.112 DEBUG:boto:Method: GET 2022-09-09T22:38:13.113 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:13.115 DEBUG:boto:Data: 2022-09-09T22:38:13.115 DEBUG:boto:Headers: {} 2022-09-09T22:38:13.116 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:13.117 DEBUG:boto:Port: 8000 2022-09-09T22:38:13.118 DEBUG:boto:Params: {} 2022-09-09T22:38:13.119 DEBUG:boto:Token: None 2022-09-09T22:38:13.121 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:13 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:38:13.122 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:i65SgUW/gb37Fz5+A2UmGJ6/EH0= 2022-09-09T22:38:13.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:13 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:i65SgUW/gb37Fz5+A2UmGJ6/EH0=', 'Content-Length': '0'} 2022-09-09T22:38:13.126 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000862e32276d070b18-00631bc055-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:13.128 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:13.129 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:13.130 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:13.131 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:13.132 DEBUG:boto:Method: GET 2022-09-09T22:38:13.133 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:13.134 DEBUG:boto:Data: 2022-09-09T22:38:13.134 DEBUG:boto:Headers: {} 2022-09-09T22:38:13.135 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:13.136 DEBUG:boto:Port: 8001 2022-09-09T22:38:13.137 DEBUG:boto:Params: {} 2022-09-09T22:38:13.138 DEBUG:boto:Token: None 2022-09-09T22:38:13.140 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:13 GMT /jkcgbj-17/ 2022-09-09T22:38:13.141 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:y5+DkCjydc29GY+HA8HIYBAvo6Y= 2022-09-09T22:38:13.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:13 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:y5+DkCjydc29GY+HA8HIYBAvo6Y=', 'Content-Length': '0'} 2022-09-09T22:38:13.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da2bcd7efad4d3fc-00631bc055-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:13.149 DEBUG:boto:b'jkcgbj-17obj-test-zone31000falseobj-test-zone32022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:13.150 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:13.151 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:13.151 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:13.152 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:13.152 DEBUG:boto:Method: GET 2022-09-09T22:38:13.153 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:38:13.153 DEBUG:boto:Data: 2022-09-09T22:38:13.154 DEBUG:boto:Headers: {} 2022-09-09T22:38:13.154 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:13.155 DEBUG:boto:Port: 8000 2022-09-09T22:38:13.156 DEBUG:boto:Params: {} 2022-09-09T22:38:13.157 DEBUG:boto:Token: None 2022-09-09T22:38:13.157 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:13 GMT /jkcgbj-17/ 2022-09-09T22:38:13.158 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:y5+DkCjydc29GY+HA8HIYBAvo6Y= 2022-09-09T22:38:13.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:13 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:y5+DkCjydc29GY+HA8HIYBAvo6Y=', 'Content-Length': '0'} 2022-09-09T22:38:13.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c2cfa2c09062973-00631bc055-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:13.210 DEBUG:boto:b'jkcgbj-17obj-test-zone31000falseobj-test-zone32022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:13.212 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-17 zones={test-zone3, test-zone2} 2022-09-09T22:38:13.213 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jkcgbj-18 2022-09-09T22:38:13.214 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:13.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:13.216 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:38:13.270 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:38:13.271 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:38:13.347 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:38:13.340+0000 7fc6d6e2cdc0 1 bucket sync caught up with source: 2022-09-09T22:38:13.347 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000010.27.5, , 00000000003.16.9, , 00000000007.15.9, , , , , ] 2022-09-09T22:38:13.347 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000010.27.5, , 00000000003.16.9, , 00000000007.15.9, , , , , ] 2022-09-09T22:38:13.348 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:38:13.340+0000 7fc6d6e2cdc0 0 bucket checkpoint complete 2022-09-09T22:38:13.374 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jkcgbj-18 2022-09-09T22:38:13.374 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:13.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:13.376 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:38:13.430 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:38:13.431 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:38:13.497 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:38:13.489+0000 7f403b1b6dc0 1 bucket sync caught up with source: 2022-09-09T22:38:13.497 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000010.27.5, , 00000000003.16.9, , 00000000007.15.9, , , , , ] 2022-09-09T22:38:13.497 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000010.27.5, , 00000000003.16.9, , 00000000007.15.9, , , , , ] 2022-09-09T22:38:13.498 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:38:13.489+0000 7f403b1b6dc0 0 bucket checkpoint complete 2022-09-09T22:38:13.523 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jkcgbj-18 2022-09-09T22:38:13.523 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:13.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:13.525 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:38:13.578 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:38:13.579 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:38:13.647 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:38:13.640+0000 7fb4483dbdc0 1 bucket sync caught up with source: 2022-09-09T22:38:13.648 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000010.27.5, , 00000000003.16.9, , 00000000007.15.9, , , , , ] 2022-09-09T22:38:13.649 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000010.27.5, , 00000000003.16.9, , 00000000007.15.9, , , , , ] 2022-09-09T22:38:13.649 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:38:13.640+0000 7fb4483dbdc0 0 bucket checkpoint complete 2022-09-09T22:38:13.671 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jkcgbj-18 2022-09-09T22:38:13.671 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:13.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:13.673 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:38:13.731 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:38:13.731 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:38:13.791 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:38:13.785+0000 7f407b69fdc0 1 bucket sync caught up with source: 2022-09-09T22:38:13.791 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000003.37.9, , 00000000010.52.5, , 00000000007.21.9, , , , , ] 2022-09-09T22:38:13.792 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000003.37.9, , 00000000010.52.5, , 00000000007.21.9, , , , , ] 2022-09-09T22:38:13.792 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:38:13.785+0000 7f407b69fdc0 0 bucket checkpoint complete 2022-09-09T22:38:13.818 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jkcgbj-18 2022-09-09T22:38:13.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:13.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:13.820 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:38:13.873 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:38:13.873 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:38:13.940 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:38:13.933+0000 7ff28d269dc0 1 bucket sync caught up with source: 2022-09-09T22:38:13.940 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000003.37.9, , 00000000010.52.5, , 00000000007.21.9, , , , , ] 2022-09-09T22:38:13.940 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000003.37.9, , 00000000010.52.5, , 00000000007.21.9, , , , , ] 2022-09-09T22:38:13.941 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:38:13.933+0000 7ff28d269dc0 0 bucket checkpoint complete 2022-09-09T22:38:13.966 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jkcgbj-18 2022-09-09T22:38:13.967 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jkcgbj-18 2022-09-09T22:38:13.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:13.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:13.968 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:38:14.022 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:38:14.023 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:38:14.089 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:38:14.085+0000 7f9c48141dc0 1 bucket sync caught up with source: 2022-09-09T22:38:14.090 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000003.37.9, , 00000000003.38.9, , 00000000015.36.5, , , , , ] 2022-09-09T22:38:14.090 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000003.37.9, , 00000000003.38.9, , 00000000015.36.5, , , , , ] 2022-09-09T22:38:14.091 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:38:14.085+0000 7f9c48141dc0 0 bucket checkpoint complete 2022-09-09T22:38:14.117 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jkcgbj-18 2022-09-09T22:38:14.118 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:14.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:14.121 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:38:14.174 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:38:14.175 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:38:14.250 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:38:14.244+0000 7fb237b4bdc0 1 bucket sync caught up with source: 2022-09-09T22:38:14.251 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000003.37.9, , 00000000003.38.9, , 00000000015.36.5, , , , , ] 2022-09-09T22:38:14.251 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000003.37.9, , 00000000003.38.9, , 00000000015.36.5, , , , , ] 2022-09-09T22:38:14.252 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:38:14.244+0000 7fb237b4bdc0 0 bucket checkpoint complete 2022-09-09T22:38:14.282 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jkcgbj-18 2022-09-09T22:38:14.283 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-18 zones={test-zone2, test-zone1} 2022-09-09T22:38:14.283 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:14.284 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:14.284 DEBUG:boto:Method: HEAD 2022-09-09T22:38:14.285 DEBUG:boto:Path: /jkcgbj-18/ 2022-09-09T22:38:14.285 DEBUG:boto:Data: 2022-09-09T22:38:14.286 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.287 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:14.287 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.288 DEBUG:boto:Params: {} 2022-09-09T22:38:14.288 DEBUG:boto:Token: None 2022-09-09T22:38:14.289 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/ 2022-09-09T22:38:14.290 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4Zcl182CLfapTAx3wWqH05OpbGY= 2022-09-09T22:38:14.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4Zcl182CLfapTAx3wWqH05OpbGY=', 'Content-Length': '0'} 2022-09-09T22:38:14.295 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001cacca5b357fb003-00631bc056-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.295 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:14.296 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:14.297 DEBUG:boto:Method: HEAD 2022-09-09T22:38:14.297 DEBUG:boto:Path: /jkcgbj-18/ 2022-09-09T22:38:14.298 DEBUG:boto:Data: 2022-09-09T22:38:14.298 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.299 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:14.299 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.300 DEBUG:boto:Params: {} 2022-09-09T22:38:14.301 DEBUG:boto:Token: None 2022-09-09T22:38:14.302 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/ 2022-09-09T22:38:14.302 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4Zcl182CLfapTAx3wWqH05OpbGY= 2022-09-09T22:38:14.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4Zcl182CLfapTAx3wWqH05OpbGY=', 'Content-Length': '0'} 2022-09-09T22:38:14.308 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b884d9b3d1d0d47d-00631bc056-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.308 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:38:14.309 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:14.310 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:14.310 DEBUG:boto:path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:14.311 DEBUG:boto:auth_path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:14.312 DEBUG:boto:Method: GET 2022-09-09T22:38:14.312 DEBUG:boto:Path: /jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:14.313 DEBUG:boto:Data: 2022-09-09T22:38:14.314 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.314 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:14.315 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.315 DEBUG:boto:Params: {} 2022-09-09T22:38:14.316 DEBUG:boto:Token: None 2022-09-09T22:38:14.316 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/?versions 2022-09-09T22:38:14.317 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:thHsrJAnW1m0blI+WzSbOcRakKw= 2022-09-09T22:38:14.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:thHsrJAnW1m0blI+WzSbOcRakKw=', 'Content-Length': '0'} 2022-09-09T22:38:14.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000302aa4173e4be859-00631bc056-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.324 DEBUG:boto:b'jkcgbj-18999falsedummyRQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5true2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1a8lrwM74XPlF1mwXf8hNl9Y9ISdV53ltrue2022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2e.gQ6yFECscD-oc4OV0dmk6aAg9jH.otrue2022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1true2022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:14.325 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:38:14.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:38:14.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:38:14.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:38:14.327 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:38:14.328 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:14.328 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:14.329 DEBUG:boto:path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:14.329 DEBUG:boto:auth_path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:14.330 DEBUG:boto:Method: GET 2022-09-09T22:38:14.330 DEBUG:boto:Path: /jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:14.331 DEBUG:boto:Data: 2022-09-09T22:38:14.331 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.332 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:14.332 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.333 DEBUG:boto:Params: {} 2022-09-09T22:38:14.333 DEBUG:boto:Token: None 2022-09-09T22:38:14.334 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/?versions 2022-09-09T22:38:14.335 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:thHsrJAnW1m0blI+WzSbOcRakKw= 2022-09-09T22:38:14.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:thHsrJAnW1m0blI+WzSbOcRakKw=', 'Content-Length': '0'} 2022-09-09T22:38:14.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c45be661c983e40-00631bc056-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.342 DEBUG:boto:b'jkcgbj-18999falsedummyRQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5true2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1a8lrwM74XPlF1mwXf8hNl9Y9ISdV53ltrue2022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2e.gQ6yFECscD-oc4OV0dmk6aAg9jH.otrue2022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1true2022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:14.343 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:38:14.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:38:14.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:38:14.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:38:14.345 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:14.345 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:14.346 DEBUG:boto:path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:14.347 DEBUG:boto:auth_path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:14.347 DEBUG:boto:Method: GET 2022-09-09T22:38:14.348 DEBUG:boto:Path: /jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:14.348 DEBUG:boto:Data: 2022-09-09T22:38:14.349 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.350 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:14.351 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.352 DEBUG:boto:Params: {} 2022-09-09T22:38:14.353 DEBUG:boto:Token: None 2022-09-09T22:38:14.353 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/?versions 2022-09-09T22:38:14.354 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:thHsrJAnW1m0blI+WzSbOcRakKw= 2022-09-09T22:38:14.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:thHsrJAnW1m0blI+WzSbOcRakKw=', 'Content-Length': '0'} 2022-09-09T22:38:14.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000693ccb7087af1435-00631bc056-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.361 DEBUG:boto:b'jkcgbj-18999falsedummyRQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5true2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1a8lrwM74XPlF1mwXf8hNl9Y9ISdV53ltrue2022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2e.gQ6yFECscD-oc4OV0dmk6aAg9jH.otrue2022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1true2022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:14.363 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:14.364 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:14.365 DEBUG:boto:path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:14.366 DEBUG:boto:auth_path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:14.366 DEBUG:boto:Method: GET 2022-09-09T22:38:14.367 DEBUG:boto:Path: /jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:14.368 DEBUG:boto:Data: 2022-09-09T22:38:14.369 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.370 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:14.371 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.372 DEBUG:boto:Params: {} 2022-09-09T22:38:14.373 DEBUG:boto:Token: None 2022-09-09T22:38:14.374 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/?versions 2022-09-09T22:38:14.374 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:thHsrJAnW1m0blI+WzSbOcRakKw= 2022-09-09T22:38:14.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:thHsrJAnW1m0blI+WzSbOcRakKw=', 'Content-Length': '0'} 2022-09-09T22:38:14.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1554433448e826a-00631bc056-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.381 DEBUG:boto:b'jkcgbj-18999falsedummyRQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5true2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1a8lrwM74XPlF1mwXf8hNl9Y9ISdV53ltrue2022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2e.gQ6yFECscD-oc4OV0dmk6aAg9jH.otrue2022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1true2022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:14.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:38:14.382 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:38:14.383 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:38:14.384 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.384 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.385 DEBUG:boto:Method: GET 2022-09-09T22:38:14.385 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.386 DEBUG:boto:Data: 2022-09-09T22:38:14.386 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.387 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:14.388 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.389 DEBUG:boto:Params: {} 2022-09-09T22:38:14.389 DEBUG:boto:Token: None 2022-09-09T22:38:14.390 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.391 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:oR8gmAxe12fGeoxmPmuBuWZ9yQc= 2022-09-09T22:38:14.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:oR8gmAxe12fGeoxmPmuBuWZ9yQc=', 'Content-Length': '0'} 2022-09-09T22:38:14.394 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000851aae52a0ecef6b-00631bc056-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.395 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:38:14.396 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:38:14.396 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.397 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.397 DEBUG:boto:Method: GET 2022-09-09T22:38:14.398 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.398 DEBUG:boto:Data: 2022-09-09T22:38:14.399 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.399 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:14.400 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.401 DEBUG:boto:Params: {} 2022-09-09T22:38:14.401 DEBUG:boto:Token: None 2022-09-09T22:38:14.402 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.403 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:oR8gmAxe12fGeoxmPmuBuWZ9yQc= 2022-09-09T22:38:14.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:oR8gmAxe12fGeoxmPmuBuWZ9yQc=', 'Content-Length': '0'} 2022-09-09T22:38:14.406 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aa75ca8a0cb3e5ed-00631bc056-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.407 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:38:14.408 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:38:14.408 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.409 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.409 DEBUG:boto:Method: HEAD 2022-09-09T22:38:14.410 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.410 DEBUG:boto:Data: 2022-09-09T22:38:14.411 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.411 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:14.412 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.412 DEBUG:boto:Params: {} 2022-09-09T22:38:14.413 DEBUG:boto:Token: None 2022-09-09T22:38:14.414 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.414 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8uLiTK2CbYO7re8aajY7mkT1tQA= 2022-09-09T22:38:14.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8uLiTK2CbYO7re8aajY7mkT1tQA=', 'Content-Length': '0'} 2022-09-09T22:38:14.418 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003c8761d84634d377-00631bc056-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.419 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:38:14.419 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:38:14.420 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.420 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.421 DEBUG:boto:Method: HEAD 2022-09-09T22:38:14.422 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.422 DEBUG:boto:Data: 2022-09-09T22:38:14.423 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.423 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:14.424 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.424 DEBUG:boto:Params: {} 2022-09-09T22:38:14.425 DEBUG:boto:Token: None 2022-09-09T22:38:14.425 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.426 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8uLiTK2CbYO7re8aajY7mkT1tQA= 2022-09-09T22:38:14.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8uLiTK2CbYO7re8aajY7mkT1tQA=', 'Content-Length': '0'} 2022-09-09T22:38:14.429 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000145b841fb710e7c8-00631bc056-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:38:14.431 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:38:14.431 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:38:14.432 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.432 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.433 DEBUG:boto:Method: GET 2022-09-09T22:38:14.434 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.434 DEBUG:boto:Data: 2022-09-09T22:38:14.435 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.435 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:14.436 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.436 DEBUG:boto:Params: {} 2022-09-09T22:38:14.437 DEBUG:boto:Token: None 2022-09-09T22:38:14.437 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.438 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:oR8gmAxe12fGeoxmPmuBuWZ9yQc= 2022-09-09T22:38:14.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:oR8gmAxe12fGeoxmPmuBuWZ9yQc=', 'Content-Length': '0'} 2022-09-09T22:38:14.442 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009d72ab8430ecced8-00631bc056-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.443 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:38:14.443 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:38:14.444 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.444 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.445 DEBUG:boto:Method: GET 2022-09-09T22:38:14.445 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.446 DEBUG:boto:Data: 2022-09-09T22:38:14.446 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.447 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:14.447 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.448 DEBUG:boto:Params: {} 2022-09-09T22:38:14.449 DEBUG:boto:Token: None 2022-09-09T22:38:14.449 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.450 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:oR8gmAxe12fGeoxmPmuBuWZ9yQc= 2022-09-09T22:38:14.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:oR8gmAxe12fGeoxmPmuBuWZ9yQc=', 'Content-Length': '0'} 2022-09-09T22:38:14.454 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003fd6a33ec99531fd-00631bc056-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.455 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:14.455 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:14.456 DEBUG:boto:path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:38:14.456 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:38:14.457 DEBUG:boto:Method: GET 2022-09-09T22:38:14.457 DEBUG:boto:Path: /jkcgbj-18/?prefix=dummy 2022-09-09T22:38:14.458 DEBUG:boto:Data: 2022-09-09T22:38:14.459 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.459 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:14.460 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.460 DEBUG:boto:Params: {} 2022-09-09T22:38:14.461 DEBUG:boto:Token: None 2022-09-09T22:38:14.461 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/ 2022-09-09T22:38:14.462 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jBGixj+kYu4w415FO0tKFdbQnNY= 2022-09-09T22:38:14.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jBGixj+kYu4w415FO0tKFdbQnNY=', 'Content-Length': '0'} 2022-09-09T22:38:14.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a7227486f1ac76a-00631bc056-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.510 DEBUG:boto:b'jkcgbj-18dummy1000falsedummy2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:14.511 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:14.512 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:14.512 DEBUG:boto:path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:38:14.513 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:38:14.513 DEBUG:boto:Method: GET 2022-09-09T22:38:14.514 DEBUG:boto:Path: /jkcgbj-18/?prefix=dummy 2022-09-09T22:38:14.514 DEBUG:boto:Data: 2022-09-09T22:38:14.515 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.515 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:14.516 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.516 DEBUG:boto:Params: {} 2022-09-09T22:38:14.517 DEBUG:boto:Token: None 2022-09-09T22:38:14.518 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/ 2022-09-09T22:38:14.518 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jBGixj+kYu4w415FO0tKFdbQnNY= 2022-09-09T22:38:14.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jBGixj+kYu4w415FO0tKFdbQnNY=', 'Content-Length': '0'} 2022-09-09T22:38:14.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000873ef39e4ede9bf0-00631bc056-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.525 DEBUG:boto:b'jkcgbj-18dummy1000falsedummy2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:14.526 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:38:14.526 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:38:14.527 DEBUG:boto:Method: HEAD 2022-09-09T22:38:14.527 DEBUG:boto:Path: /jkcgbj-18/dummy 2022-09-09T22:38:14.528 DEBUG:boto:Data: 2022-09-09T22:38:14.528 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.529 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:14.529 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.530 DEBUG:boto:Params: {} 2022-09-09T22:38:14.530 DEBUG:boto:Token: None 2022-09-09T22:38:14.531 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/dummy 2022-09-09T22:38:14.532 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:px3NFK/2p5rcwGFJPb4aAOp3nao= 2022-09-09T22:38:14.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:px3NFK/2p5rcwGFJPb4aAOp3nao=', 'Content-Length': '0'} 2022-09-09T22:38:14.536 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001e89086ff2ee2158-00631bc056-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.536 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:38:14.537 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:38:14.537 DEBUG:boto:Method: HEAD 2022-09-09T22:38:14.538 DEBUG:boto:Path: /jkcgbj-18/dummy 2022-09-09T22:38:14.538 DEBUG:boto:Data: 2022-09-09T22:38:14.539 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.539 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:14.540 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.540 DEBUG:boto:Params: {} 2022-09-09T22:38:14.541 DEBUG:boto:Token: None 2022-09-09T22:38:14.541 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/dummy 2022-09-09T22:38:14.542 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:px3NFK/2p5rcwGFJPb4aAOp3nao= 2022-09-09T22:38:14.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:px3NFK/2p5rcwGFJPb4aAOp3nao=', 'Content-Length': '0'} 2022-09-09T22:38:14.547 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bfe04738bb49937b-00631bc056-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:38:14.548 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:38:14.548 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:38:14.549 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.549 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.550 DEBUG:boto:Method: GET 2022-09-09T22:38:14.551 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.551 DEBUG:boto:Data: 2022-09-09T22:38:14.552 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.552 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:14.553 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.553 DEBUG:boto:Params: {} 2022-09-09T22:38:14.554 DEBUG:boto:Token: None 2022-09-09T22:38:14.554 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.555 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:oR8gmAxe12fGeoxmPmuBuWZ9yQc= 2022-09-09T22:38:14.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:oR8gmAxe12fGeoxmPmuBuWZ9yQc=', 'Content-Length': '0'} 2022-09-09T22:38:14.558 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000075382b47a063d9a5-00631bc056-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.559 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:38:14.560 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:38:14.560 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.561 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.561 DEBUG:boto:Method: GET 2022-09-09T22:38:14.562 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.563 DEBUG:boto:Data: 2022-09-09T22:38:14.563 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.564 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:14.564 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.565 DEBUG:boto:Params: {} 2022-09-09T22:38:14.565 DEBUG:boto:Token: None 2022-09-09T22:38:14.566 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:14.567 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:oR8gmAxe12fGeoxmPmuBuWZ9yQc= 2022-09-09T22:38:14.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:oR8gmAxe12fGeoxmPmuBuWZ9yQc=', 'Content-Length': '0'} 2022-09-09T22:38:14.570 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005528542fc0be43d2-00631bc056-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.571 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:14.572 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:14.572 DEBUG:boto:path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:38:14.573 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:38:14.573 DEBUG:boto:Method: GET 2022-09-09T22:38:14.574 DEBUG:boto:Path: /jkcgbj-18/?prefix=dummy 2022-09-09T22:38:14.575 DEBUG:boto:Data: 2022-09-09T22:38:14.575 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.576 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:14.576 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.577 DEBUG:boto:Params: {} 2022-09-09T22:38:14.577 DEBUG:boto:Token: None 2022-09-09T22:38:14.578 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/ 2022-09-09T22:38:14.579 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jBGixj+kYu4w415FO0tKFdbQnNY= 2022-09-09T22:38:14.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jBGixj+kYu4w415FO0tKFdbQnNY=', 'Content-Length': '0'} 2022-09-09T22:38:14.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066307e335a9a44cd-00631bc056-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.626 DEBUG:boto:b'jkcgbj-18dummy1000falsedummy2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:14.627 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:14.628 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:14.628 DEBUG:boto:path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:38:14.629 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:38:14.629 DEBUG:boto:Method: GET 2022-09-09T22:38:14.630 DEBUG:boto:Path: /jkcgbj-18/?prefix=dummy 2022-09-09T22:38:14.630 DEBUG:boto:Data: 2022-09-09T22:38:14.631 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.631 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:14.632 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.633 DEBUG:boto:Params: {} 2022-09-09T22:38:14.633 DEBUG:boto:Token: None 2022-09-09T22:38:14.634 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/ 2022-09-09T22:38:14.635 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jBGixj+kYu4w415FO0tKFdbQnNY= 2022-09-09T22:38:14.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jBGixj+kYu4w415FO0tKFdbQnNY=', 'Content-Length': '0'} 2022-09-09T22:38:14.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019172bcd1c1d2d4a-00631bc056-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.641 DEBUG:boto:b'jkcgbj-18dummy1000falsedummy2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:14.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:38:14.642 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:14.643 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:14.643 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.644 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.644 DEBUG:boto:Method: GET 2022-09-09T22:38:14.645 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.645 DEBUG:boto:Data: 2022-09-09T22:38:14.646 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.647 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:14.647 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.648 DEBUG:boto:Params: {} 2022-09-09T22:38:14.648 DEBUG:boto:Token: None 2022-09-09T22:38:14.649 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.649 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:P6KpL4PrVpMKHHXQ2at0yQfjWMc= 2022-09-09T22:38:14.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:P6KpL4PrVpMKHHXQ2at0yQfjWMc=', 'Content-Length': '0'} 2022-09-09T22:38:14.653 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b80a842a14ae7c1b-00631bc056-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.654 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:14.655 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:14.655 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.656 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.656 DEBUG:boto:Method: GET 2022-09-09T22:38:14.657 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.658 DEBUG:boto:Data: 2022-09-09T22:38:14.658 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.659 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:14.659 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.660 DEBUG:boto:Params: {} 2022-09-09T22:38:14.661 DEBUG:boto:Token: None 2022-09-09T22:38:14.661 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.662 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:P6KpL4PrVpMKHHXQ2at0yQfjWMc= 2022-09-09T22:38:14.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:P6KpL4PrVpMKHHXQ2at0yQfjWMc=', 'Content-Length': '0'} 2022-09-09T22:38:14.666 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000021a2d5c88c780b41-00631bc056-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.706 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:14.707 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:14.707 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.708 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.708 DEBUG:boto:Method: HEAD 2022-09-09T22:38:14.709 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.710 DEBUG:boto:Data: 2022-09-09T22:38:14.711 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.712 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:14.712 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.713 DEBUG:boto:Params: {} 2022-09-09T22:38:14.713 DEBUG:boto:Token: None 2022-09-09T22:38:14.714 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.715 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:t5Byd44rtFHEWlLW0JEB725/65M= 2022-09-09T22:38:14.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:t5Byd44rtFHEWlLW0JEB725/65M=', 'Content-Length': '0'} 2022-09-09T22:38:14.721 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000188d1b7fcddccdd5-00631bc056-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.722 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:14.722 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:14.723 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.723 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.724 DEBUG:boto:Method: HEAD 2022-09-09T22:38:14.724 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.725 DEBUG:boto:Data: 2022-09-09T22:38:14.725 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.726 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:14.726 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.727 DEBUG:boto:Params: {} 2022-09-09T22:38:14.727 DEBUG:boto:Token: None 2022-09-09T22:38:14.728 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.729 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:t5Byd44rtFHEWlLW0JEB725/65M= 2022-09-09T22:38:14.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:t5Byd44rtFHEWlLW0JEB725/65M=', 'Content-Length': '0'} 2022-09-09T22:38:14.733 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000725256f6ae4f0442-00631bc056-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:38:14.734 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:14.734 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:14.735 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.735 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.736 DEBUG:boto:Method: GET 2022-09-09T22:38:14.737 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.737 DEBUG:boto:Data: 2022-09-09T22:38:14.738 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.738 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:14.739 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.739 DEBUG:boto:Params: {} 2022-09-09T22:38:14.740 DEBUG:boto:Token: None 2022-09-09T22:38:14.740 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.741 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:P6KpL4PrVpMKHHXQ2at0yQfjWMc= 2022-09-09T22:38:14.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:P6KpL4PrVpMKHHXQ2at0yQfjWMc=', 'Content-Length': '0'} 2022-09-09T22:38:14.745 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000288dec0d58d18fa6-00631bc056-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.746 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:14.746 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:14.747 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.747 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.748 DEBUG:boto:Method: GET 2022-09-09T22:38:14.748 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.749 DEBUG:boto:Data: 2022-09-09T22:38:14.749 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.750 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:14.750 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.751 DEBUG:boto:Params: {} 2022-09-09T22:38:14.752 DEBUG:boto:Token: None 2022-09-09T22:38:14.752 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.753 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:P6KpL4PrVpMKHHXQ2at0yQfjWMc= 2022-09-09T22:38:14.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:P6KpL4PrVpMKHHXQ2at0yQfjWMc=', 'Content-Length': '0'} 2022-09-09T22:38:14.756 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e1531280247e9242-00631bc056-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.757 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:14.757 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:14.758 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:14.758 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:14.759 DEBUG:boto:Method: GET 2022-09-09T22:38:14.759 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:14.760 DEBUG:boto:Data: 2022-09-09T22:38:14.760 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.761 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:14.761 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.762 DEBUG:boto:Params: {} 2022-09-09T22:38:14.763 DEBUG:boto:Token: None 2022-09-09T22:38:14.763 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/ 2022-09-09T22:38:14.764 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jBGixj+kYu4w415FO0tKFdbQnNY= 2022-09-09T22:38:14.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jBGixj+kYu4w415FO0tKFdbQnNY=', 'Content-Length': '0'} 2022-09-09T22:38:14.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000565f5a4cc2c6e073-00631bc056-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.770 DEBUG:boto:b'jkcgbj-18obj-test-zone11000falseobj-test-zone12022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:14.771 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:14.772 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:14.772 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:14.773 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:14.773 DEBUG:boto:Method: GET 2022-09-09T22:38:14.774 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:14.775 DEBUG:boto:Data: 2022-09-09T22:38:14.775 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.776 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:14.776 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.777 DEBUG:boto:Params: {} 2022-09-09T22:38:14.777 DEBUG:boto:Token: None 2022-09-09T22:38:14.778 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/ 2022-09-09T22:38:14.779 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jBGixj+kYu4w415FO0tKFdbQnNY= 2022-09-09T22:38:14.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jBGixj+kYu4w415FO0tKFdbQnNY=', 'Content-Length': '0'} 2022-09-09T22:38:14.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb02821a3750acb5-00631bc056-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.792 DEBUG:boto:b'jkcgbj-18obj-test-zone11000falseobj-test-zone12022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:14.793 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:14.794 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:14.794 DEBUG:boto:Method: HEAD 2022-09-09T22:38:14.795 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1 2022-09-09T22:38:14.795 DEBUG:boto:Data: 2022-09-09T22:38:14.796 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.796 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:14.797 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.797 DEBUG:boto:Params: {} 2022-09-09T22:38:14.798 DEBUG:boto:Token: None 2022-09-09T22:38:14.798 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/obj-test-zone1 2022-09-09T22:38:14.799 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:inONKHbS+KjUdpdOOmtx0Pf5sSQ= 2022-09-09T22:38:14.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:inONKHbS+KjUdpdOOmtx0Pf5sSQ=', 'Content-Length': '0'} 2022-09-09T22:38:14.803 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3f4f3ce1d944d72-00631bc056-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.804 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:14.805 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:14.805 DEBUG:boto:Method: HEAD 2022-09-09T22:38:14.806 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1 2022-09-09T22:38:14.806 DEBUG:boto:Data: 2022-09-09T22:38:14.807 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.807 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:14.808 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.808 DEBUG:boto:Params: {} 2022-09-09T22:38:14.809 DEBUG:boto:Token: None 2022-09-09T22:38:14.809 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/obj-test-zone1 2022-09-09T22:38:14.810 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:inONKHbS+KjUdpdOOmtx0Pf5sSQ= 2022-09-09T22:38:14.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:inONKHbS+KjUdpdOOmtx0Pf5sSQ=', 'Content-Length': '0'} 2022-09-09T22:38:14.814 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008dbfee08b04b84e5-00631bc056-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:38:14.816 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:14.816 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:14.817 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.817 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.818 DEBUG:boto:Method: GET 2022-09-09T22:38:14.818 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.819 DEBUG:boto:Data: 2022-09-09T22:38:14.820 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.820 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:14.821 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.821 DEBUG:boto:Params: {} 2022-09-09T22:38:14.822 DEBUG:boto:Token: None 2022-09-09T22:38:14.823 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.823 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:P6KpL4PrVpMKHHXQ2at0yQfjWMc= 2022-09-09T22:38:14.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:P6KpL4PrVpMKHHXQ2at0yQfjWMc=', 'Content-Length': '0'} 2022-09-09T22:38:14.828 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000de204b32bbd44a1-00631bc056-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.874 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:14.875 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:14.875 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.876 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.877 DEBUG:boto:Method: GET 2022-09-09T22:38:14.877 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.878 DEBUG:boto:Data: 2022-09-09T22:38:14.878 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.879 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:14.879 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.880 DEBUG:boto:Params: {} 2022-09-09T22:38:14.880 DEBUG:boto:Token: None 2022-09-09T22:38:14.881 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:14.882 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:P6KpL4PrVpMKHHXQ2at0yQfjWMc= 2022-09-09T22:38:14.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:P6KpL4PrVpMKHHXQ2at0yQfjWMc=', 'Content-Length': '0'} 2022-09-09T22:38:14.886 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005aa9ade9890c2596-00631bc056-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.887 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:14.887 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:14.888 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:14.889 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:14.889 DEBUG:boto:Method: GET 2022-09-09T22:38:14.890 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:14.891 DEBUG:boto:Data: 2022-09-09T22:38:14.891 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.892 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:14.892 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.893 DEBUG:boto:Params: {} 2022-09-09T22:38:14.894 DEBUG:boto:Token: None 2022-09-09T22:38:14.894 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/ 2022-09-09T22:38:14.895 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jBGixj+kYu4w415FO0tKFdbQnNY= 2022-09-09T22:38:14.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jBGixj+kYu4w415FO0tKFdbQnNY=', 'Content-Length': '0'} 2022-09-09T22:38:14.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072f4c3be7aecceb9-00631bc056-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.902 DEBUG:boto:b'jkcgbj-18obj-test-zone11000falseobj-test-zone12022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:14.903 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:14.903 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:14.904 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:14.904 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:14.905 DEBUG:boto:Method: GET 2022-09-09T22:38:14.905 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:14.906 DEBUG:boto:Data: 2022-09-09T22:38:14.907 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.907 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:14.908 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.908 DEBUG:boto:Params: {} 2022-09-09T22:38:14.909 DEBUG:boto:Token: None 2022-09-09T22:38:14.909 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/ 2022-09-09T22:38:14.910 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jBGixj+kYu4w415FO0tKFdbQnNY= 2022-09-09T22:38:14.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jBGixj+kYu4w415FO0tKFdbQnNY=', 'Content-Length': '0'} 2022-09-09T22:38:14.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006ab27e502e3ffaf-00631bc056-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.962 DEBUG:boto:b'jkcgbj-18obj-test-zone11000falseobj-test-zone12022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:14.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:38:14.964 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:14.964 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:14.965 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:14.965 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:14.966 DEBUG:boto:Method: GET 2022-09-09T22:38:14.966 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:14.967 DEBUG:boto:Data: 2022-09-09T22:38:14.968 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.968 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:14.969 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.969 DEBUG:boto:Params: {} 2022-09-09T22:38:14.970 DEBUG:boto:Token: None 2022-09-09T22:38:14.971 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:14.971 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:10M6cUifV2KT+cBtWYY3Jn1zPeI= 2022-09-09T22:38:14.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:10M6cUifV2KT+cBtWYY3Jn1zPeI=', 'Content-Length': '0'} 2022-09-09T22:38:14.975 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9b0e50e925b5cf0-00631bc056-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.976 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:14.976 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:14.977 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:14.977 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:14.978 DEBUG:boto:Method: GET 2022-09-09T22:38:14.978 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:14.979 DEBUG:boto:Data: 2022-09-09T22:38:14.980 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.980 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:14.981 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.981 DEBUG:boto:Params: {} 2022-09-09T22:38:14.982 DEBUG:boto:Token: None 2022-09-09T22:38:14.982 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:14.983 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:10M6cUifV2KT+cBtWYY3Jn1zPeI= 2022-09-09T22:38:14.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:10M6cUifV2KT+cBtWYY3Jn1zPeI=', 'Content-Length': '0'} 2022-09-09T22:38:14.987 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002204ffc5e454a5c7-00631bc056-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:14.988 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:14.988 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:14.989 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:14.989 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:14.990 DEBUG:boto:Method: HEAD 2022-09-09T22:38:14.990 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:14.991 DEBUG:boto:Data: 2022-09-09T22:38:14.991 DEBUG:boto:Headers: {} 2022-09-09T22:38:14.992 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:14.992 DEBUG:boto:Port: 8000 2022-09-09T22:38:14.993 DEBUG:boto:Params: {} 2022-09-09T22:38:14.993 DEBUG:boto:Token: None 2022-09-09T22:38:14.994 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:14 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:14.995 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9w+hBgMjc+R3ES8J1u+qZeexRDA= 2022-09-09T22:38:14.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9w+hBgMjc+R3ES8J1u+qZeexRDA=', 'Content-Length': '0'} 2022-09-09T22:38:14.999 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e02baacd2524681-00631bc056-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.000 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:15.000 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:15.001 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:15.001 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:15.002 DEBUG:boto:Method: HEAD 2022-09-09T22:38:15.002 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:15.003 DEBUG:boto:Data: 2022-09-09T22:38:15.003 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.004 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:15.005 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.005 DEBUG:boto:Params: {} 2022-09-09T22:38:15.006 DEBUG:boto:Token: None 2022-09-09T22:38:15.007 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:15.007 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4p6fPW4dT7jZZDsUXMOzVv0DNyo= 2022-09-09T22:38:15.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4p6fPW4dT7jZZDsUXMOzVv0DNyo=', 'Content-Length': '0'} 2022-09-09T22:38:15.011 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f9cb9e4566e0ba0-00631bc057-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:38:15.012 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:15.013 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:15.013 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:15.014 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:15.014 DEBUG:boto:Method: GET 2022-09-09T22:38:15.015 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:15.015 DEBUG:boto:Data: 2022-09-09T22:38:15.016 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.016 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:15.017 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.017 DEBUG:boto:Params: {} 2022-09-09T22:38:15.018 DEBUG:boto:Token: None 2022-09-09T22:38:15.019 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:15.019 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:tPmjP9+6MQJlXjss+werodPzLEw= 2022-09-09T22:38:15.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:tPmjP9+6MQJlXjss+werodPzLEw=', 'Content-Length': '0'} 2022-09-09T22:38:15.023 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d52deec22077b53b-00631bc057-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.066 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:15.067 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:15.067 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:15.068 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:15.069 DEBUG:boto:Method: GET 2022-09-09T22:38:15.070 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:15.071 DEBUG:boto:Data: 2022-09-09T22:38:15.072 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.072 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:15.073 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.073 DEBUG:boto:Params: {} 2022-09-09T22:38:15.074 DEBUG:boto:Token: None 2022-09-09T22:38:15.075 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:15.075 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:tPmjP9+6MQJlXjss+werodPzLEw= 2022-09-09T22:38:15.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:tPmjP9+6MQJlXjss+werodPzLEw=', 'Content-Length': '0'} 2022-09-09T22:38:15.079 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a16d8c6ab887c90e-00631bc057-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.081 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:15.082 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:15.083 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:15.084 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:15.085 DEBUG:boto:Method: GET 2022-09-09T22:38:15.086 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:15.087 DEBUG:boto:Data: 2022-09-09T22:38:15.088 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.089 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:15.090 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.091 DEBUG:boto:Params: {} 2022-09-09T22:38:15.092 DEBUG:boto:Token: None 2022-09-09T22:38:15.093 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/ 2022-09-09T22:38:15.094 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:D7/sMWTrpMI6Ib6dVIhx5RFMla0= 2022-09-09T22:38:15.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:D7/sMWTrpMI6Ib6dVIhx5RFMla0=', 'Content-Length': '0'} 2022-09-09T22:38:15.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6b5bc012fc09a12-00631bc057-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.102 DEBUG:boto:b'jkcgbj-18obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:15.103 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:15.104 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:15.104 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:15.105 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:15.106 DEBUG:boto:Method: GET 2022-09-09T22:38:15.106 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:15.107 DEBUG:boto:Data: 2022-09-09T22:38:15.107 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.108 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:15.108 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.109 DEBUG:boto:Params: {} 2022-09-09T22:38:15.110 DEBUG:boto:Token: None 2022-09-09T22:38:15.110 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/ 2022-09-09T22:38:15.111 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:D7/sMWTrpMI6Ib6dVIhx5RFMla0= 2022-09-09T22:38:15.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:D7/sMWTrpMI6Ib6dVIhx5RFMla0=', 'Content-Length': '0'} 2022-09-09T22:38:15.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021bff7b0fd3dd214-00631bc057-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.158 DEBUG:boto:b'jkcgbj-18obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:15.159 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:15.160 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:15.160 DEBUG:boto:Method: HEAD 2022-09-09T22:38:15.161 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2 2022-09-09T22:38:15.161 DEBUG:boto:Data: 2022-09-09T22:38:15.162 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.162 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:15.163 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.163 DEBUG:boto:Params: {} 2022-09-09T22:38:15.164 DEBUG:boto:Token: None 2022-09-09T22:38:15.164 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/obj-test-zone2 2022-09-09T22:38:15.165 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:J9jbVvgt4Arvk77dT7mubtbDIG4= 2022-09-09T22:38:15.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:J9jbVvgt4Arvk77dT7mubtbDIG4=', 'Content-Length': '0'} 2022-09-09T22:38:15.170 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000efdbc5d14bc52b3a-00631bc057-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.170 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:15.171 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:15.172 DEBUG:boto:Method: HEAD 2022-09-09T22:38:15.172 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2 2022-09-09T22:38:15.173 DEBUG:boto:Data: 2022-09-09T22:38:15.173 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.174 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:15.174 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.175 DEBUG:boto:Params: {} 2022-09-09T22:38:15.175 DEBUG:boto:Token: None 2022-09-09T22:38:15.176 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/obj-test-zone2 2022-09-09T22:38:15.176 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:J9jbVvgt4Arvk77dT7mubtbDIG4= 2022-09-09T22:38:15.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:J9jbVvgt4Arvk77dT7mubtbDIG4=', 'Content-Length': '0'} 2022-09-09T22:38:15.181 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000865572648e3ed3e2-00631bc057-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:38:15.182 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:15.183 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:15.183 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:15.184 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:15.184 DEBUG:boto:Method: GET 2022-09-09T22:38:15.185 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:15.185 DEBUG:boto:Data: 2022-09-09T22:38:15.186 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.186 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:15.187 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.187 DEBUG:boto:Params: {} 2022-09-09T22:38:15.188 DEBUG:boto:Token: None 2022-09-09T22:38:15.188 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:15.189 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:tPmjP9+6MQJlXjss+werodPzLEw= 2022-09-09T22:38:15.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:tPmjP9+6MQJlXjss+werodPzLEw=', 'Content-Length': '0'} 2022-09-09T22:38:15.193 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5f25d6a1de00d0d-00631bc057-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.194 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:15.194 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:15.195 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:15.196 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:15.196 DEBUG:boto:Method: GET 2022-09-09T22:38:15.197 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:15.197 DEBUG:boto:Data: 2022-09-09T22:38:15.198 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.198 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:15.199 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.200 DEBUG:boto:Params: {} 2022-09-09T22:38:15.200 DEBUG:boto:Token: None 2022-09-09T22:38:15.201 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:15.201 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:tPmjP9+6MQJlXjss+werodPzLEw= 2022-09-09T22:38:15.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:tPmjP9+6MQJlXjss+werodPzLEw=', 'Content-Length': '0'} 2022-09-09T22:38:15.205 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049edd64a93696376-00631bc057-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.206 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:15.207 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:15.207 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:15.208 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:15.208 DEBUG:boto:Method: GET 2022-09-09T22:38:15.209 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:15.209 DEBUG:boto:Data: 2022-09-09T22:38:15.210 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.211 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:15.211 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.212 DEBUG:boto:Params: {} 2022-09-09T22:38:15.212 DEBUG:boto:Token: None 2022-09-09T22:38:15.213 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/ 2022-09-09T22:38:15.213 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:D7/sMWTrpMI6Ib6dVIhx5RFMla0= 2022-09-09T22:38:15.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:D7/sMWTrpMI6Ib6dVIhx5RFMla0=', 'Content-Length': '0'} 2022-09-09T22:38:15.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ffe1116c230181d-00631bc057-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.220 DEBUG:boto:b'jkcgbj-18obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:15.221 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:15.222 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:15.222 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:15.223 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:15.223 DEBUG:boto:Method: GET 2022-09-09T22:38:15.224 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:15.224 DEBUG:boto:Data: 2022-09-09T22:38:15.225 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.226 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:15.226 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.227 DEBUG:boto:Params: {} 2022-09-09T22:38:15.227 DEBUG:boto:Token: None 2022-09-09T22:38:15.228 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/ 2022-09-09T22:38:15.228 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:D7/sMWTrpMI6Ib6dVIhx5RFMla0= 2022-09-09T22:38:15.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:D7/sMWTrpMI6Ib6dVIhx5RFMla0=', 'Content-Length': '0'} 2022-09-09T22:38:15.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001b3a16569839996-00631bc057-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.235 DEBUG:boto:b'jkcgbj-18obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:15.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:38:15.237 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:15.237 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:15.238 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.238 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.239 DEBUG:boto:Method: GET 2022-09-09T22:38:15.239 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.240 DEBUG:boto:Data: 2022-09-09T22:38:15.240 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.241 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:15.241 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.242 DEBUG:boto:Params: {} 2022-09-09T22:38:15.243 DEBUG:boto:Token: None 2022-09-09T22:38:15.243 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.244 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nMqV7tVvnrRJ+Bw0zLVLYXc7DHI= 2022-09-09T22:38:15.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nMqV7tVvnrRJ+Bw0zLVLYXc7DHI=', 'Content-Length': '0'} 2022-09-09T22:38:15.248 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac785118ec49fa53-00631bc057-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.248 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:15.249 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:15.249 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.250 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.251 DEBUG:boto:Method: GET 2022-09-09T22:38:15.251 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.252 DEBUG:boto:Data: 2022-09-09T22:38:15.252 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.253 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:15.253 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.254 DEBUG:boto:Params: {} 2022-09-09T22:38:15.254 DEBUG:boto:Token: None 2022-09-09T22:38:15.255 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.255 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nMqV7tVvnrRJ+Bw0zLVLYXc7DHI= 2022-09-09T22:38:15.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nMqV7tVvnrRJ+Bw0zLVLYXc7DHI=', 'Content-Length': '0'} 2022-09-09T22:38:15.259 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff1cd17dfd8be03c-00631bc057-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.260 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:15.261 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:15.261 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.262 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.262 DEBUG:boto:Method: HEAD 2022-09-09T22:38:15.263 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.263 DEBUG:boto:Data: 2022-09-09T22:38:15.264 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.264 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:15.265 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.265 DEBUG:boto:Params: {} 2022-09-09T22:38:15.266 DEBUG:boto:Token: None 2022-09-09T22:38:15.266 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.267 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8giqg9QQR4ule2Npc192YsLQEyw= 2022-09-09T22:38:15.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8giqg9QQR4ule2Npc192YsLQEyw=', 'Content-Length': '0'} 2022-09-09T22:38:15.270 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003bd7b874afbff180-00631bc057-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.271 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:15.272 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:15.272 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.273 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.273 DEBUG:boto:Method: HEAD 2022-09-09T22:38:15.274 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.274 DEBUG:boto:Data: 2022-09-09T22:38:15.275 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.275 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:15.276 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.276 DEBUG:boto:Params: {} 2022-09-09T22:38:15.277 DEBUG:boto:Token: None 2022-09-09T22:38:15.277 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.278 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8giqg9QQR4ule2Npc192YsLQEyw= 2022-09-09T22:38:15.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8giqg9QQR4ule2Npc192YsLQEyw=', 'Content-Length': '0'} 2022-09-09T22:38:15.282 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035ce94c743d9d2a8-00631bc057-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:38:15.283 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:15.284 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:15.284 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.285 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.285 DEBUG:boto:Method: GET 2022-09-09T22:38:15.286 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.286 DEBUG:boto:Data: 2022-09-09T22:38:15.287 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.287 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:15.288 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.288 DEBUG:boto:Params: {} 2022-09-09T22:38:15.289 DEBUG:boto:Token: None 2022-09-09T22:38:15.290 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.290 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nMqV7tVvnrRJ+Bw0zLVLYXc7DHI= 2022-09-09T22:38:15.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nMqV7tVvnrRJ+Bw0zLVLYXc7DHI=', 'Content-Length': '0'} 2022-09-09T22:38:15.294 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000020318e7f2a3ed207-00631bc057-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.342 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:15.343 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:15.343 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.344 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.344 DEBUG:boto:Method: GET 2022-09-09T22:38:15.345 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.345 DEBUG:boto:Data: 2022-09-09T22:38:15.346 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.347 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:15.347 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.348 DEBUG:boto:Params: {} 2022-09-09T22:38:15.348 DEBUG:boto:Token: None 2022-09-09T22:38:15.349 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.350 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nMqV7tVvnrRJ+Bw0zLVLYXc7DHI= 2022-09-09T22:38:15.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nMqV7tVvnrRJ+Bw0zLVLYXc7DHI=', 'Content-Length': '0'} 2022-09-09T22:38:15.354 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000578318d3619ac7ff-00631bc057-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.355 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:15.355 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:15.356 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:15.357 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:15.357 DEBUG:boto:Method: GET 2022-09-09T22:38:15.358 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:15.359 DEBUG:boto:Data: 2022-09-09T22:38:15.359 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.360 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:15.360 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.361 DEBUG:boto:Params: {} 2022-09-09T22:38:15.362 DEBUG:boto:Token: None 2022-09-09T22:38:15.363 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/ 2022-09-09T22:38:15.363 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:D7/sMWTrpMI6Ib6dVIhx5RFMla0= 2022-09-09T22:38:15.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:D7/sMWTrpMI6Ib6dVIhx5RFMla0=', 'Content-Length': '0'} 2022-09-09T22:38:15.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c72c369d32c090f-00631bc057-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.369 DEBUG:boto:b'jkcgbj-18obj-test-zone31000falseobj-test-zone32022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:15.370 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:15.371 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:15.371 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:15.372 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:15.372 DEBUG:boto:Method: GET 2022-09-09T22:38:15.373 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:15.373 DEBUG:boto:Data: 2022-09-09T22:38:15.374 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.374 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:15.375 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.375 DEBUG:boto:Params: {} 2022-09-09T22:38:15.376 DEBUG:boto:Token: None 2022-09-09T22:38:15.377 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/ 2022-09-09T22:38:15.377 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:D7/sMWTrpMI6Ib6dVIhx5RFMla0= 2022-09-09T22:38:15.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:D7/sMWTrpMI6Ib6dVIhx5RFMla0=', 'Content-Length': '0'} 2022-09-09T22:38:15.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000826101a0d3e1eb2b-00631bc057-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.384 DEBUG:boto:b'jkcgbj-18obj-test-zone31000falseobj-test-zone32022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:15.384 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:15.385 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:15.385 DEBUG:boto:Method: HEAD 2022-09-09T22:38:15.386 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3 2022-09-09T22:38:15.387 DEBUG:boto:Data: 2022-09-09T22:38:15.387 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.388 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:15.388 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.389 DEBUG:boto:Params: {} 2022-09-09T22:38:15.389 DEBUG:boto:Token: None 2022-09-09T22:38:15.390 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/obj-test-zone3 2022-09-09T22:38:15.390 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:XFB0qloJCfFLcgtBhZzK5VJ2l1w= 2022-09-09T22:38:15.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:XFB0qloJCfFLcgtBhZzK5VJ2l1w=', 'Content-Length': '0'} 2022-09-09T22:38:15.395 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f3879f70cbd5a21-00631bc057-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.395 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:15.396 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:15.396 DEBUG:boto:Method: HEAD 2022-09-09T22:38:15.397 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3 2022-09-09T22:38:15.397 DEBUG:boto:Data: 2022-09-09T22:38:15.398 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.398 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:15.399 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.399 DEBUG:boto:Params: {} 2022-09-09T22:38:15.400 DEBUG:boto:Token: None 2022-09-09T22:38:15.401 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/obj-test-zone3 2022-09-09T22:38:15.401 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:XFB0qloJCfFLcgtBhZzK5VJ2l1w= 2022-09-09T22:38:15.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:XFB0qloJCfFLcgtBhZzK5VJ2l1w=', 'Content-Length': '0'} 2022-09-09T22:38:15.405 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000080d35cace5875f0b-00631bc057-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:38:15.407 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:15.407 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:15.408 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.408 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.409 DEBUG:boto:Method: GET 2022-09-09T22:38:15.409 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.410 DEBUG:boto:Data: 2022-09-09T22:38:15.410 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.411 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:15.411 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.412 DEBUG:boto:Params: {} 2022-09-09T22:38:15.412 DEBUG:boto:Token: None 2022-09-09T22:38:15.413 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.414 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nMqV7tVvnrRJ+Bw0zLVLYXc7DHI= 2022-09-09T22:38:15.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nMqV7tVvnrRJ+Bw0zLVLYXc7DHI=', 'Content-Length': '0'} 2022-09-09T22:38:15.417 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000096266d2390f71c51-00631bc057-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.458 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:15.459 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:15.459 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.460 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.460 DEBUG:boto:Method: GET 2022-09-09T22:38:15.461 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.462 DEBUG:boto:Data: 2022-09-09T22:38:15.462 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.463 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:15.463 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.464 DEBUG:boto:Params: {} 2022-09-09T22:38:15.465 DEBUG:boto:Token: None 2022-09-09T22:38:15.465 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:15.466 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nMqV7tVvnrRJ+Bw0zLVLYXc7DHI= 2022-09-09T22:38:15.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nMqV7tVvnrRJ+Bw0zLVLYXc7DHI=', 'Content-Length': '0'} 2022-09-09T22:38:15.470 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000610507309e239147-00631bc057-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.471 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:15.472 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:15.473 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:15.474 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:15.474 DEBUG:boto:Method: GET 2022-09-09T22:38:15.475 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:15.475 DEBUG:boto:Data: 2022-09-09T22:38:15.476 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.476 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:15.477 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.477 DEBUG:boto:Params: {} 2022-09-09T22:38:15.478 DEBUG:boto:Token: None 2022-09-09T22:38:15.478 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/ 2022-09-09T22:38:15.479 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:D7/sMWTrpMI6Ib6dVIhx5RFMla0= 2022-09-09T22:38:15.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:D7/sMWTrpMI6Ib6dVIhx5RFMla0=', 'Content-Length': '0'} 2022-09-09T22:38:15.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073e68f58fc01b9da-00631bc057-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.486 DEBUG:boto:b'jkcgbj-18obj-test-zone31000falseobj-test-zone32022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:15.487 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:15.488 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:15.488 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:15.489 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:15.489 DEBUG:boto:Method: GET 2022-09-09T22:38:15.490 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:15.490 DEBUG:boto:Data: 2022-09-09T22:38:15.491 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.491 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:15.492 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.492 DEBUG:boto:Params: {} 2022-09-09T22:38:15.493 DEBUG:boto:Token: None 2022-09-09T22:38:15.493 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/ 2022-09-09T22:38:15.494 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:D7/sMWTrpMI6Ib6dVIhx5RFMla0= 2022-09-09T22:38:15.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:D7/sMWTrpMI6Ib6dVIhx5RFMla0=', 'Content-Length': '0'} 2022-09-09T22:38:15.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2dc6c255f11ec87-00631bc057-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.501 DEBUG:boto:b'jkcgbj-18obj-test-zone31000falseobj-test-zone32022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:15.502 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-18 zones={test-zone2, test-zone1} 2022-09-09T22:38:15.502 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-18 zones={test-zone3, test-zone1} 2022-09-09T22:38:15.503 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:15.503 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:15.504 DEBUG:boto:Method: HEAD 2022-09-09T22:38:15.504 DEBUG:boto:Path: /jkcgbj-18/ 2022-09-09T22:38:15.505 DEBUG:boto:Data: 2022-09-09T22:38:15.505 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.506 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:15.506 DEBUG:boto:Port: 8001 2022-09-09T22:38:15.507 DEBUG:boto:Params: {} 2022-09-09T22:38:15.507 DEBUG:boto:Token: None 2022-09-09T22:38:15.508 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/ 2022-09-09T22:38:15.509 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ZgxfEVhoNExM1yYQfqh7OqpAizw= 2022-09-09T22:38:15.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ZgxfEVhoNExM1yYQfqh7OqpAizw=', 'Content-Length': '0'} 2022-09-09T22:38:15.513 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000003e70dd7dcdf8c71-00631bc057-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.514 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:15.515 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:15.515 DEBUG:boto:Method: HEAD 2022-09-09T22:38:15.516 DEBUG:boto:Path: /jkcgbj-18/ 2022-09-09T22:38:15.516 DEBUG:boto:Data: 2022-09-09T22:38:15.517 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.517 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:15.518 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.518 DEBUG:boto:Params: {} 2022-09-09T22:38:15.519 DEBUG:boto:Token: None 2022-09-09T22:38:15.519 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/ 2022-09-09T22:38:15.520 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ZgxfEVhoNExM1yYQfqh7OqpAizw= 2022-09-09T22:38:15.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ZgxfEVhoNExM1yYQfqh7OqpAizw=', 'Content-Length': '0'} 2022-09-09T22:38:15.525 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000074faee4c387bc7d9-00631bc057-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.525 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:38:15.526 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:15.527 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:15.527 DEBUG:boto:path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:15.528 DEBUG:boto:auth_path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:15.528 DEBUG:boto:Method: GET 2022-09-09T22:38:15.529 DEBUG:boto:Path: /jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:15.529 DEBUG:boto:Data: 2022-09-09T22:38:15.530 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.530 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:15.531 DEBUG:boto:Port: 8001 2022-09-09T22:38:15.531 DEBUG:boto:Params: {} 2022-09-09T22:38:15.532 DEBUG:boto:Token: None 2022-09-09T22:38:15.533 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/?versions 2022-09-09T22:38:15.533 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JugT6SRsmH+EiwLUK40hKnlNa0s= 2022-09-09T22:38:15.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JugT6SRsmH+EiwLUK40hKnlNa0s=', 'Content-Length': '0'} 2022-09-09T22:38:15.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000113636eab7c63296-00631bc057-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.540 DEBUG:boto:b'jkcgbj-18999falsedummyRQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5true2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1a8lrwM74XPlF1mwXf8hNl9Y9ISdV53ltrue2022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2e.gQ6yFECscD-oc4OV0dmk6aAg9jH.otrue2022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1true2022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:15.541 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:38:15.541 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:38:15.542 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:38:15.542 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:38:15.543 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:38:15.543 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:15.544 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:15.544 DEBUG:boto:path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:15.545 DEBUG:boto:auth_path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:15.545 DEBUG:boto:Method: GET 2022-09-09T22:38:15.546 DEBUG:boto:Path: /jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:15.546 DEBUG:boto:Data: 2022-09-09T22:38:15.547 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.547 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:15.548 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.548 DEBUG:boto:Params: {} 2022-09-09T22:38:15.549 DEBUG:boto:Token: None 2022-09-09T22:38:15.550 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/?versions 2022-09-09T22:38:15.550 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JugT6SRsmH+EiwLUK40hKnlNa0s= 2022-09-09T22:38:15.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JugT6SRsmH+EiwLUK40hKnlNa0s=', 'Content-Length': '0'} 2022-09-09T22:38:15.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004836e78df85de9b7-00631bc057-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.558 DEBUG:boto:b'jkcgbj-18999falsedummyRQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5true2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1a8lrwM74XPlF1mwXf8hNl9Y9ISdV53ltrue2022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2e.gQ6yFECscD-oc4OV0dmk6aAg9jH.otrue2022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1true2022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:15.558 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:38:15.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:38:15.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:38:15.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:38:15.560 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:15.561 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:15.562 DEBUG:boto:path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:15.562 DEBUG:boto:auth_path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:15.563 DEBUG:boto:Method: GET 2022-09-09T22:38:15.563 DEBUG:boto:Path: /jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:15.564 DEBUG:boto:Data: 2022-09-09T22:38:15.564 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.565 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:15.565 DEBUG:boto:Port: 8001 2022-09-09T22:38:15.566 DEBUG:boto:Params: {} 2022-09-09T22:38:15.566 DEBUG:boto:Token: None 2022-09-09T22:38:15.567 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/?versions 2022-09-09T22:38:15.568 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JugT6SRsmH+EiwLUK40hKnlNa0s= 2022-09-09T22:38:15.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JugT6SRsmH+EiwLUK40hKnlNa0s=', 'Content-Length': '0'} 2022-09-09T22:38:15.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebb2195544759577-00631bc057-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.574 DEBUG:boto:b'jkcgbj-18999falsedummyRQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5true2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1a8lrwM74XPlF1mwXf8hNl9Y9ISdV53ltrue2022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2e.gQ6yFECscD-oc4OV0dmk6aAg9jH.otrue2022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1true2022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:15.575 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:15.576 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:15.576 DEBUG:boto:path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:15.577 DEBUG:boto:auth_path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:15.577 DEBUG:boto:Method: GET 2022-09-09T22:38:15.578 DEBUG:boto:Path: /jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:15.578 DEBUG:boto:Data: 2022-09-09T22:38:15.579 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.579 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:15.580 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.580 DEBUG:boto:Params: {} 2022-09-09T22:38:15.581 DEBUG:boto:Token: None 2022-09-09T22:38:15.581 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/?versions 2022-09-09T22:38:15.582 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JugT6SRsmH+EiwLUK40hKnlNa0s= 2022-09-09T22:38:15.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JugT6SRsmH+EiwLUK40hKnlNa0s=', 'Content-Length': '0'} 2022-09-09T22:38:15.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f957d893873be708-00631bc057-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.589 DEBUG:boto:b'jkcgbj-18999falsedummyRQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5true2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1a8lrwM74XPlF1mwXf8hNl9Y9ISdV53ltrue2022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2e.gQ6yFECscD-oc4OV0dmk6aAg9jH.otrue2022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1true2022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:15.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:38:15.590 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:38:15.591 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:38:15.591 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.592 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.593 DEBUG:boto:Method: GET 2022-09-09T22:38:15.593 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.594 DEBUG:boto:Data: 2022-09-09T22:38:15.594 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.595 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:15.595 DEBUG:boto:Port: 8001 2022-09-09T22:38:15.596 DEBUG:boto:Params: {} 2022-09-09T22:38:15.597 DEBUG:boto:Token: None 2022-09-09T22:38:15.597 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.598 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wEuSqn222AAosYcVITHIf53SXX0= 2022-09-09T22:38:15.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wEuSqn222AAosYcVITHIf53SXX0=', 'Content-Length': '0'} 2022-09-09T22:38:15.602 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d80b3e7f343ebd73-00631bc057-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.603 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:38:15.603 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:38:15.604 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.604 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.605 DEBUG:boto:Method: GET 2022-09-09T22:38:15.605 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.606 DEBUG:boto:Data: 2022-09-09T22:38:15.607 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.607 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:15.608 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.608 DEBUG:boto:Params: {} 2022-09-09T22:38:15.609 DEBUG:boto:Token: None 2022-09-09T22:38:15.609 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.610 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wEuSqn222AAosYcVITHIf53SXX0= 2022-09-09T22:38:15.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wEuSqn222AAosYcVITHIf53SXX0=', 'Content-Length': '0'} 2022-09-09T22:38:15.613 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bd383475f51a5b63-00631bc057-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.614 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:38:15.615 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:38:15.615 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.616 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.616 DEBUG:boto:Method: HEAD 2022-09-09T22:38:15.617 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.617 DEBUG:boto:Data: 2022-09-09T22:38:15.618 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.618 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:15.619 DEBUG:boto:Port: 8001 2022-09-09T22:38:15.620 DEBUG:boto:Params: {} 2022-09-09T22:38:15.621 DEBUG:boto:Token: None 2022-09-09T22:38:15.621 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.622 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jkUemv9oWJn+tGNBaLXPn4AvUNA= 2022-09-09T22:38:15.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jkUemv9oWJn+tGNBaLXPn4AvUNA=', 'Content-Length': '0'} 2022-09-09T22:38:15.627 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dd64944076f34fb9-00631bc057-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.628 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:38:15.629 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:38:15.630 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.631 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.632 DEBUG:boto:Method: HEAD 2022-09-09T22:38:15.633 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.634 DEBUG:boto:Data: 2022-09-09T22:38:15.635 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.636 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:15.636 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.637 DEBUG:boto:Params: {} 2022-09-09T22:38:15.638 DEBUG:boto:Token: None 2022-09-09T22:38:15.639 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.640 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jkUemv9oWJn+tGNBaLXPn4AvUNA= 2022-09-09T22:38:15.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jkUemv9oWJn+tGNBaLXPn4AvUNA=', 'Content-Length': '0'} 2022-09-09T22:38:15.643 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f65aae6b1d7afac2-00631bc057-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:38:15.644 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:38:15.645 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:38:15.646 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.646 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.647 DEBUG:boto:Method: GET 2022-09-09T22:38:15.648 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.648 DEBUG:boto:Data: 2022-09-09T22:38:15.649 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.649 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:15.650 DEBUG:boto:Port: 8001 2022-09-09T22:38:15.650 DEBUG:boto:Params: {} 2022-09-09T22:38:15.651 DEBUG:boto:Token: None 2022-09-09T22:38:15.651 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.652 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wEuSqn222AAosYcVITHIf53SXX0= 2022-09-09T22:38:15.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wEuSqn222AAosYcVITHIf53SXX0=', 'Content-Length': '0'} 2022-09-09T22:38:15.655 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e408679a37757c8d-00631bc057-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.655 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:38:15.656 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:38:15.657 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.657 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.658 DEBUG:boto:Method: GET 2022-09-09T22:38:15.658 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.659 DEBUG:boto:Data: 2022-09-09T22:38:15.660 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.661 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:15.661 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.662 DEBUG:boto:Params: {} 2022-09-09T22:38:15.663 DEBUG:boto:Token: None 2022-09-09T22:38:15.664 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.664 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wEuSqn222AAosYcVITHIf53SXX0= 2022-09-09T22:38:15.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wEuSqn222AAosYcVITHIf53SXX0=', 'Content-Length': '0'} 2022-09-09T22:38:15.667 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c9dc97bcbb6cb188-00631bc057-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.668 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:15.669 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:15.669 DEBUG:boto:path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:38:15.670 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:38:15.670 DEBUG:boto:Method: GET 2022-09-09T22:38:15.671 DEBUG:boto:Path: /jkcgbj-18/?prefix=dummy 2022-09-09T22:38:15.671 DEBUG:boto:Data: 2022-09-09T22:38:15.672 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.672 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:15.673 DEBUG:boto:Port: 8001 2022-09-09T22:38:15.674 DEBUG:boto:Params: {} 2022-09-09T22:38:15.675 DEBUG:boto:Token: None 2022-09-09T22:38:15.676 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/ 2022-09-09T22:38:15.677 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:D7/sMWTrpMI6Ib6dVIhx5RFMla0= 2022-09-09T22:38:15.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:D7/sMWTrpMI6Ib6dVIhx5RFMla0=', 'Content-Length': '0'} 2022-09-09T22:38:15.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000304ff50f618479d5-00631bc057-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.730 DEBUG:boto:b'jkcgbj-18dummy1000falsedummy2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:15.731 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:15.732 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:15.732 DEBUG:boto:path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:38:15.733 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:38:15.733 DEBUG:boto:Method: GET 2022-09-09T22:38:15.734 DEBUG:boto:Path: /jkcgbj-18/?prefix=dummy 2022-09-09T22:38:15.734 DEBUG:boto:Data: 2022-09-09T22:38:15.735 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.735 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:15.736 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.736 DEBUG:boto:Params: {} 2022-09-09T22:38:15.737 DEBUG:boto:Token: None 2022-09-09T22:38:15.738 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/ 2022-09-09T22:38:15.738 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:D7/sMWTrpMI6Ib6dVIhx5RFMla0= 2022-09-09T22:38:15.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:D7/sMWTrpMI6Ib6dVIhx5RFMla0=', 'Content-Length': '0'} 2022-09-09T22:38:15.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c50009b1f722b54-00631bc057-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.745 DEBUG:boto:b'jkcgbj-18dummy1000falsedummy2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:15.745 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:38:15.746 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:38:15.746 DEBUG:boto:Method: HEAD 2022-09-09T22:38:15.747 DEBUG:boto:Path: /jkcgbj-18/dummy 2022-09-09T22:38:15.748 DEBUG:boto:Data: 2022-09-09T22:38:15.748 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.749 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:15.749 DEBUG:boto:Port: 8001 2022-09-09T22:38:15.750 DEBUG:boto:Params: {} 2022-09-09T22:38:15.750 DEBUG:boto:Token: None 2022-09-09T22:38:15.751 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/dummy 2022-09-09T22:38:15.751 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:kKaEcHahnECUs5aDaAi2HeA1CQQ= 2022-09-09T22:38:15.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:kKaEcHahnECUs5aDaAi2HeA1CQQ=', 'Content-Length': '0'} 2022-09-09T22:38:15.755 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000626fa9d2182f39ea-00631bc057-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.756 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:38:15.756 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:38:15.757 DEBUG:boto:Method: HEAD 2022-09-09T22:38:15.757 DEBUG:boto:Path: /jkcgbj-18/dummy 2022-09-09T22:38:15.758 DEBUG:boto:Data: 2022-09-09T22:38:15.759 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.759 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:15.760 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.760 DEBUG:boto:Params: {} 2022-09-09T22:38:15.761 DEBUG:boto:Token: None 2022-09-09T22:38:15.762 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/dummy 2022-09-09T22:38:15.762 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:kKaEcHahnECUs5aDaAi2HeA1CQQ= 2022-09-09T22:38:15.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:kKaEcHahnECUs5aDaAi2HeA1CQQ=', 'Content-Length': '0'} 2022-09-09T22:38:15.767 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b666fb8b6c7fe11c-00631bc057-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:38:15.768 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:38:15.769 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:38:15.770 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.770 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.771 DEBUG:boto:Method: GET 2022-09-09T22:38:15.771 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.772 DEBUG:boto:Data: 2022-09-09T22:38:15.773 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.774 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:15.775 DEBUG:boto:Port: 8001 2022-09-09T22:38:15.776 DEBUG:boto:Params: {} 2022-09-09T22:38:15.777 DEBUG:boto:Token: None 2022-09-09T22:38:15.778 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.778 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wEuSqn222AAosYcVITHIf53SXX0= 2022-09-09T22:38:15.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wEuSqn222AAosYcVITHIf53SXX0=', 'Content-Length': '0'} 2022-09-09T22:38:15.782 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000018033b809865e60-00631bc057-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.783 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:38:15.783 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:38:15.784 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.784 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.785 DEBUG:boto:Method: GET 2022-09-09T22:38:15.785 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.786 DEBUG:boto:Data: 2022-09-09T22:38:15.787 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.787 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:15.788 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.789 DEBUG:boto:Params: {} 2022-09-09T22:38:15.790 DEBUG:boto:Token: None 2022-09-09T22:38:15.791 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:15.792 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wEuSqn222AAosYcVITHIf53SXX0= 2022-09-09T22:38:15.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wEuSqn222AAosYcVITHIf53SXX0=', 'Content-Length': '0'} 2022-09-09T22:38:15.795 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000080d1a6ef85091f68-00631bc057-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.796 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:15.797 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:15.797 DEBUG:boto:path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:38:15.798 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:38:15.798 DEBUG:boto:Method: GET 2022-09-09T22:38:15.799 DEBUG:boto:Path: /jkcgbj-18/?prefix=dummy 2022-09-09T22:38:15.800 DEBUG:boto:Data: 2022-09-09T22:38:15.800 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.801 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:15.801 DEBUG:boto:Port: 8001 2022-09-09T22:38:15.802 DEBUG:boto:Params: {} 2022-09-09T22:38:15.802 DEBUG:boto:Token: None 2022-09-09T22:38:15.803 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/ 2022-09-09T22:38:15.804 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:D7/sMWTrpMI6Ib6dVIhx5RFMla0= 2022-09-09T22:38:15.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:D7/sMWTrpMI6Ib6dVIhx5RFMla0=', 'Content-Length': '0'} 2022-09-09T22:38:15.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c9364d42f3441e3-00631bc057-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.854 DEBUG:boto:b'jkcgbj-18dummy1000falsedummy2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:15.855 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:15.856 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:15.856 DEBUG:boto:path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:38:15.857 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:38:15.857 DEBUG:boto:Method: GET 2022-09-09T22:38:15.858 DEBUG:boto:Path: /jkcgbj-18/?prefix=dummy 2022-09-09T22:38:15.858 DEBUG:boto:Data: 2022-09-09T22:38:15.859 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.859 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:15.860 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.860 DEBUG:boto:Params: {} 2022-09-09T22:38:15.861 DEBUG:boto:Token: None 2022-09-09T22:38:15.862 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/ 2022-09-09T22:38:15.862 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:D7/sMWTrpMI6Ib6dVIhx5RFMla0= 2022-09-09T22:38:15.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:D7/sMWTrpMI6Ib6dVIhx5RFMla0=', 'Content-Length': '0'} 2022-09-09T22:38:15.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081b0d6e37b16d8d5-00631bc057-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.870 DEBUG:boto:b'jkcgbj-18dummy1000falsedummy2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:15.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:38:15.872 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:15.872 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:15.873 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:15.874 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:15.874 DEBUG:boto:Method: GET 2022-09-09T22:38:15.875 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:15.876 DEBUG:boto:Data: 2022-09-09T22:38:15.876 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.877 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:15.878 DEBUG:boto:Port: 8001 2022-09-09T22:38:15.878 DEBUG:boto:Params: {} 2022-09-09T22:38:15.879 DEBUG:boto:Token: None 2022-09-09T22:38:15.880 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:15.880 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:n32LZ0Vq8NNbMu6RiPSLK6IRfAw= 2022-09-09T22:38:15.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:n32LZ0Vq8NNbMu6RiPSLK6IRfAw=', 'Content-Length': '0'} 2022-09-09T22:38:15.884 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af3de0c751ebd11c-00631bc057-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.886 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:15.886 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:15.887 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:15.887 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:15.888 DEBUG:boto:Method: GET 2022-09-09T22:38:15.889 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:15.889 DEBUG:boto:Data: 2022-09-09T22:38:15.890 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.891 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:15.892 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.893 DEBUG:boto:Params: {} 2022-09-09T22:38:15.894 DEBUG:boto:Token: None 2022-09-09T22:38:15.895 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:15.897 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:n32LZ0Vq8NNbMu6RiPSLK6IRfAw= 2022-09-09T22:38:15.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:n32LZ0Vq8NNbMu6RiPSLK6IRfAw=', 'Content-Length': '0'} 2022-09-09T22:38:15.901 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000960b37ef9ce01f2d-00631bc057-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.946 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:15.947 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:15.947 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:15.948 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:15.948 DEBUG:boto:Method: HEAD 2022-09-09T22:38:15.949 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:15.949 DEBUG:boto:Data: 2022-09-09T22:38:15.950 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.950 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:15.951 DEBUG:boto:Port: 8001 2022-09-09T22:38:15.951 DEBUG:boto:Params: {} 2022-09-09T22:38:15.952 DEBUG:boto:Token: None 2022-09-09T22:38:15.953 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:15.953 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Q67bpZkWtNaAE71xhuN9bbO7I3I= 2022-09-09T22:38:15.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Q67bpZkWtNaAE71xhuN9bbO7I3I=', 'Content-Length': '0'} 2022-09-09T22:38:15.957 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077fdd9cd95d6e139-00631bc057-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.958 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:15.959 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:15.959 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:15.960 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:15.960 DEBUG:boto:Method: HEAD 2022-09-09T22:38:15.961 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:15.961 DEBUG:boto:Data: 2022-09-09T22:38:15.962 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.962 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:15.963 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.963 DEBUG:boto:Params: {} 2022-09-09T22:38:15.964 DEBUG:boto:Token: None 2022-09-09T22:38:15.964 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:15.965 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Q67bpZkWtNaAE71xhuN9bbO7I3I= 2022-09-09T22:38:15.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Q67bpZkWtNaAE71xhuN9bbO7I3I=', 'Content-Length': '0'} 2022-09-09T22:38:15.969 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae099a1c38a59fcc-00631bc057-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:38:15.970 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:15.971 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:15.971 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:15.972 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:15.972 DEBUG:boto:Method: GET 2022-09-09T22:38:15.973 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:15.973 DEBUG:boto:Data: 2022-09-09T22:38:15.974 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.974 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:15.975 DEBUG:boto:Port: 8001 2022-09-09T22:38:15.975 DEBUG:boto:Params: {} 2022-09-09T22:38:15.976 DEBUG:boto:Token: None 2022-09-09T22:38:15.977 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:15.977 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:n32LZ0Vq8NNbMu6RiPSLK6IRfAw= 2022-09-09T22:38:15.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:n32LZ0Vq8NNbMu6RiPSLK6IRfAw=', 'Content-Length': '0'} 2022-09-09T22:38:15.981 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c5edb647db099c5-00631bc057-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.982 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:15.983 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:15.983 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:15.984 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:15.984 DEBUG:boto:Method: GET 2022-09-09T22:38:15.985 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:15.985 DEBUG:boto:Data: 2022-09-09T22:38:15.986 DEBUG:boto:Headers: {} 2022-09-09T22:38:15.987 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:15.987 DEBUG:boto:Port: 8000 2022-09-09T22:38:15.988 DEBUG:boto:Params: {} 2022-09-09T22:38:15.988 DEBUG:boto:Token: None 2022-09-09T22:38:15.989 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:15 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:15.990 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:n32LZ0Vq8NNbMu6RiPSLK6IRfAw= 2022-09-09T22:38:15.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:15 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:n32LZ0Vq8NNbMu6RiPSLK6IRfAw=', 'Content-Length': '0'} 2022-09-09T22:38:15.993 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022083ae38c841677-00631bc057-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:15.995 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:15.995 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:15.996 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:15.996 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:15.997 DEBUG:boto:Method: GET 2022-09-09T22:38:15.998 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:15.999 DEBUG:boto:Data: 2022-09-09T22:38:15.999 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.000 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:16.000 DEBUG:boto:Port: 8001 2022-09-09T22:38:16.001 DEBUG:boto:Params: {} 2022-09-09T22:38:16.002 DEBUG:boto:Token: None 2022-09-09T22:38:16.002 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/ 2022-09-09T22:38:16.003 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rSYRGsGXzQqulXo2/8yiWVOArjo= 2022-09-09T22:38:16.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rSYRGsGXzQqulXo2/8yiWVOArjo=', 'Content-Length': '0'} 2022-09-09T22:38:16.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c91ab70dcd68312b-00631bc058-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.010 DEBUG:boto:b'jkcgbj-18obj-test-zone11000falseobj-test-zone12022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:16.011 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:16.011 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:16.012 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:16.012 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:16.013 DEBUG:boto:Method: GET 2022-09-09T22:38:16.014 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:16.014 DEBUG:boto:Data: 2022-09-09T22:38:16.015 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.015 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:16.016 DEBUG:boto:Port: 8000 2022-09-09T22:38:16.016 DEBUG:boto:Params: {} 2022-09-09T22:38:16.017 DEBUG:boto:Token: None 2022-09-09T22:38:16.018 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/ 2022-09-09T22:38:16.019 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rSYRGsGXzQqulXo2/8yiWVOArjo= 2022-09-09T22:38:16.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rSYRGsGXzQqulXo2/8yiWVOArjo=', 'Content-Length': '0'} 2022-09-09T22:38:16.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044a05fa5674e61ae-00631bc058-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.025 DEBUG:boto:b'jkcgbj-18obj-test-zone11000falseobj-test-zone12022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:16.026 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:16.027 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:16.028 DEBUG:boto:Method: HEAD 2022-09-09T22:38:16.029 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1 2022-09-09T22:38:16.030 DEBUG:boto:Data: 2022-09-09T22:38:16.031 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.032 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:16.033 DEBUG:boto:Port: 8001 2022-09-09T22:38:16.033 DEBUG:boto:Params: {} 2022-09-09T22:38:16.035 DEBUG:boto:Token: None 2022-09-09T22:38:16.035 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/obj-test-zone1 2022-09-09T22:38:16.036 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FPEWSkoR/YCkwq6LX6Ctzoztqjo= 2022-09-09T22:38:16.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FPEWSkoR/YCkwq6LX6Ctzoztqjo=', 'Content-Length': '0'} 2022-09-09T22:38:16.042 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056bbd2472ee6bebb-00631bc058-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.043 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:16.044 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:16.045 DEBUG:boto:Method: HEAD 2022-09-09T22:38:16.045 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1 2022-09-09T22:38:16.046 DEBUG:boto:Data: 2022-09-09T22:38:16.047 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.048 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:16.049 DEBUG:boto:Port: 8000 2022-09-09T22:38:16.050 DEBUG:boto:Params: {} 2022-09-09T22:38:16.051 DEBUG:boto:Token: None 2022-09-09T22:38:16.052 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/obj-test-zone1 2022-09-09T22:38:16.053 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FPEWSkoR/YCkwq6LX6Ctzoztqjo= 2022-09-09T22:38:16.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FPEWSkoR/YCkwq6LX6Ctzoztqjo=', 'Content-Length': '0'} 2022-09-09T22:38:16.058 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007676a8ab769e45d4-00631bc058-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:38:16.060 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:16.061 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:16.062 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:16.063 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:16.064 DEBUG:boto:Method: GET 2022-09-09T22:38:16.065 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:16.066 DEBUG:boto:Data: 2022-09-09T22:38:16.067 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.068 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:16.069 DEBUG:boto:Port: 8001 2022-09-09T22:38:16.070 DEBUG:boto:Params: {} 2022-09-09T22:38:16.071 DEBUG:boto:Token: None 2022-09-09T22:38:16.072 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:16.073 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:PfkBTA281QpQre/PWoMIhMFKwQs= 2022-09-09T22:38:16.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:PfkBTA281QpQre/PWoMIhMFKwQs=', 'Content-Length': '0'} 2022-09-09T22:38:16.078 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000080bb28bffbf61570-00631bc058-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.122 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:16.123 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:16.124 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:16.125 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:16.126 DEBUG:boto:Method: GET 2022-09-09T22:38:16.127 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:16.128 DEBUG:boto:Data: 2022-09-09T22:38:16.129 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.130 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:16.130 DEBUG:boto:Port: 8000 2022-09-09T22:38:16.131 DEBUG:boto:Params: {} 2022-09-09T22:38:16.132 DEBUG:boto:Token: None 2022-09-09T22:38:16.133 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:16.134 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:PfkBTA281QpQre/PWoMIhMFKwQs= 2022-09-09T22:38:16.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:PfkBTA281QpQre/PWoMIhMFKwQs=', 'Content-Length': '0'} 2022-09-09T22:38:16.139 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000010f34dd567291340-00631bc058-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.141 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:16.142 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:16.142 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:16.143 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:16.144 DEBUG:boto:Method: GET 2022-09-09T22:38:16.145 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:16.146 DEBUG:boto:Data: 2022-09-09T22:38:16.147 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.147 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:16.148 DEBUG:boto:Port: 8001 2022-09-09T22:38:16.149 DEBUG:boto:Params: {} 2022-09-09T22:38:16.150 DEBUG:boto:Token: None 2022-09-09T22:38:16.151 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/ 2022-09-09T22:38:16.152 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rSYRGsGXzQqulXo2/8yiWVOArjo= 2022-09-09T22:38:16.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rSYRGsGXzQqulXo2/8yiWVOArjo=', 'Content-Length': '0'} 2022-09-09T22:38:16.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9902d91eb01adbb-00631bc058-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.167 DEBUG:boto:b'jkcgbj-18obj-test-zone11000falseobj-test-zone12022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:16.168 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:16.169 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:16.169 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:16.170 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:16.170 DEBUG:boto:Method: GET 2022-09-09T22:38:16.171 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:16.171 DEBUG:boto:Data: 2022-09-09T22:38:16.172 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.172 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:16.173 DEBUG:boto:Port: 8000 2022-09-09T22:38:16.173 DEBUG:boto:Params: {} 2022-09-09T22:38:16.174 DEBUG:boto:Token: None 2022-09-09T22:38:16.175 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/ 2022-09-09T22:38:16.175 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rSYRGsGXzQqulXo2/8yiWVOArjo= 2022-09-09T22:38:16.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rSYRGsGXzQqulXo2/8yiWVOArjo=', 'Content-Length': '0'} 2022-09-09T22:38:16.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006d391323c74d71a-00631bc058-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.222 DEBUG:boto:b'jkcgbj-18obj-test-zone11000falseobj-test-zone12022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:16.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:38:16.224 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:16.224 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:16.225 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.225 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.226 DEBUG:boto:Method: GET 2022-09-09T22:38:16.227 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.227 DEBUG:boto:Data: 2022-09-09T22:38:16.228 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.228 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:16.229 DEBUG:boto:Port: 8001 2022-09-09T22:38:16.229 DEBUG:boto:Params: {} 2022-09-09T22:38:16.230 DEBUG:boto:Token: None 2022-09-09T22:38:16.231 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.231 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R3oTqAJsOqexBmGvXDYzWv2TmB4= 2022-09-09T22:38:16.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R3oTqAJsOqexBmGvXDYzWv2TmB4=', 'Content-Length': '0'} 2022-09-09T22:38:16.235 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000685db3121db9ac7f-00631bc058-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.236 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:16.237 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:16.237 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.238 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.238 DEBUG:boto:Method: GET 2022-09-09T22:38:16.239 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.239 DEBUG:boto:Data: 2022-09-09T22:38:16.240 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.241 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:16.241 DEBUG:boto:Port: 8000 2022-09-09T22:38:16.242 DEBUG:boto:Params: {} 2022-09-09T22:38:16.242 DEBUG:boto:Token: None 2022-09-09T22:38:16.243 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.244 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R3oTqAJsOqexBmGvXDYzWv2TmB4= 2022-09-09T22:38:16.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R3oTqAJsOqexBmGvXDYzWv2TmB4=', 'Content-Length': '0'} 2022-09-09T22:38:16.247 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000751ff69c36711f1f-00631bc058-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.248 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:16.249 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:16.249 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.250 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.250 DEBUG:boto:Method: HEAD 2022-09-09T22:38:16.251 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.252 DEBUG:boto:Data: 2022-09-09T22:38:16.252 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.253 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:16.253 DEBUG:boto:Port: 8001 2022-09-09T22:38:16.254 DEBUG:boto:Params: {} 2022-09-09T22:38:16.254 DEBUG:boto:Token: None 2022-09-09T22:38:16.255 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.255 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:B1SZ5S4W/fNiCLjo11xXQTbH8fg= 2022-09-09T22:38:16.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:B1SZ5S4W/fNiCLjo11xXQTbH8fg=', 'Content-Length': '0'} 2022-09-09T22:38:16.259 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b53a7e13516f3338-00631bc058-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.260 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:16.260 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:16.261 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.261 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.262 DEBUG:boto:Method: HEAD 2022-09-09T22:38:16.262 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.263 DEBUG:boto:Data: 2022-09-09T22:38:16.263 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.264 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:16.264 DEBUG:boto:Port: 8000 2022-09-09T22:38:16.265 DEBUG:boto:Params: {} 2022-09-09T22:38:16.265 DEBUG:boto:Token: None 2022-09-09T22:38:16.266 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.266 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:B1SZ5S4W/fNiCLjo11xXQTbH8fg= 2022-09-09T22:38:16.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:B1SZ5S4W/fNiCLjo11xXQTbH8fg=', 'Content-Length': '0'} 2022-09-09T22:38:16.270 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000560eb11ce4e26840-00631bc058-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:38:16.271 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:16.272 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:16.272 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.273 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.273 DEBUG:boto:Method: GET 2022-09-09T22:38:16.274 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.274 DEBUG:boto:Data: 2022-09-09T22:38:16.275 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.275 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:16.276 DEBUG:boto:Port: 8001 2022-09-09T22:38:16.276 DEBUG:boto:Params: {} 2022-09-09T22:38:16.277 DEBUG:boto:Token: None 2022-09-09T22:38:16.277 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.278 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R3oTqAJsOqexBmGvXDYzWv2TmB4= 2022-09-09T22:38:16.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R3oTqAJsOqexBmGvXDYzWv2TmB4=', 'Content-Length': '0'} 2022-09-09T22:38:16.282 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8f4424cf48b5c29-00631bc058-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.322 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:16.323 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:16.323 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.324 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.325 DEBUG:boto:Method: GET 2022-09-09T22:38:16.325 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.326 DEBUG:boto:Data: 2022-09-09T22:38:16.326 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.327 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:16.327 DEBUG:boto:Port: 8000 2022-09-09T22:38:16.328 DEBUG:boto:Params: {} 2022-09-09T22:38:16.328 DEBUG:boto:Token: None 2022-09-09T22:38:16.329 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.330 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R3oTqAJsOqexBmGvXDYzWv2TmB4= 2022-09-09T22:38:16.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R3oTqAJsOqexBmGvXDYzWv2TmB4=', 'Content-Length': '0'} 2022-09-09T22:38:16.333 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed15f2e5c3a76c29-00631bc058-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.334 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:16.335 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:16.335 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:16.336 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:16.336 DEBUG:boto:Method: GET 2022-09-09T22:38:16.337 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:16.338 DEBUG:boto:Data: 2022-09-09T22:38:16.338 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.339 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:16.339 DEBUG:boto:Port: 8001 2022-09-09T22:38:16.340 DEBUG:boto:Params: {} 2022-09-09T22:38:16.340 DEBUG:boto:Token: None 2022-09-09T22:38:16.341 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/ 2022-09-09T22:38:16.341 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rSYRGsGXzQqulXo2/8yiWVOArjo= 2022-09-09T22:38:16.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rSYRGsGXzQqulXo2/8yiWVOArjo=', 'Content-Length': '0'} 2022-09-09T22:38:16.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078ae9fbabe38a116-00631bc058-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.349 DEBUG:boto:b'jkcgbj-18obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:16.350 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:16.350 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:16.351 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:16.352 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:16.352 DEBUG:boto:Method: GET 2022-09-09T22:38:16.353 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:16.353 DEBUG:boto:Data: 2022-09-09T22:38:16.354 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.354 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:16.355 DEBUG:boto:Port: 8000 2022-09-09T22:38:16.355 DEBUG:boto:Params: {} 2022-09-09T22:38:16.356 DEBUG:boto:Token: None 2022-09-09T22:38:16.356 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/ 2022-09-09T22:38:16.357 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rSYRGsGXzQqulXo2/8yiWVOArjo= 2022-09-09T22:38:16.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rSYRGsGXzQqulXo2/8yiWVOArjo=', 'Content-Length': '0'} 2022-09-09T22:38:16.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a79f4668f5175ef-00631bc058-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.364 DEBUG:boto:b'jkcgbj-18obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:16.365 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:16.365 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:16.366 DEBUG:boto:Method: HEAD 2022-09-09T22:38:16.366 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2 2022-09-09T22:38:16.367 DEBUG:boto:Data: 2022-09-09T22:38:16.367 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.368 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:16.368 DEBUG:boto:Port: 8001 2022-09-09T22:38:16.369 DEBUG:boto:Params: {} 2022-09-09T22:38:16.369 DEBUG:boto:Token: None 2022-09-09T22:38:16.370 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/obj-test-zone2 2022-09-09T22:38:16.370 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:afZSiMO1C5nYAABNQLrbutZ30PA= 2022-09-09T22:38:16.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:afZSiMO1C5nYAABNQLrbutZ30PA=', 'Content-Length': '0'} 2022-09-09T22:38:16.375 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000081e435ebd3916959-00631bc058-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.375 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:16.376 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:16.376 DEBUG:boto:Method: HEAD 2022-09-09T22:38:16.377 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2 2022-09-09T22:38:16.377 DEBUG:boto:Data: 2022-09-09T22:38:16.378 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.378 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:16.379 DEBUG:boto:Port: 8000 2022-09-09T22:38:16.379 DEBUG:boto:Params: {} 2022-09-09T22:38:16.380 DEBUG:boto:Token: None 2022-09-09T22:38:16.380 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/obj-test-zone2 2022-09-09T22:38:16.381 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:afZSiMO1C5nYAABNQLrbutZ30PA= 2022-09-09T22:38:16.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:afZSiMO1C5nYAABNQLrbutZ30PA=', 'Content-Length': '0'} 2022-09-09T22:38:16.386 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a661390dd8537a4-00631bc058-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:38:16.387 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:16.388 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:16.388 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.389 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.389 DEBUG:boto:Method: GET 2022-09-09T22:38:16.390 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.390 DEBUG:boto:Data: 2022-09-09T22:38:16.391 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.391 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:16.392 DEBUG:boto:Port: 8001 2022-09-09T22:38:16.392 DEBUG:boto:Params: {} 2022-09-09T22:38:16.393 DEBUG:boto:Token: None 2022-09-09T22:38:16.394 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.394 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R3oTqAJsOqexBmGvXDYzWv2TmB4= 2022-09-09T22:38:16.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R3oTqAJsOqexBmGvXDYzWv2TmB4=', 'Content-Length': '0'} 2022-09-09T22:38:16.398 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad76db35768a7d15-00631bc058-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.442 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:16.443 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:16.443 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.444 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.444 DEBUG:boto:Method: GET 2022-09-09T22:38:16.445 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.445 DEBUG:boto:Data: 2022-09-09T22:38:16.446 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.446 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:16.447 DEBUG:boto:Port: 8000 2022-09-09T22:38:16.447 DEBUG:boto:Params: {} 2022-09-09T22:38:16.448 DEBUG:boto:Token: None 2022-09-09T22:38:16.448 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:16.449 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R3oTqAJsOqexBmGvXDYzWv2TmB4= 2022-09-09T22:38:16.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R3oTqAJsOqexBmGvXDYzWv2TmB4=', 'Content-Length': '0'} 2022-09-09T22:38:16.453 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b4a1894fdff86798-00631bc058-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.454 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:16.454 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:16.455 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:16.455 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:16.456 DEBUG:boto:Method: GET 2022-09-09T22:38:16.456 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:16.457 DEBUG:boto:Data: 2022-09-09T22:38:16.457 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.458 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:16.459 DEBUG:boto:Port: 8001 2022-09-09T22:38:16.459 DEBUG:boto:Params: {} 2022-09-09T22:38:16.460 DEBUG:boto:Token: None 2022-09-09T22:38:16.460 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/ 2022-09-09T22:38:16.461 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rSYRGsGXzQqulXo2/8yiWVOArjo= 2022-09-09T22:38:16.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rSYRGsGXzQqulXo2/8yiWVOArjo=', 'Content-Length': '0'} 2022-09-09T22:38:16.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f779714c2e18b598-00631bc058-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.467 DEBUG:boto:b'jkcgbj-18obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:16.468 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:16.468 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:16.469 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:16.469 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:16.470 DEBUG:boto:Method: GET 2022-09-09T22:38:16.471 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:16.471 DEBUG:boto:Data: 2022-09-09T22:38:16.472 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.472 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:16.473 DEBUG:boto:Port: 8000 2022-09-09T22:38:16.473 DEBUG:boto:Params: {} 2022-09-09T22:38:16.474 DEBUG:boto:Token: None 2022-09-09T22:38:16.474 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/ 2022-09-09T22:38:16.475 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rSYRGsGXzQqulXo2/8yiWVOArjo= 2022-09-09T22:38:16.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rSYRGsGXzQqulXo2/8yiWVOArjo=', 'Content-Length': '0'} 2022-09-09T22:38:16.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036345b2375c63da8-00631bc058-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.482 DEBUG:boto:b'jkcgbj-18obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:16.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:38:16.483 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:16.483 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:16.484 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.484 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.485 DEBUG:boto:Method: GET 2022-09-09T22:38:16.485 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.486 DEBUG:boto:Data: 2022-09-09T22:38:16.487 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.487 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:16.488 DEBUG:boto:Port: 8001 2022-09-09T22:38:16.488 DEBUG:boto:Params: {} 2022-09-09T22:38:16.489 DEBUG:boto:Token: None 2022-09-09T22:38:16.489 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.490 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:igXK1fWPt4TLOT7NZCONHl5IkYo= 2022-09-09T22:38:16.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:igXK1fWPt4TLOT7NZCONHl5IkYo=', 'Content-Length': '0'} 2022-09-09T22:38:16.494 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a59f579069e0f858-00631bc058-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.494 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:16.495 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:16.495 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.496 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.497 DEBUG:boto:Method: GET 2022-09-09T22:38:16.497 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.498 DEBUG:boto:Data: 2022-09-09T22:38:16.498 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.499 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:16.499 DEBUG:boto:Port: 8000 2022-09-09T22:38:16.500 DEBUG:boto:Params: {} 2022-09-09T22:38:16.500 DEBUG:boto:Token: None 2022-09-09T22:38:16.501 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.501 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:igXK1fWPt4TLOT7NZCONHl5IkYo= 2022-09-09T22:38:16.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:igXK1fWPt4TLOT7NZCONHl5IkYo=', 'Content-Length': '0'} 2022-09-09T22:38:16.505 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba0177b884b2d0d0-00631bc058-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.506 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:16.506 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:16.507 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.507 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.508 DEBUG:boto:Method: HEAD 2022-09-09T22:38:16.508 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.509 DEBUG:boto:Data: 2022-09-09T22:38:16.509 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.510 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:16.510 DEBUG:boto:Port: 8001 2022-09-09T22:38:16.511 DEBUG:boto:Params: {} 2022-09-09T22:38:16.511 DEBUG:boto:Token: None 2022-09-09T22:38:16.512 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.513 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:C3lZ7Vzbk6KEnFrhUAbLeNsdqW4= 2022-09-09T22:38:16.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:C3lZ7Vzbk6KEnFrhUAbLeNsdqW4=', 'Content-Length': '0'} 2022-09-09T22:38:16.516 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f333c9eb4d4fac2c-00631bc058-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.517 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:16.517 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:16.518 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.518 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.519 DEBUG:boto:Method: HEAD 2022-09-09T22:38:16.519 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.520 DEBUG:boto:Data: 2022-09-09T22:38:16.520 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.521 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:16.521 DEBUG:boto:Port: 8000 2022-09-09T22:38:16.522 DEBUG:boto:Params: {} 2022-09-09T22:38:16.522 DEBUG:boto:Token: None 2022-09-09T22:38:16.523 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.524 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:C3lZ7Vzbk6KEnFrhUAbLeNsdqW4= 2022-09-09T22:38:16.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:C3lZ7Vzbk6KEnFrhUAbLeNsdqW4=', 'Content-Length': '0'} 2022-09-09T22:38:16.527 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc679d28fb633de3-00631bc058-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:38:16.528 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:16.529 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:16.529 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.530 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.531 DEBUG:boto:Method: GET 2022-09-09T22:38:16.531 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.532 DEBUG:boto:Data: 2022-09-09T22:38:16.532 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.533 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:16.533 DEBUG:boto:Port: 8001 2022-09-09T22:38:16.534 DEBUG:boto:Params: {} 2022-09-09T22:38:16.534 DEBUG:boto:Token: None 2022-09-09T22:38:16.535 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.535 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:igXK1fWPt4TLOT7NZCONHl5IkYo= 2022-09-09T22:38:16.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:igXK1fWPt4TLOT7NZCONHl5IkYo=', 'Content-Length': '0'} 2022-09-09T22:38:16.539 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008331498624e124d8-00631bc058-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.540 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:16.540 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:16.541 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.542 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.542 DEBUG:boto:Method: GET 2022-09-09T22:38:16.543 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.543 DEBUG:boto:Data: 2022-09-09T22:38:16.544 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.544 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:16.545 DEBUG:boto:Port: 8000 2022-09-09T22:38:16.545 DEBUG:boto:Params: {} 2022-09-09T22:38:16.546 DEBUG:boto:Token: None 2022-09-09T22:38:16.547 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.547 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:igXK1fWPt4TLOT7NZCONHl5IkYo= 2022-09-09T22:38:16.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:igXK1fWPt4TLOT7NZCONHl5IkYo=', 'Content-Length': '0'} 2022-09-09T22:38:16.551 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b588b29b6d12307-00631bc058-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.552 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:16.552 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:16.553 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:16.553 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:16.554 DEBUG:boto:Method: GET 2022-09-09T22:38:16.554 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:16.555 DEBUG:boto:Data: 2022-09-09T22:38:16.555 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.556 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:16.556 DEBUG:boto:Port: 8001 2022-09-09T22:38:16.557 DEBUG:boto:Params: {} 2022-09-09T22:38:16.557 DEBUG:boto:Token: None 2022-09-09T22:38:16.558 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/ 2022-09-09T22:38:16.559 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rSYRGsGXzQqulXo2/8yiWVOArjo= 2022-09-09T22:38:16.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rSYRGsGXzQqulXo2/8yiWVOArjo=', 'Content-Length': '0'} 2022-09-09T22:38:16.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e50d3737f9175b45-00631bc058-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.606 DEBUG:boto:b'jkcgbj-18obj-test-zone31000falseobj-test-zone32022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:16.607 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:16.608 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:16.608 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:16.609 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:16.609 DEBUG:boto:Method: GET 2022-09-09T22:38:16.610 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:16.610 DEBUG:boto:Data: 2022-09-09T22:38:16.611 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.611 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:16.612 DEBUG:boto:Port: 8000 2022-09-09T22:38:16.612 DEBUG:boto:Params: {} 2022-09-09T22:38:16.613 DEBUG:boto:Token: None 2022-09-09T22:38:16.614 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/ 2022-09-09T22:38:16.614 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rSYRGsGXzQqulXo2/8yiWVOArjo= 2022-09-09T22:38:16.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rSYRGsGXzQqulXo2/8yiWVOArjo=', 'Content-Length': '0'} 2022-09-09T22:38:16.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007210a2af7c13a64-00631bc058-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.621 DEBUG:boto:b'jkcgbj-18obj-test-zone31000falseobj-test-zone32022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:16.622 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:16.623 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:16.623 DEBUG:boto:Method: HEAD 2022-09-09T22:38:16.624 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3 2022-09-09T22:38:16.624 DEBUG:boto:Data: 2022-09-09T22:38:16.625 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.625 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:16.626 DEBUG:boto:Port: 8001 2022-09-09T22:38:16.626 DEBUG:boto:Params: {} 2022-09-09T22:38:16.627 DEBUG:boto:Token: None 2022-09-09T22:38:16.627 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/obj-test-zone3 2022-09-09T22:38:16.628 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sNZrLjMi+GTeTpTRsDOdTqqnhRQ= 2022-09-09T22:38:16.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sNZrLjMi+GTeTpTRsDOdTqqnhRQ=', 'Content-Length': '0'} 2022-09-09T22:38:16.632 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000014faaee957f8632a-00631bc058-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.633 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:16.634 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:16.634 DEBUG:boto:Method: HEAD 2022-09-09T22:38:16.635 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3 2022-09-09T22:38:16.635 DEBUG:boto:Data: 2022-09-09T22:38:16.636 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.636 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:16.637 DEBUG:boto:Port: 8000 2022-09-09T22:38:16.637 DEBUG:boto:Params: {} 2022-09-09T22:38:16.638 DEBUG:boto:Token: None 2022-09-09T22:38:16.638 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/obj-test-zone3 2022-09-09T22:38:16.639 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sNZrLjMi+GTeTpTRsDOdTqqnhRQ= 2022-09-09T22:38:16.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sNZrLjMi+GTeTpTRsDOdTqqnhRQ=', 'Content-Length': '0'} 2022-09-09T22:38:16.643 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006916459fc61272a3-00631bc058-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:38:16.645 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:16.645 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:16.646 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.646 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.647 DEBUG:boto:Method: GET 2022-09-09T22:38:16.647 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.648 DEBUG:boto:Data: 2022-09-09T22:38:16.649 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.649 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:16.650 DEBUG:boto:Port: 8001 2022-09-09T22:38:16.650 DEBUG:boto:Params: {} 2022-09-09T22:38:16.651 DEBUG:boto:Token: None 2022-09-09T22:38:16.651 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.652 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:igXK1fWPt4TLOT7NZCONHl5IkYo= 2022-09-09T22:38:16.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:igXK1fWPt4TLOT7NZCONHl5IkYo=', 'Content-Length': '0'} 2022-09-09T22:38:16.656 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fad4f300f9eb94b6-00631bc058-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.657 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:16.657 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:16.658 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.659 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.659 DEBUG:boto:Method: GET 2022-09-09T22:38:16.660 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.660 DEBUG:boto:Data: 2022-09-09T22:38:16.661 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.661 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:16.662 DEBUG:boto:Port: 8000 2022-09-09T22:38:16.662 DEBUG:boto:Params: {} 2022-09-09T22:38:16.663 DEBUG:boto:Token: None 2022-09-09T22:38:16.663 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:16.664 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:igXK1fWPt4TLOT7NZCONHl5IkYo= 2022-09-09T22:38:16.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:igXK1fWPt4TLOT7NZCONHl5IkYo=', 'Content-Length': '0'} 2022-09-09T22:38:16.667 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb91df3a5d6843ff-00631bc058-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.714 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:16.715 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:16.716 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:16.716 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:16.717 DEBUG:boto:Method: GET 2022-09-09T22:38:16.717 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:16.718 DEBUG:boto:Data: 2022-09-09T22:38:16.719 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.720 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:16.720 DEBUG:boto:Port: 8001 2022-09-09T22:38:16.721 DEBUG:boto:Params: {} 2022-09-09T22:38:16.722 DEBUG:boto:Token: None 2022-09-09T22:38:16.723 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/ 2022-09-09T22:38:16.724 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rSYRGsGXzQqulXo2/8yiWVOArjo= 2022-09-09T22:38:16.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rSYRGsGXzQqulXo2/8yiWVOArjo=', 'Content-Length': '0'} 2022-09-09T22:38:16.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003aa615ba712176dd-00631bc058-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.731 DEBUG:boto:b'jkcgbj-18obj-test-zone31000falseobj-test-zone32022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:16.733 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:16.734 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:16.735 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:16.735 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:16.736 DEBUG:boto:Method: GET 2022-09-09T22:38:16.737 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:16.738 DEBUG:boto:Data: 2022-09-09T22:38:16.738 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.739 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:16.740 DEBUG:boto:Port: 8000 2022-09-09T22:38:16.740 DEBUG:boto:Params: {} 2022-09-09T22:38:16.741 DEBUG:boto:Token: None 2022-09-09T22:38:16.741 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/ 2022-09-09T22:38:16.742 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rSYRGsGXzQqulXo2/8yiWVOArjo= 2022-09-09T22:38:16.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rSYRGsGXzQqulXo2/8yiWVOArjo=', 'Content-Length': '0'} 2022-09-09T22:38:16.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000379efae4ee58718b-00631bc058-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.750 DEBUG:boto:b'jkcgbj-18obj-test-zone31000falseobj-test-zone32022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:16.751 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-18 zones={test-zone3, test-zone1} 2022-09-09T22:38:16.752 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-18 zones={test-zone3, test-zone2} 2022-09-09T22:38:16.752 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:16.753 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:16.753 DEBUG:boto:Method: HEAD 2022-09-09T22:38:16.754 DEBUG:boto:Path: /jkcgbj-18/ 2022-09-09T22:38:16.754 DEBUG:boto:Data: 2022-09-09T22:38:16.755 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.755 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:16.756 DEBUG:boto:Port: 8001 2022-09-09T22:38:16.757 DEBUG:boto:Params: {} 2022-09-09T22:38:16.757 DEBUG:boto:Token: None 2022-09-09T22:38:16.758 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/ 2022-09-09T22:38:16.758 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:MmfZiel1+tc0d2Y2ZCaT7e8KWhw= 2022-09-09T22:38:16.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:MmfZiel1+tc0d2Y2ZCaT7e8KWhw=', 'Content-Length': '0'} 2022-09-09T22:38:16.762 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f885d95654eefb5-00631bc058-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.763 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:16.764 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:16.764 DEBUG:boto:Method: HEAD 2022-09-09T22:38:16.765 DEBUG:boto:Path: /jkcgbj-18/ 2022-09-09T22:38:16.765 DEBUG:boto:Data: 2022-09-09T22:38:16.766 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.766 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:16.767 DEBUG:boto:Port: 8000 2022-09-09T22:38:16.767 DEBUG:boto:Params: {} 2022-09-09T22:38:16.768 DEBUG:boto:Token: None 2022-09-09T22:38:16.769 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/ 2022-09-09T22:38:16.769 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:MmfZiel1+tc0d2Y2ZCaT7e8KWhw= 2022-09-09T22:38:16.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:MmfZiel1+tc0d2Y2ZCaT7e8KWhw=', 'Content-Length': '0'} 2022-09-09T22:38:16.775 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000004b0ca2fd29106b-00631bc058-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.776 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:38:16.776 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:16.777 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:16.778 DEBUG:boto:path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:16.779 DEBUG:boto:auth_path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:16.779 DEBUG:boto:Method: GET 2022-09-09T22:38:16.780 DEBUG:boto:Path: /jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:16.781 DEBUG:boto:Data: 2022-09-09T22:38:16.781 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.782 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:16.782 DEBUG:boto:Port: 8001 2022-09-09T22:38:16.783 DEBUG:boto:Params: {} 2022-09-09T22:38:16.784 DEBUG:boto:Token: None 2022-09-09T22:38:16.785 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/?versions 2022-09-09T22:38:16.785 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0uwaA0hbeqyILl5kDVMl4dyTPJk= 2022-09-09T22:38:16.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0uwaA0hbeqyILl5kDVMl4dyTPJk=', 'Content-Length': '0'} 2022-09-09T22:38:16.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d91b609d445df536-00631bc058-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.834 DEBUG:boto:b'jkcgbj-18999falsedummyRQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5true2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1a8lrwM74XPlF1mwXf8hNl9Y9ISdV53ltrue2022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2e.gQ6yFECscD-oc4OV0dmk6aAg9jH.otrue2022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1true2022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:16.835 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:38:16.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:38:16.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:38:16.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:38:16.839 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:38:16.840 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:16.841 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:16.841 DEBUG:boto:path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:16.842 DEBUG:boto:auth_path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:16.843 DEBUG:boto:Method: GET 2022-09-09T22:38:16.844 DEBUG:boto:Path: /jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:16.844 DEBUG:boto:Data: 2022-09-09T22:38:16.845 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.845 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:16.846 DEBUG:boto:Port: 8000 2022-09-09T22:38:16.847 DEBUG:boto:Params: {} 2022-09-09T22:38:16.848 DEBUG:boto:Token: None 2022-09-09T22:38:16.848 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/?versions 2022-09-09T22:38:16.849 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0uwaA0hbeqyILl5kDVMl4dyTPJk= 2022-09-09T22:38:16.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0uwaA0hbeqyILl5kDVMl4dyTPJk=', 'Content-Length': '0'} 2022-09-09T22:38:16.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008744d56a531d44ee-00631bc058-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.856 DEBUG:boto:b'jkcgbj-18999falsedummyRQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5true2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1a8lrwM74XPlF1mwXf8hNl9Y9ISdV53ltrue2022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2e.gQ6yFECscD-oc4OV0dmk6aAg9jH.otrue2022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1true2022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:16.857 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:38:16.858 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:38:16.858 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:38:16.859 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:38:16.859 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:16.860 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:16.861 DEBUG:boto:path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:16.861 DEBUG:boto:auth_path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:16.862 DEBUG:boto:Method: GET 2022-09-09T22:38:16.862 DEBUG:boto:Path: /jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:16.863 DEBUG:boto:Data: 2022-09-09T22:38:16.863 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.864 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:16.865 DEBUG:boto:Port: 8001 2022-09-09T22:38:16.865 DEBUG:boto:Params: {} 2022-09-09T22:38:16.866 DEBUG:boto:Token: None 2022-09-09T22:38:16.867 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/?versions 2022-09-09T22:38:16.867 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0uwaA0hbeqyILl5kDVMl4dyTPJk= 2022-09-09T22:38:16.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0uwaA0hbeqyILl5kDVMl4dyTPJk=', 'Content-Length': '0'} 2022-09-09T22:38:16.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1a3ba2e419cb268-00631bc058-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.875 DEBUG:boto:b'jkcgbj-18999falsedummyRQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5true2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1a8lrwM74XPlF1mwXf8hNl9Y9ISdV53ltrue2022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2e.gQ6yFECscD-oc4OV0dmk6aAg9jH.otrue2022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1true2022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:16.876 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:16.876 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:16.877 DEBUG:boto:path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:16.877 DEBUG:boto:auth_path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:16.878 DEBUG:boto:Method: GET 2022-09-09T22:38:16.878 DEBUG:boto:Path: /jkcgbj-18/?versions&max-keys=999 2022-09-09T22:38:16.879 DEBUG:boto:Data: 2022-09-09T22:38:16.879 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.880 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:16.880 DEBUG:boto:Port: 8000 2022-09-09T22:38:16.881 DEBUG:boto:Params: {} 2022-09-09T22:38:16.882 DEBUG:boto:Token: None 2022-09-09T22:38:16.882 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/?versions 2022-09-09T22:38:16.883 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0uwaA0hbeqyILl5kDVMl4dyTPJk= 2022-09-09T22:38:16.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0uwaA0hbeqyILl5kDVMl4dyTPJk=', 'Content-Length': '0'} 2022-09-09T22:38:16.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4aefe3106abf7cf-00631bc058-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.890 DEBUG:boto:b'jkcgbj-18999falsedummyRQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5true2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1a8lrwM74XPlF1mwXf8hNl9Y9ISdV53ltrue2022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2e.gQ6yFECscD-oc4OV0dmk6aAg9jH.otrue2022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1true2022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:16.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:38:16.893 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:38:16.894 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:38:16.895 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:16.896 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:16.897 DEBUG:boto:Method: GET 2022-09-09T22:38:16.897 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:16.899 DEBUG:boto:Data: 2022-09-09T22:38:16.899 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.900 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:16.900 DEBUG:boto:Port: 8001 2022-09-09T22:38:16.901 DEBUG:boto:Params: {} 2022-09-09T22:38:16.901 DEBUG:boto:Token: None 2022-09-09T22:38:16.902 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:16.903 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wBEN/Zuo/+Sc/to/w6H4cQK/HbQ= 2022-09-09T22:38:16.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wBEN/Zuo/+Sc/to/w6H4cQK/HbQ=', 'Content-Length': '0'} 2022-09-09T22:38:16.907 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006ce817c43f41e29b-00631bc058-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.907 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:38:16.908 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:38:16.908 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:16.909 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:16.910 DEBUG:boto:Method: GET 2022-09-09T22:38:16.910 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:16.911 DEBUG:boto:Data: 2022-09-09T22:38:16.911 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.912 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:16.912 DEBUG:boto:Port: 8000 2022-09-09T22:38:16.913 DEBUG:boto:Params: {} 2022-09-09T22:38:16.914 DEBUG:boto:Token: None 2022-09-09T22:38:16.914 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:16.915 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wBEN/Zuo/+Sc/to/w6H4cQK/HbQ= 2022-09-09T22:38:16.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wBEN/Zuo/+Sc/to/w6H4cQK/HbQ=', 'Content-Length': '0'} 2022-09-09T22:38:16.919 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009073306233108209-00631bc058-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.920 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:38:16.921 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:38:16.922 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:16.923 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:16.924 DEBUG:boto:Method: HEAD 2022-09-09T22:38:16.925 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:16.926 DEBUG:boto:Data: 2022-09-09T22:38:16.926 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.927 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:16.928 DEBUG:boto:Port: 8001 2022-09-09T22:38:16.929 DEBUG:boto:Params: {} 2022-09-09T22:38:16.931 DEBUG:boto:Token: None 2022-09-09T22:38:16.932 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:16.933 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Y8MVQlVXDanjJTdWmZemFEuog/g= 2022-09-09T22:38:16.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Y8MVQlVXDanjJTdWmZemFEuog/g=', 'Content-Length': '0'} 2022-09-09T22:38:16.938 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a4dd94540af69294-00631bc058-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.940 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:38:16.941 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:38:16.942 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:16.943 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:16.944 DEBUG:boto:Method: HEAD 2022-09-09T22:38:16.945 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:16.946 DEBUG:boto:Data: 2022-09-09T22:38:16.947 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.948 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:16.949 DEBUG:boto:Port: 8000 2022-09-09T22:38:16.950 DEBUG:boto:Params: {} 2022-09-09T22:38:16.951 DEBUG:boto:Token: None 2022-09-09T22:38:16.952 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:16.953 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Y8MVQlVXDanjJTdWmZemFEuog/g= 2022-09-09T22:38:16.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Y8MVQlVXDanjJTdWmZemFEuog/g=', 'Content-Length': '0'} 2022-09-09T22:38:16.958 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f82ce138b43d162d-00631bc058-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:38:16.960 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:38:16.961 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:38:16.962 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:16.963 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:16.964 DEBUG:boto:Method: GET 2022-09-09T22:38:16.965 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:16.966 DEBUG:boto:Data: 2022-09-09T22:38:16.967 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.968 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:16.969 DEBUG:boto:Port: 8001 2022-09-09T22:38:16.970 DEBUG:boto:Params: {} 2022-09-09T22:38:16.971 DEBUG:boto:Token: None 2022-09-09T22:38:16.972 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:16.974 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wBEN/Zuo/+Sc/to/w6H4cQK/HbQ= 2022-09-09T22:38:16.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wBEN/Zuo/+Sc/to/w6H4cQK/HbQ=', 'Content-Length': '0'} 2022-09-09T22:38:16.979 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000079fcef2c3182adbc-00631bc058-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.980 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:38:16.981 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:38:16.982 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:16.983 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:16.984 DEBUG:boto:Method: GET 2022-09-09T22:38:16.985 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:16.985 DEBUG:boto:Data: 2022-09-09T22:38:16.986 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.986 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:16.987 DEBUG:boto:Port: 8000 2022-09-09T22:38:16.987 DEBUG:boto:Params: {} 2022-09-09T22:38:16.988 DEBUG:boto:Token: None 2022-09-09T22:38:16.989 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:16 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:16.989 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wBEN/Zuo/+Sc/to/w6H4cQK/HbQ= 2022-09-09T22:38:16.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:16 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wBEN/Zuo/+Sc/to/w6H4cQK/HbQ=', 'Content-Length': '0'} 2022-09-09T22:38:16.993 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000016f2b98ad71c1c5e-00631bc058-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:16.994 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:16.995 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:16.996 DEBUG:boto:path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:38:16.996 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:38:16.997 DEBUG:boto:Method: GET 2022-09-09T22:38:16.997 DEBUG:boto:Path: /jkcgbj-18/?prefix=dummy 2022-09-09T22:38:16.998 DEBUG:boto:Data: 2022-09-09T22:38:16.998 DEBUG:boto:Headers: {} 2022-09-09T22:38:16.999 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:16.999 DEBUG:boto:Port: 8001 2022-09-09T22:38:17.000 DEBUG:boto:Params: {} 2022-09-09T22:38:17.000 DEBUG:boto:Token: None 2022-09-09T22:38:17.001 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/ 2022-09-09T22:38:17.002 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Z2AIpI//cgUVljm3GHBcPJEtCtI= 2022-09-09T22:38:17.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Z2AIpI//cgUVljm3GHBcPJEtCtI=', 'Content-Length': '0'} 2022-09-09T22:38:17.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d67e1aba883a620-00631bc058-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.054 DEBUG:boto:b'jkcgbj-18dummy1000falsedummy2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:17.055 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:17.056 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:17.056 DEBUG:boto:path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:38:17.057 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:38:17.057 DEBUG:boto:Method: GET 2022-09-09T22:38:17.058 DEBUG:boto:Path: /jkcgbj-18/?prefix=dummy 2022-09-09T22:38:17.059 DEBUG:boto:Data: 2022-09-09T22:38:17.059 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.060 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:17.060 DEBUG:boto:Port: 8000 2022-09-09T22:38:17.061 DEBUG:boto:Params: {} 2022-09-09T22:38:17.062 DEBUG:boto:Token: None 2022-09-09T22:38:17.063 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/ 2022-09-09T22:38:17.063 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Z2AIpI//cgUVljm3GHBcPJEtCtI= 2022-09-09T22:38:17.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Z2AIpI//cgUVljm3GHBcPJEtCtI=', 'Content-Length': '0'} 2022-09-09T22:38:17.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000624f8ab01ef7d3f5-00631bc059-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.071 DEBUG:boto:b'jkcgbj-18dummy1000falsedummy2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:17.073 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:38:17.074 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:38:17.074 DEBUG:boto:Method: HEAD 2022-09-09T22:38:17.075 DEBUG:boto:Path: /jkcgbj-18/dummy 2022-09-09T22:38:17.075 DEBUG:boto:Data: 2022-09-09T22:38:17.076 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.076 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:17.077 DEBUG:boto:Port: 8001 2022-09-09T22:38:17.077 DEBUG:boto:Params: {} 2022-09-09T22:38:17.078 DEBUG:boto:Token: None 2022-09-09T22:38:17.078 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/dummy 2022-09-09T22:38:17.079 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:lfr3wDIbNXGRWusF5+0kyzc0daM= 2022-09-09T22:38:17.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:lfr3wDIbNXGRWusF5+0kyzc0daM=', 'Content-Length': '0'} 2022-09-09T22:38:17.083 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b26d38adf7963081-00631bc059-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.084 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:38:17.085 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:38:17.086 DEBUG:boto:Method: HEAD 2022-09-09T22:38:17.086 DEBUG:boto:Path: /jkcgbj-18/dummy 2022-09-09T22:38:17.087 DEBUG:boto:Data: 2022-09-09T22:38:17.087 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.088 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:17.088 DEBUG:boto:Port: 8000 2022-09-09T22:38:17.089 DEBUG:boto:Params: {} 2022-09-09T22:38:17.089 DEBUG:boto:Token: None 2022-09-09T22:38:17.090 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/dummy 2022-09-09T22:38:17.091 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:lfr3wDIbNXGRWusF5+0kyzc0daM= 2022-09-09T22:38:17.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:lfr3wDIbNXGRWusF5+0kyzc0daM=', 'Content-Length': '0'} 2022-09-09T22:38:17.095 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dcca22bcc2383498-00631bc059-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:38:17.098 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:38:17.098 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:38:17.099 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:17.100 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:17.100 DEBUG:boto:Method: GET 2022-09-09T22:38:17.101 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:17.101 DEBUG:boto:Data: 2022-09-09T22:38:17.102 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.102 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:17.103 DEBUG:boto:Port: 8001 2022-09-09T22:38:17.104 DEBUG:boto:Params: {} 2022-09-09T22:38:17.104 DEBUG:boto:Token: None 2022-09-09T22:38:17.105 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:17.106 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:F82QN5imtXBMlaqfGRA9VsUJy4s= 2022-09-09T22:38:17.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:F82QN5imtXBMlaqfGRA9VsUJy4s=', 'Content-Length': '0'} 2022-09-09T22:38:17.109 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d7e6cf0784ac1c9d-00631bc059-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.110 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:38:17.111 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:38:17.111 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:17.112 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:17.113 DEBUG:boto:Method: GET 2022-09-09T22:38:17.113 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:17.114 DEBUG:boto:Data: 2022-09-09T22:38:17.114 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.115 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:17.116 DEBUG:boto:Port: 8000 2022-09-09T22:38:17.116 DEBUG:boto:Params: {} 2022-09-09T22:38:17.117 DEBUG:boto:Token: None 2022-09-09T22:38:17.117 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:38:17.118 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:F82QN5imtXBMlaqfGRA9VsUJy4s= 2022-09-09T22:38:17.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:F82QN5imtXBMlaqfGRA9VsUJy4s=', 'Content-Length': '0'} 2022-09-09T22:38:17.122 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000067b4a2ab69767ab1-00631bc059-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.123 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:17.123 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:17.124 DEBUG:boto:path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:38:17.124 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:38:17.125 DEBUG:boto:Method: GET 2022-09-09T22:38:17.125 DEBUG:boto:Path: /jkcgbj-18/?prefix=dummy 2022-09-09T22:38:17.126 DEBUG:boto:Data: 2022-09-09T22:38:17.127 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.127 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:17.128 DEBUG:boto:Port: 8001 2022-09-09T22:38:17.128 DEBUG:boto:Params: {} 2022-09-09T22:38:17.129 DEBUG:boto:Token: None 2022-09-09T22:38:17.129 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/ 2022-09-09T22:38:17.130 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Z2AIpI//cgUVljm3GHBcPJEtCtI= 2022-09-09T22:38:17.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Z2AIpI//cgUVljm3GHBcPJEtCtI=', 'Content-Length': '0'} 2022-09-09T22:38:17.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001525f3c188b4d1c6-00631bc059-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.137 DEBUG:boto:b'jkcgbj-18dummy1000falsedummy2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:17.138 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:17.138 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:17.139 DEBUG:boto:path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:38:17.139 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:38:17.140 DEBUG:boto:Method: GET 2022-09-09T22:38:17.140 DEBUG:boto:Path: /jkcgbj-18/?prefix=dummy 2022-09-09T22:38:17.141 DEBUG:boto:Data: 2022-09-09T22:38:17.141 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.142 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:17.142 DEBUG:boto:Port: 8000 2022-09-09T22:38:17.143 DEBUG:boto:Params: {} 2022-09-09T22:38:17.143 DEBUG:boto:Token: None 2022-09-09T22:38:17.144 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/ 2022-09-09T22:38:17.145 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Z2AIpI//cgUVljm3GHBcPJEtCtI= 2022-09-09T22:38:17.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Z2AIpI//cgUVljm3GHBcPJEtCtI=', 'Content-Length': '0'} 2022-09-09T22:38:17.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002470d39744e41752-00631bc059-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.151 DEBUG:boto:b'jkcgbj-18dummy1000falsedummy2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:17.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:38:17.153 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:17.153 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:17.154 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.154 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.155 DEBUG:boto:Method: GET 2022-09-09T22:38:17.155 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.156 DEBUG:boto:Data: 2022-09-09T22:38:17.156 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.157 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:17.157 DEBUG:boto:Port: 8001 2022-09-09T22:38:17.158 DEBUG:boto:Params: {} 2022-09-09T22:38:17.158 DEBUG:boto:Token: None 2022-09-09T22:38:17.159 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.160 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UCtPJBqoaqp2J+y9U+cNRgv3dpo= 2022-09-09T22:38:17.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UCtPJBqoaqp2J+y9U+cNRgv3dpo=', 'Content-Length': '0'} 2022-09-09T22:38:17.163 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025be625fd55ff81d-00631bc059-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.206 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:17.207 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:17.208 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.208 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.209 DEBUG:boto:Method: GET 2022-09-09T22:38:17.209 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.210 DEBUG:boto:Data: 2022-09-09T22:38:17.211 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.212 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:17.213 DEBUG:boto:Port: 8000 2022-09-09T22:38:17.214 DEBUG:boto:Params: {} 2022-09-09T22:38:17.215 DEBUG:boto:Token: None 2022-09-09T22:38:17.215 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.216 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UCtPJBqoaqp2J+y9U+cNRgv3dpo= 2022-09-09T22:38:17.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UCtPJBqoaqp2J+y9U+cNRgv3dpo=', 'Content-Length': '0'} 2022-09-09T22:38:17.221 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000081f530d088dd1e9d-00631bc059-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.223 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:17.224 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:17.225 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.225 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.226 DEBUG:boto:Method: HEAD 2022-09-09T22:38:17.227 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.228 DEBUG:boto:Data: 2022-09-09T22:38:17.229 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.230 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:17.231 DEBUG:boto:Port: 8001 2022-09-09T22:38:17.232 DEBUG:boto:Params: {} 2022-09-09T22:38:17.233 DEBUG:boto:Token: None 2022-09-09T22:38:17.235 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.236 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0f/paHyAgZCly+O1yioSFbheeRA= 2022-09-09T22:38:17.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0f/paHyAgZCly+O1yioSFbheeRA=', 'Content-Length': '0'} 2022-09-09T22:38:17.240 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059dcc0def083403b-00631bc059-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.241 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:17.242 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:17.243 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.244 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.245 DEBUG:boto:Method: HEAD 2022-09-09T22:38:17.246 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.247 DEBUG:boto:Data: 2022-09-09T22:38:17.247 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.248 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:17.249 DEBUG:boto:Port: 8000 2022-09-09T22:38:17.250 DEBUG:boto:Params: {} 2022-09-09T22:38:17.251 DEBUG:boto:Token: None 2022-09-09T22:38:17.252 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.253 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0f/paHyAgZCly+O1yioSFbheeRA= 2022-09-09T22:38:17.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0f/paHyAgZCly+O1yioSFbheeRA=', 'Content-Length': '0'} 2022-09-09T22:38:17.258 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2d52b6726d7fb94-00631bc059-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:38:17.260 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:17.261 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:17.262 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.263 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.264 DEBUG:boto:Method: GET 2022-09-09T22:38:17.265 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.265 DEBUG:boto:Data: 2022-09-09T22:38:17.266 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.266 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:17.267 DEBUG:boto:Port: 8001 2022-09-09T22:38:17.267 DEBUG:boto:Params: {} 2022-09-09T22:38:17.268 DEBUG:boto:Token: None 2022-09-09T22:38:17.268 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.269 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UCtPJBqoaqp2J+y9U+cNRgv3dpo= 2022-09-09T22:38:17.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UCtPJBqoaqp2J+y9U+cNRgv3dpo=', 'Content-Length': '0'} 2022-09-09T22:38:17.273 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091e82e2bfcc03dec-00631bc059-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.274 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:17.275 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:17.275 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.276 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.276 DEBUG:boto:Method: GET 2022-09-09T22:38:17.277 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.277 DEBUG:boto:Data: 2022-09-09T22:38:17.278 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.278 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:17.279 DEBUG:boto:Port: 8000 2022-09-09T22:38:17.280 DEBUG:boto:Params: {} 2022-09-09T22:38:17.280 DEBUG:boto:Token: None 2022-09-09T22:38:17.281 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.282 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UCtPJBqoaqp2J+y9U+cNRgv3dpo= 2022-09-09T22:38:17.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UCtPJBqoaqp2J+y9U+cNRgv3dpo=', 'Content-Length': '0'} 2022-09-09T22:38:17.285 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ba5dfe27f844f0a-00631bc059-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.330 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:17.331 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:17.331 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:17.332 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:17.333 DEBUG:boto:Method: GET 2022-09-09T22:38:17.334 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:17.335 DEBUG:boto:Data: 2022-09-09T22:38:17.336 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.337 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:17.338 DEBUG:boto:Port: 8001 2022-09-09T22:38:17.339 DEBUG:boto:Params: {} 2022-09-09T22:38:17.339 DEBUG:boto:Token: None 2022-09-09T22:38:17.340 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/ 2022-09-09T22:38:17.341 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Z2AIpI//cgUVljm3GHBcPJEtCtI= 2022-09-09T22:38:17.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Z2AIpI//cgUVljm3GHBcPJEtCtI=', 'Content-Length': '0'} 2022-09-09T22:38:17.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000228c0a72a5d44354-00631bc059-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.349 DEBUG:boto:b'jkcgbj-18obj-test-zone11000falseobj-test-zone12022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:17.350 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:17.351 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:17.351 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:17.352 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:17.352 DEBUG:boto:Method: GET 2022-09-09T22:38:17.353 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:17.354 DEBUG:boto:Data: 2022-09-09T22:38:17.354 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.355 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:17.355 DEBUG:boto:Port: 8000 2022-09-09T22:38:17.356 DEBUG:boto:Params: {} 2022-09-09T22:38:17.356 DEBUG:boto:Token: None 2022-09-09T22:38:17.357 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/ 2022-09-09T22:38:17.358 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Z2AIpI//cgUVljm3GHBcPJEtCtI= 2022-09-09T22:38:17.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Z2AIpI//cgUVljm3GHBcPJEtCtI=', 'Content-Length': '0'} 2022-09-09T22:38:17.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b414cbe7b25d6ef8-00631bc059-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.364 DEBUG:boto:b'jkcgbj-18obj-test-zone11000falseobj-test-zone12022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:17.365 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:17.365 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:17.366 DEBUG:boto:Method: HEAD 2022-09-09T22:38:17.366 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1 2022-09-09T22:38:17.367 DEBUG:boto:Data: 2022-09-09T22:38:17.367 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.368 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:17.368 DEBUG:boto:Port: 8001 2022-09-09T22:38:17.369 DEBUG:boto:Params: {} 2022-09-09T22:38:17.369 DEBUG:boto:Token: None 2022-09-09T22:38:17.370 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/obj-test-zone1 2022-09-09T22:38:17.371 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:WT13m2M5R/oRB32TIsx64e1MHio= 2022-09-09T22:38:17.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:WT13m2M5R/oRB32TIsx64e1MHio=', 'Content-Length': '0'} 2022-09-09T22:38:17.376 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000028fa993d94db719b-00631bc059-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.377 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:17.377 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:17.378 DEBUG:boto:Method: HEAD 2022-09-09T22:38:17.378 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1 2022-09-09T22:38:17.379 DEBUG:boto:Data: 2022-09-09T22:38:17.379 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.380 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:17.380 DEBUG:boto:Port: 8000 2022-09-09T22:38:17.381 DEBUG:boto:Params: {} 2022-09-09T22:38:17.381 DEBUG:boto:Token: None 2022-09-09T22:38:17.382 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/obj-test-zone1 2022-09-09T22:38:17.383 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:WT13m2M5R/oRB32TIsx64e1MHio= 2022-09-09T22:38:17.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:WT13m2M5R/oRB32TIsx64e1MHio=', 'Content-Length': '0'} 2022-09-09T22:38:17.387 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000065cb9199185b11e2-00631bc059-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:38:17.389 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:17.390 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:17.390 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.391 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.392 DEBUG:boto:Method: GET 2022-09-09T22:38:17.392 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.393 DEBUG:boto:Data: 2022-09-09T22:38:17.393 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.394 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:17.394 DEBUG:boto:Port: 8001 2022-09-09T22:38:17.395 DEBUG:boto:Params: {} 2022-09-09T22:38:17.397 DEBUG:boto:Token: None 2022-09-09T22:38:17.398 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.399 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UCtPJBqoaqp2J+y9U+cNRgv3dpo= 2022-09-09T22:38:17.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UCtPJBqoaqp2J+y9U+cNRgv3dpo=', 'Content-Length': '0'} 2022-09-09T22:38:17.402 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000966c245a51df377f-00631bc059-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.446 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:17.447 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:17.448 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.449 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.449 DEBUG:boto:Method: GET 2022-09-09T22:38:17.450 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.451 DEBUG:boto:Data: 2022-09-09T22:38:17.452 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.452 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:17.453 DEBUG:boto:Port: 8000 2022-09-09T22:38:17.453 DEBUG:boto:Params: {} 2022-09-09T22:38:17.454 DEBUG:boto:Token: None 2022-09-09T22:38:17.454 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:38:17.455 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UCtPJBqoaqp2J+y9U+cNRgv3dpo= 2022-09-09T22:38:17.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UCtPJBqoaqp2J+y9U+cNRgv3dpo=', 'Content-Length': '0'} 2022-09-09T22:38:17.459 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad554bb5c06881f2-00631bc059-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.460 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:17.460 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:17.461 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:17.462 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:17.463 DEBUG:boto:Method: GET 2022-09-09T22:38:17.463 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:17.464 DEBUG:boto:Data: 2022-09-09T22:38:17.464 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.465 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:17.465 DEBUG:boto:Port: 8001 2022-09-09T22:38:17.466 DEBUG:boto:Params: {} 2022-09-09T22:38:17.466 DEBUG:boto:Token: None 2022-09-09T22:38:17.467 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/ 2022-09-09T22:38:17.468 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Z2AIpI//cgUVljm3GHBcPJEtCtI= 2022-09-09T22:38:17.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Z2AIpI//cgUVljm3GHBcPJEtCtI=', 'Content-Length': '0'} 2022-09-09T22:38:17.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d516bd5a3b7b76b9-00631bc059-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.474 DEBUG:boto:b'jkcgbj-18obj-test-zone11000falseobj-test-zone12022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:17.475 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:17.476 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:17.476 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:17.477 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:17.477 DEBUG:boto:Method: GET 2022-09-09T22:38:17.478 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:38:17.478 DEBUG:boto:Data: 2022-09-09T22:38:17.479 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.479 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:17.480 DEBUG:boto:Port: 8000 2022-09-09T22:38:17.481 DEBUG:boto:Params: {} 2022-09-09T22:38:17.481 DEBUG:boto:Token: None 2022-09-09T22:38:17.482 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/ 2022-09-09T22:38:17.482 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Z2AIpI//cgUVljm3GHBcPJEtCtI= 2022-09-09T22:38:17.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Z2AIpI//cgUVljm3GHBcPJEtCtI=', 'Content-Length': '0'} 2022-09-09T22:38:17.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fcf1083674fdbda-00631bc059-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.530 DEBUG:boto:b'jkcgbj-18obj-test-zone11000falseobj-test-zone12022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:17.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:38:17.532 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:17.532 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:17.533 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.533 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.534 DEBUG:boto:Method: GET 2022-09-09T22:38:17.535 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.535 DEBUG:boto:Data: 2022-09-09T22:38:17.536 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.536 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:17.537 DEBUG:boto:Port: 8001 2022-09-09T22:38:17.537 DEBUG:boto:Params: {} 2022-09-09T22:38:17.538 DEBUG:boto:Token: None 2022-09-09T22:38:17.539 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.540 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:PG7t6tHG9T4cg2Qf3PONDCmKMBc= 2022-09-09T22:38:17.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:PG7t6tHG9T4cg2Qf3PONDCmKMBc=', 'Content-Length': '0'} 2022-09-09T22:38:17.544 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f2594a23bee8d42-00631bc059-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.544 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:17.545 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:17.546 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.546 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.547 DEBUG:boto:Method: GET 2022-09-09T22:38:17.547 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.548 DEBUG:boto:Data: 2022-09-09T22:38:17.548 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.549 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:17.549 DEBUG:boto:Port: 8000 2022-09-09T22:38:17.550 DEBUG:boto:Params: {} 2022-09-09T22:38:17.550 DEBUG:boto:Token: None 2022-09-09T22:38:17.551 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.552 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:PG7t6tHG9T4cg2Qf3PONDCmKMBc= 2022-09-09T22:38:17.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:PG7t6tHG9T4cg2Qf3PONDCmKMBc=', 'Content-Length': '0'} 2022-09-09T22:38:17.556 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f7914edaeb5eac61-00631bc059-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.557 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:17.558 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:17.559 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.559 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.560 DEBUG:boto:Method: HEAD 2022-09-09T22:38:17.560 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.561 DEBUG:boto:Data: 2022-09-09T22:38:17.562 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.562 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:17.563 DEBUG:boto:Port: 8001 2022-09-09T22:38:17.563 DEBUG:boto:Params: {} 2022-09-09T22:38:17.564 DEBUG:boto:Token: None 2022-09-09T22:38:17.565 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.565 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:WhLV9mpHSs2VtI9o9oWxRt4oWpw= 2022-09-09T22:38:17.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:WhLV9mpHSs2VtI9o9oWxRt4oWpw=', 'Content-Length': '0'} 2022-09-09T22:38:17.569 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037ee37b0915c272f-00631bc059-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.570 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:17.570 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:17.571 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.571 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.572 DEBUG:boto:Method: HEAD 2022-09-09T22:38:17.572 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.573 DEBUG:boto:Data: 2022-09-09T22:38:17.573 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.574 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:17.574 DEBUG:boto:Port: 8000 2022-09-09T22:38:17.575 DEBUG:boto:Params: {} 2022-09-09T22:38:17.576 DEBUG:boto:Token: None 2022-09-09T22:38:17.576 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.577 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:WhLV9mpHSs2VtI9o9oWxRt4oWpw= 2022-09-09T22:38:17.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:WhLV9mpHSs2VtI9o9oWxRt4oWpw=', 'Content-Length': '0'} 2022-09-09T22:38:17.581 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004bf567f18921f4d1-00631bc059-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:38:17.583 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:17.584 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:17.585 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.586 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.587 DEBUG:boto:Method: GET 2022-09-09T22:38:17.588 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.589 DEBUG:boto:Data: 2022-09-09T22:38:17.590 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.591 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:17.592 DEBUG:boto:Port: 8001 2022-09-09T22:38:17.593 DEBUG:boto:Params: {} 2022-09-09T22:38:17.594 DEBUG:boto:Token: None 2022-09-09T22:38:17.595 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.597 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:PG7t6tHG9T4cg2Qf3PONDCmKMBc= 2022-09-09T22:38:17.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:PG7t6tHG9T4cg2Qf3PONDCmKMBc=', 'Content-Length': '0'} 2022-09-09T22:38:17.601 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007089a8a5f9196da9-00631bc059-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.603 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:17.604 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:17.605 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.606 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.607 DEBUG:boto:Method: GET 2022-09-09T22:38:17.608 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.609 DEBUG:boto:Data: 2022-09-09T22:38:17.610 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.611 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:17.612 DEBUG:boto:Port: 8000 2022-09-09T22:38:17.613 DEBUG:boto:Params: {} 2022-09-09T22:38:17.614 DEBUG:boto:Token: None 2022-09-09T22:38:17.615 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.616 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:PG7t6tHG9T4cg2Qf3PONDCmKMBc= 2022-09-09T22:38:17.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:PG7t6tHG9T4cg2Qf3PONDCmKMBc=', 'Content-Length': '0'} 2022-09-09T22:38:17.621 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000934794113378242c-00631bc059-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.622 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:17.623 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:17.623 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:17.624 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:17.624 DEBUG:boto:Method: GET 2022-09-09T22:38:17.625 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:17.625 DEBUG:boto:Data: 2022-09-09T22:38:17.626 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.626 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:17.627 DEBUG:boto:Port: 8001 2022-09-09T22:38:17.627 DEBUG:boto:Params: {} 2022-09-09T22:38:17.628 DEBUG:boto:Token: None 2022-09-09T22:38:17.629 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/ 2022-09-09T22:38:17.630 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Z2AIpI//cgUVljm3GHBcPJEtCtI= 2022-09-09T22:38:17.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Z2AIpI//cgUVljm3GHBcPJEtCtI=', 'Content-Length': '0'} 2022-09-09T22:38:17.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054b417802416a87a-00631bc059-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.637 DEBUG:boto:b'jkcgbj-18obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:17.638 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:17.639 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:17.640 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:17.640 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:17.641 DEBUG:boto:Method: GET 2022-09-09T22:38:17.641 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:17.642 DEBUG:boto:Data: 2022-09-09T22:38:17.643 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.643 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:17.644 DEBUG:boto:Port: 8000 2022-09-09T22:38:17.645 DEBUG:boto:Params: {} 2022-09-09T22:38:17.646 DEBUG:boto:Token: None 2022-09-09T22:38:17.646 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/ 2022-09-09T22:38:17.647 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Z2AIpI//cgUVljm3GHBcPJEtCtI= 2022-09-09T22:38:17.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Z2AIpI//cgUVljm3GHBcPJEtCtI=', 'Content-Length': '0'} 2022-09-09T22:38:17.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3dbeb87471bd578-00631bc059-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.655 DEBUG:boto:b'jkcgbj-18obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:17.656 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:17.656 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:17.657 DEBUG:boto:Method: HEAD 2022-09-09T22:38:17.658 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2 2022-09-09T22:38:17.658 DEBUG:boto:Data: 2022-09-09T22:38:17.659 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.659 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:17.660 DEBUG:boto:Port: 8001 2022-09-09T22:38:17.660 DEBUG:boto:Params: {} 2022-09-09T22:38:17.661 DEBUG:boto:Token: None 2022-09-09T22:38:17.662 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/obj-test-zone2 2022-09-09T22:38:17.663 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:/zygkNKMPDetDjkm5lxo6vXl0mM= 2022-09-09T22:38:17.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:/zygkNKMPDetDjkm5lxo6vXl0mM=', 'Content-Length': '0'} 2022-09-09T22:38:17.667 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007309a21f2ec07833-00631bc059-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.668 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:17.669 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:17.669 DEBUG:boto:Method: HEAD 2022-09-09T22:38:17.670 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2 2022-09-09T22:38:17.670 DEBUG:boto:Data: 2022-09-09T22:38:17.671 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.672 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:17.672 DEBUG:boto:Port: 8000 2022-09-09T22:38:17.673 DEBUG:boto:Params: {} 2022-09-09T22:38:17.674 DEBUG:boto:Token: None 2022-09-09T22:38:17.674 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/obj-test-zone2 2022-09-09T22:38:17.675 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:/zygkNKMPDetDjkm5lxo6vXl0mM= 2022-09-09T22:38:17.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:/zygkNKMPDetDjkm5lxo6vXl0mM=', 'Content-Length': '0'} 2022-09-09T22:38:17.679 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a50f19027033253-00631bc059-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:38:17.681 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:17.681 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:17.682 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.683 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.683 DEBUG:boto:Method: GET 2022-09-09T22:38:17.684 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.684 DEBUG:boto:Data: 2022-09-09T22:38:17.685 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.685 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:17.686 DEBUG:boto:Port: 8001 2022-09-09T22:38:17.687 DEBUG:boto:Params: {} 2022-09-09T22:38:17.688 DEBUG:boto:Token: None 2022-09-09T22:38:17.688 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.689 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:PG7t6tHG9T4cg2Qf3PONDCmKMBc= 2022-09-09T22:38:17.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:PG7t6tHG9T4cg2Qf3PONDCmKMBc=', 'Content-Length': '0'} 2022-09-09T22:38:17.692 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003404e9c40d0b974f-00631bc059-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.738 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:17.739 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:17.739 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.740 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.740 DEBUG:boto:Method: GET 2022-09-09T22:38:17.741 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.741 DEBUG:boto:Data: 2022-09-09T22:38:17.742 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.743 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:17.743 DEBUG:boto:Port: 8000 2022-09-09T22:38:17.744 DEBUG:boto:Params: {} 2022-09-09T22:38:17.744 DEBUG:boto:Token: None 2022-09-09T22:38:17.745 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:38:17.745 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:PG7t6tHG9T4cg2Qf3PONDCmKMBc= 2022-09-09T22:38:17.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:PG7t6tHG9T4cg2Qf3PONDCmKMBc=', 'Content-Length': '0'} 2022-09-09T22:38:17.749 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:02 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b846b620f3a652bf-00631bc059-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.750 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:17.751 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:17.751 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:17.752 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:17.752 DEBUG:boto:Method: GET 2022-09-09T22:38:17.753 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:17.754 DEBUG:boto:Data: 2022-09-09T22:38:17.754 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.755 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:17.755 DEBUG:boto:Port: 8001 2022-09-09T22:38:17.756 DEBUG:boto:Params: {} 2022-09-09T22:38:17.756 DEBUG:boto:Token: None 2022-09-09T22:38:17.757 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/ 2022-09-09T22:38:17.758 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Z2AIpI//cgUVljm3GHBcPJEtCtI= 2022-09-09T22:38:17.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Z2AIpI//cgUVljm3GHBcPJEtCtI=', 'Content-Length': '0'} 2022-09-09T22:38:17.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000108046c6694cdec-00631bc059-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.764 DEBUG:boto:b'jkcgbj-18obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:17.766 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:17.767 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:17.767 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:17.768 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:17.768 DEBUG:boto:Method: GET 2022-09-09T22:38:17.769 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:38:17.770 DEBUG:boto:Data: 2022-09-09T22:38:17.770 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.771 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:17.772 DEBUG:boto:Port: 8000 2022-09-09T22:38:17.772 DEBUG:boto:Params: {} 2022-09-09T22:38:17.774 DEBUG:boto:Token: None 2022-09-09T22:38:17.775 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/ 2022-09-09T22:38:17.776 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Z2AIpI//cgUVljm3GHBcPJEtCtI= 2022-09-09T22:38:17.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Z2AIpI//cgUVljm3GHBcPJEtCtI=', 'Content-Length': '0'} 2022-09-09T22:38:17.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6a2b5764de78467-00631bc059-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.783 DEBUG:boto:b'jkcgbj-18obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:17.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:38:17.785 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:17.785 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:17.786 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:17.786 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:17.787 DEBUG:boto:Method: GET 2022-09-09T22:38:17.788 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:17.788 DEBUG:boto:Data: 2022-09-09T22:38:17.789 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.789 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:17.790 DEBUG:boto:Port: 8001 2022-09-09T22:38:17.790 DEBUG:boto:Params: {} 2022-09-09T22:38:17.791 DEBUG:boto:Token: None 2022-09-09T22:38:17.792 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:17.792 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:AYLvnTnWHZxwxBfYwWUrV/sOUpI= 2022-09-09T22:38:17.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:AYLvnTnWHZxwxBfYwWUrV/sOUpI=', 'Content-Length': '0'} 2022-09-09T22:38:17.797 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000087dbee17d06b3030-00631bc059-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.798 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:17.800 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:17.801 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:17.801 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:17.802 DEBUG:boto:Method: GET 2022-09-09T22:38:17.802 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:17.803 DEBUG:boto:Data: 2022-09-09T22:38:17.803 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.804 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:17.804 DEBUG:boto:Port: 8000 2022-09-09T22:38:17.805 DEBUG:boto:Params: {} 2022-09-09T22:38:17.805 DEBUG:boto:Token: None 2022-09-09T22:38:17.806 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:17.807 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:AYLvnTnWHZxwxBfYwWUrV/sOUpI= 2022-09-09T22:38:17.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:AYLvnTnWHZxwxBfYwWUrV/sOUpI=', 'Content-Length': '0'} 2022-09-09T22:38:17.810 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009bb374515667bf4f-00631bc059-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.858 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:17.859 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:17.860 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:17.861 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:17.862 DEBUG:boto:Method: HEAD 2022-09-09T22:38:17.863 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:17.864 DEBUG:boto:Data: 2022-09-09T22:38:17.865 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.866 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:17.867 DEBUG:boto:Port: 8001 2022-09-09T22:38:17.868 DEBUG:boto:Params: {} 2022-09-09T22:38:17.869 DEBUG:boto:Token: None 2022-09-09T22:38:17.870 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:17.871 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:E+8ovH4FSr2K+T46cpae4lULvaw= 2022-09-09T22:38:17.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:E+8ovH4FSr2K+T46cpae4lULvaw=', 'Content-Length': '0'} 2022-09-09T22:38:17.875 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f82e6787fc685604-00631bc059-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.876 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:17.877 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:17.877 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:17.878 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:17.878 DEBUG:boto:Method: HEAD 2022-09-09T22:38:17.879 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:17.880 DEBUG:boto:Data: 2022-09-09T22:38:17.880 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.881 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:17.881 DEBUG:boto:Port: 8000 2022-09-09T22:38:17.882 DEBUG:boto:Params: {} 2022-09-09T22:38:17.882 DEBUG:boto:Token: None 2022-09-09T22:38:17.883 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:17.883 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:E+8ovH4FSr2K+T46cpae4lULvaw= 2022-09-09T22:38:17.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:E+8ovH4FSr2K+T46cpae4lULvaw=', 'Content-Length': '0'} 2022-09-09T22:38:17.887 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029843149a56f89a0-00631bc059-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:38:17.888 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:17.889 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:17.889 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:17.890 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:17.890 DEBUG:boto:Method: GET 2022-09-09T22:38:17.891 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:17.891 DEBUG:boto:Data: 2022-09-09T22:38:17.892 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.892 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:17.893 DEBUG:boto:Port: 8001 2022-09-09T22:38:17.893 DEBUG:boto:Params: {} 2022-09-09T22:38:17.894 DEBUG:boto:Token: None 2022-09-09T22:38:17.895 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:17.895 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:AYLvnTnWHZxwxBfYwWUrV/sOUpI= 2022-09-09T22:38:17.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:AYLvnTnWHZxwxBfYwWUrV/sOUpI=', 'Content-Length': '0'} 2022-09-09T22:38:17.899 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e7d65dd735ffe516-00631bc059-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.942 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:17.943 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:17.944 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:17.945 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:17.946 DEBUG:boto:Method: GET 2022-09-09T22:38:17.947 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:17.948 DEBUG:boto:Data: 2022-09-09T22:38:17.949 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.950 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:17.951 DEBUG:boto:Port: 8000 2022-09-09T22:38:17.952 DEBUG:boto:Params: {} 2022-09-09T22:38:17.953 DEBUG:boto:Token: None 2022-09-09T22:38:17.954 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:17.955 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:AYLvnTnWHZxwxBfYwWUrV/sOUpI= 2022-09-09T22:38:17.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:AYLvnTnWHZxwxBfYwWUrV/sOUpI=', 'Content-Length': '0'} 2022-09-09T22:38:17.965 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000893f5fb69a4e6693-00631bc059-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.966 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:17.967 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:17.968 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:17.969 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:17.970 DEBUG:boto:Method: GET 2022-09-09T22:38:17.971 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:17.972 DEBUG:boto:Data: 2022-09-09T22:38:17.973 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.974 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:17.975 DEBUG:boto:Port: 8001 2022-09-09T22:38:17.976 DEBUG:boto:Params: {} 2022-09-09T22:38:17.977 DEBUG:boto:Token: None 2022-09-09T22:38:17.979 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:17 GMT /jkcgbj-18/ 2022-09-09T22:38:17.980 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Z2AIpI//cgUVljm3GHBcPJEtCtI= 2022-09-09T22:38:17.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Z2AIpI//cgUVljm3GHBcPJEtCtI=', 'Content-Length': '0'} 2022-09-09T22:38:17.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000568c83896beec19e-00631bc059-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:17.989 DEBUG:boto:b'jkcgbj-18obj-test-zone31000falseobj-test-zone32022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:17.990 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:17.991 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:17.992 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:17.993 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:17.995 DEBUG:boto:Method: GET 2022-09-09T22:38:17.996 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:17.997 DEBUG:boto:Data: 2022-09-09T22:38:17.998 DEBUG:boto:Headers: {} 2022-09-09T22:38:17.999 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:18.000 DEBUG:boto:Port: 8000 2022-09-09T22:38:18.001 DEBUG:boto:Params: {} 2022-09-09T22:38:18.002 DEBUG:boto:Token: None 2022-09-09T22:38:18.003 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:18 GMT /jkcgbj-18/ 2022-09-09T22:38:18.004 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:AwjvvydC68HrwzwTTu2bdqDuokw= 2022-09-09T22:38:18.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:AwjvvydC68HrwzwTTu2bdqDuokw=', 'Content-Length': '0'} 2022-09-09T22:38:18.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fd3de55fe0498be-00631bc05a-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.015 DEBUG:boto:b'jkcgbj-18obj-test-zone31000falseobj-test-zone32022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:18.015 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:18.016 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:18.016 DEBUG:boto:Method: HEAD 2022-09-09T22:38:18.017 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3 2022-09-09T22:38:18.017 DEBUG:boto:Data: 2022-09-09T22:38:18.018 DEBUG:boto:Headers: {} 2022-09-09T22:38:18.018 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:18.019 DEBUG:boto:Port: 8001 2022-09-09T22:38:18.019 DEBUG:boto:Params: {} 2022-09-09T22:38:18.020 DEBUG:boto:Token: None 2022-09-09T22:38:18.021 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:18 GMT /jkcgbj-18/obj-test-zone3 2022-09-09T22:38:18.021 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9GJ2/hr46t+ecsshwP5MHvpCD9Q= 2022-09-09T22:38:18.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9GJ2/hr46t+ecsshwP5MHvpCD9Q=', 'Content-Length': '0'} 2022-09-09T22:38:18.026 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006036ca8e597ee1fa-00631bc05a-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.027 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:18.028 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:18.028 DEBUG:boto:Method: HEAD 2022-09-09T22:38:18.029 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3 2022-09-09T22:38:18.029 DEBUG:boto:Data: 2022-09-09T22:38:18.030 DEBUG:boto:Headers: {} 2022-09-09T22:38:18.030 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:18.031 DEBUG:boto:Port: 8000 2022-09-09T22:38:18.031 DEBUG:boto:Params: {} 2022-09-09T22:38:18.032 DEBUG:boto:Token: None 2022-09-09T22:38:18.033 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:18 GMT /jkcgbj-18/obj-test-zone3 2022-09-09T22:38:18.033 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9GJ2/hr46t+ecsshwP5MHvpCD9Q= 2022-09-09T22:38:18.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9GJ2/hr46t+ecsshwP5MHvpCD9Q=', 'Content-Length': '0'} 2022-09-09T22:38:18.038 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000578d2291077d9150-00631bc05a-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:38:18.040 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:18.040 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:18.041 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:18.042 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:18.043 DEBUG:boto:Method: GET 2022-09-09T22:38:18.043 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:18.044 DEBUG:boto:Data: 2022-09-09T22:38:18.045 DEBUG:boto:Headers: {} 2022-09-09T22:38:18.045 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:18.046 DEBUG:boto:Port: 8001 2022-09-09T22:38:18.046 DEBUG:boto:Params: {} 2022-09-09T22:38:18.047 DEBUG:boto:Token: None 2022-09-09T22:38:18.048 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:18 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:18.049 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:cL2GJcd2kiZFMMU3mpa0jcsweIs= 2022-09-09T22:38:18.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:cL2GJcd2kiZFMMU3mpa0jcsweIs=', 'Content-Length': '0'} 2022-09-09T22:38:18.053 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002058919bac1d4dec-00631bc05a-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.055 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:18.056 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:18.057 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:18.058 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:18.059 DEBUG:boto:Method: GET 2022-09-09T22:38:18.060 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:18.061 DEBUG:boto:Data: 2022-09-09T22:38:18.062 DEBUG:boto:Headers: {} 2022-09-09T22:38:18.063 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:18.064 DEBUG:boto:Port: 8000 2022-09-09T22:38:18.065 DEBUG:boto:Params: {} 2022-09-09T22:38:18.066 DEBUG:boto:Token: None 2022-09-09T22:38:18.067 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:18 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:38:18.068 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:cL2GJcd2kiZFMMU3mpa0jcsweIs= 2022-09-09T22:38:18.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:cL2GJcd2kiZFMMU3mpa0jcsweIs=', 'Content-Length': '0'} 2022-09-09T22:38:18.074 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:36:03 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000068f3f0242a678a4d-00631bc05a-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.075 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:18.076 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:18.077 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:18.078 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:18.079 DEBUG:boto:Method: GET 2022-09-09T22:38:18.080 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:18.080 DEBUG:boto:Data: 2022-09-09T22:38:18.081 DEBUG:boto:Headers: {} 2022-09-09T22:38:18.081 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:18.082 DEBUG:boto:Port: 8001 2022-09-09T22:38:18.082 DEBUG:boto:Params: {} 2022-09-09T22:38:18.083 DEBUG:boto:Token: None 2022-09-09T22:38:18.083 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:18 GMT /jkcgbj-18/ 2022-09-09T22:38:18.084 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:AwjvvydC68HrwzwTTu2bdqDuokw= 2022-09-09T22:38:18.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:AwjvvydC68HrwzwTTu2bdqDuokw=', 'Content-Length': '0'} 2022-09-09T22:38:18.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057eeb616122f4526-00631bc05a-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.091 DEBUG:boto:b'jkcgbj-18obj-test-zone31000falseobj-test-zone32022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:18.092 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:18.093 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:18.094 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:18.095 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:18.096 DEBUG:boto:Method: GET 2022-09-09T22:38:18.096 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:38:18.097 DEBUG:boto:Data: 2022-09-09T22:38:18.097 DEBUG:boto:Headers: {} 2022-09-09T22:38:18.098 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:18.098 DEBUG:boto:Port: 8000 2022-09-09T22:38:18.099 DEBUG:boto:Params: {} 2022-09-09T22:38:18.099 DEBUG:boto:Token: None 2022-09-09T22:38:18.100 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:18 GMT /jkcgbj-18/ 2022-09-09T22:38:18.101 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:AwjvvydC68HrwzwTTu2bdqDuokw= 2022-09-09T22:38:18.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:AwjvvydC68HrwzwTTu2bdqDuokw=', 'Content-Length': '0'} 2022-09-09T22:38:18.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085eb46b99b35b307-00631bc05a-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.108 DEBUG:boto:b'jkcgbj-18obj-test-zone31000falseobj-test-zone32022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:18.109 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-18 zones={test-zone3, test-zone2} 2022-09-09T22:38:18.110 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:18.110 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:18.111 DEBUG:boto:Method: HEAD 2022-09-09T22:38:18.111 DEBUG:boto:Path: /jkcgbj-16/ 2022-09-09T22:38:18.112 DEBUG:boto:Data: 2022-09-09T22:38:18.113 DEBUG:boto:Headers: {} 2022-09-09T22:38:18.113 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:18.114 DEBUG:boto:Port: 8000 2022-09-09T22:38:18.114 DEBUG:boto:Params: {} 2022-09-09T22:38:18.115 DEBUG:boto:Token: None 2022-09-09T22:38:18.116 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:18 GMT /jkcgbj-16/ 2022-09-09T22:38:18.117 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+PDALIVbEs8NMyiy6XqXRGKnuIA= 2022-09-09T22:38:18.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+PDALIVbEs8NMyiy6XqXRGKnuIA=', 'Content-Length': '0'} 2022-09-09T22:38:18.122 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000852e65153d254b38-00631bc05a-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.123 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:18.123 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:18.124 DEBUG:boto:path=/jkcgbj-16/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-09T22:38:18.124 DEBUG:boto:auth_path=/jkcgbj-16/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-09T22:38:18.125 DEBUG:boto:Method: GET 2022-09-09T22:38:18.125 DEBUG:boto:Path: /jkcgbj-16/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-09T22:38:18.126 DEBUG:boto:Data: 2022-09-09T22:38:18.126 DEBUG:boto:Headers: {} 2022-09-09T22:38:18.127 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:18.127 DEBUG:boto:Port: 8000 2022-09-09T22:38:18.128 DEBUG:boto:Params: {} 2022-09-09T22:38:18.128 DEBUG:boto:Token: None 2022-09-09T22:38:18.129 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:18 GMT /jkcgbj-16/?versions 2022-09-09T22:38:18.130 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bsqdoo2zkDq3uyME3ec+c3YEygM= 2022-09-09T22:38:18.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bsqdoo2zkDq3uyME3ec+c3YEygM=', 'Content-Length': '0'} 2022-09-09T22:38:18.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068286b994ac0184e-00631bc05a-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.137 DEBUG:boto:b'jkcgbj-16obj-test-zone1999falseobj-test-zone1123zMLKlbUHrE3mBcpUCJblKnWDeBXDtrue2022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:18.137 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:18.138 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:38:18.138 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?acl 2022-09-09T22:38:18.139 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?acl 2022-09-09T22:38:18.139 DEBUG:boto:Method: PUT 2022-09-09T22:38:18.140 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?acl 2022-09-09T22:38:18.140 DEBUG:boto:Data: 2022-09-09T22:38:18.141 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-09T22:38:18.141 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:18.142 DEBUG:boto:Port: 8000 2022-09-09T22:38:18.143 DEBUG:boto:Params: {} 2022-09-09T22:38:18.143 DEBUG:boto:Token: None 2022-09-09T22:38:18.144 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:38:18 GMT x-amz-acl:public-read /jkcgbj-16/obj-test-zone1?acl 2022-09-09T22:38:18.144 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:GWwqKXU1euzfCzTvGuK+3BEZ1Nw= 2022-09-09T22:38:18.145 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:GWwqKXU1euzfCzTvGuK+3BEZ1Nw=', 'Content-Length': '0'} 2022-09-09T22:38:18.160 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003ba4dc1c1a4af637-00631bc05a-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.162 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:18.163 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:18.164 DEBUG:boto:Method: HEAD 2022-09-09T22:38:18.164 DEBUG:boto:Path: /jkcgbj-16/ 2022-09-09T22:38:18.165 DEBUG:boto:Data: 2022-09-09T22:38:18.166 DEBUG:boto:Headers: {} 2022-09-09T22:38:18.167 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:18.168 DEBUG:boto:Port: 8000 2022-09-09T22:38:18.168 DEBUG:boto:Params: {} 2022-09-09T22:38:18.170 DEBUG:boto:Token: None 2022-09-09T22:38:18.170 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:18 GMT /jkcgbj-16/ 2022-09-09T22:38:18.171 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+PDALIVbEs8NMyiy6XqXRGKnuIA= 2022-09-09T22:38:18.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+PDALIVbEs8NMyiy6XqXRGKnuIA=', 'Content-Length': '0'} 2022-09-09T22:38:18.178 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000013de11c0614a904-00631bc05a-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.180 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:18.181 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:18.181 DEBUG:boto:path=/jkcgbj-16/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-09T22:38:18.182 DEBUG:boto:auth_path=/jkcgbj-16/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-09T22:38:18.183 DEBUG:boto:Method: GET 2022-09-09T22:38:18.184 DEBUG:boto:Path: /jkcgbj-16/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-09T22:38:18.185 DEBUG:boto:Data: 2022-09-09T22:38:18.186 DEBUG:boto:Headers: {} 2022-09-09T22:38:18.187 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:18.188 DEBUG:boto:Port: 8000 2022-09-09T22:38:18.189 DEBUG:boto:Params: {} 2022-09-09T22:38:18.190 DEBUG:boto:Token: None 2022-09-09T22:38:18.191 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:18 GMT /jkcgbj-16/?versions 2022-09-09T22:38:18.192 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bsqdoo2zkDq3uyME3ec+c3YEygM= 2022-09-09T22:38:18.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bsqdoo2zkDq3uyME3ec+c3YEygM=', 'Content-Length': '0'} 2022-09-09T22:38:18.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029cacfdddcff5e56-00631bc05a-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.242 DEBUG:boto:b'jkcgbj-16obj-test-zone2999falseobj-test-zone2MZ.b57oN5HI9SlBmYt.8.2016m6aPMftrue2022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:18.243 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:18.244 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:38:18.244 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?acl 2022-09-09T22:38:18.245 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?acl 2022-09-09T22:38:18.246 DEBUG:boto:Method: PUT 2022-09-09T22:38:18.246 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?acl 2022-09-09T22:38:18.247 DEBUG:boto:Data: 2022-09-09T22:38:18.248 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-09T22:38:18.248 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:18.249 DEBUG:boto:Port: 8000 2022-09-09T22:38:18.249 DEBUG:boto:Params: {} 2022-09-09T22:38:18.250 DEBUG:boto:Token: None 2022-09-09T22:38:18.251 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:38:18 GMT x-amz-acl:public-read /jkcgbj-16/obj-test-zone2?acl 2022-09-09T22:38:18.251 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:DUPSoJDdj8BFi47XL3n1c9UfuOQ= 2022-09-09T22:38:18.252 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:DUPSoJDdj8BFi47XL3n1c9UfuOQ=', 'Content-Length': '0'} 2022-09-09T22:38:18.266 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003043df3faedcbba8-00631bc05a-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.267 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:18.268 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:18.268 DEBUG:boto:Method: HEAD 2022-09-09T22:38:18.268 DEBUG:boto:Path: /jkcgbj-16/ 2022-09-09T22:38:18.269 DEBUG:boto:Data: 2022-09-09T22:38:18.270 DEBUG:boto:Headers: {} 2022-09-09T22:38:18.271 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:18.272 DEBUG:boto:Port: 8001 2022-09-09T22:38:18.272 DEBUG:boto:Params: {} 2022-09-09T22:38:18.274 DEBUG:boto:Token: None 2022-09-09T22:38:18.275 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:18 GMT /jkcgbj-16/ 2022-09-09T22:38:18.276 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+PDALIVbEs8NMyiy6XqXRGKnuIA= 2022-09-09T22:38:18.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+PDALIVbEs8NMyiy6XqXRGKnuIA=', 'Content-Length': '0'} 2022-09-09T22:38:18.283 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000316fe2a30b51a21f-00631bc05a-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.284 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:38:18.285 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:38:18.286 DEBUG:boto:path=/jkcgbj-16/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-09T22:38:18.287 DEBUG:boto:auth_path=/jkcgbj-16/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-09T22:38:18.288 DEBUG:boto:Method: GET 2022-09-09T22:38:18.289 DEBUG:boto:Path: /jkcgbj-16/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-09T22:38:18.290 DEBUG:boto:Data: 2022-09-09T22:38:18.291 DEBUG:boto:Headers: {} 2022-09-09T22:38:18.292 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:18.292 DEBUG:boto:Port: 8001 2022-09-09T22:38:18.293 DEBUG:boto:Params: {} 2022-09-09T22:38:18.294 DEBUG:boto:Token: None 2022-09-09T22:38:18.295 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:18 GMT /jkcgbj-16/?versions 2022-09-09T22:38:18.296 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bsqdoo2zkDq3uyME3ec+c3YEygM= 2022-09-09T22:38:18.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bsqdoo2zkDq3uyME3ec+c3YEygM=', 'Content-Length': '0'} 2022-09-09T22:38:18.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a41f3211e7a62317-00631bc05a-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.346 DEBUG:boto:b'jkcgbj-16obj-test-zone3999falseobj-test-zone3AesdIyOhh8xrq24fWFi6fMM39PxhXM8true2022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:18.347 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:18.348 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:38:18.348 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?acl 2022-09-09T22:38:18.349 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?acl 2022-09-09T22:38:18.350 DEBUG:boto:Method: PUT 2022-09-09T22:38:18.351 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?acl 2022-09-09T22:38:18.351 DEBUG:boto:Data: 2022-09-09T22:38:18.352 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-09T22:38:18.353 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:18.355 DEBUG:boto:Port: 8001 2022-09-09T22:38:18.356 DEBUG:boto:Params: {} 2022-09-09T22:38:18.357 DEBUG:boto:Token: None 2022-09-09T22:38:18.358 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:38:18 GMT x-amz-acl:public-read /jkcgbj-16/obj-test-zone3?acl 2022-09-09T22:38:18.359 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9N7gnZ5CKm6GDe/KPp3udZf7BPU= 2022-09-09T22:38:18.360 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9N7gnZ5CKm6GDe/KPp3udZf7BPU=', 'Content-Length': '0'} 2022-09-09T22:38:18.376 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006b5e7092dce979de-00631bc05a-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.377 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:18.377 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:18.378 DEBUG:boto:Method: HEAD 2022-09-09T22:38:18.379 DEBUG:boto:Path: /jkcgbj-17/ 2022-09-09T22:38:18.380 DEBUG:boto:Data: 2022-09-09T22:38:18.380 DEBUG:boto:Headers: {} 2022-09-09T22:38:18.381 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:18.381 DEBUG:boto:Port: 8000 2022-09-09T22:38:18.382 DEBUG:boto:Params: {} 2022-09-09T22:38:18.382 DEBUG:boto:Token: None 2022-09-09T22:38:18.383 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:18 GMT /jkcgbj-17/ 2022-09-09T22:38:18.384 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8xz7iFPPbHci1Nh0UaTEXnaUkQE= 2022-09-09T22:38:18.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8xz7iFPPbHci1Nh0UaTEXnaUkQE=', 'Content-Length': '0'} 2022-09-09T22:38:18.389 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ce448960b21f7b75-00631bc05a-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.389 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:18.390 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:18.391 DEBUG:boto:path=/jkcgbj-17/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-09T22:38:18.391 DEBUG:boto:auth_path=/jkcgbj-17/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-09T22:38:18.391 DEBUG:boto:Method: GET 2022-09-09T22:38:18.392 DEBUG:boto:Path: /jkcgbj-17/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-09T22:38:18.392 DEBUG:boto:Data: 2022-09-09T22:38:18.393 DEBUG:boto:Headers: {} 2022-09-09T22:38:18.393 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:18.394 DEBUG:boto:Port: 8000 2022-09-09T22:38:18.394 DEBUG:boto:Params: {} 2022-09-09T22:38:18.395 DEBUG:boto:Token: None 2022-09-09T22:38:18.396 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:18 GMT /jkcgbj-17/?versions 2022-09-09T22:38:18.396 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:45jYBtgjGM9hYg8ggb7HrrFsbaw= 2022-09-09T22:38:18.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:45jYBtgjGM9hYg8ggb7HrrFsbaw=', 'Content-Length': '0'} 2022-09-09T22:38:18.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009592fe61f4ac134b-00631bc05a-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.446 DEBUG:boto:b'jkcgbj-17obj-test-zone1999falseobj-test-zone1uPhFM-4pEeqszoz0jfwkNTqs81x9txytrue2022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:18.448 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:18.449 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:38:18.450 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?acl 2022-09-09T22:38:18.451 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?acl 2022-09-09T22:38:18.451 DEBUG:boto:Method: PUT 2022-09-09T22:38:18.452 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?acl 2022-09-09T22:38:18.452 DEBUG:boto:Data: 2022-09-09T22:38:18.453 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-09T22:38:18.455 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:18.456 DEBUG:boto:Port: 8000 2022-09-09T22:38:18.457 DEBUG:boto:Params: {} 2022-09-09T22:38:18.458 DEBUG:boto:Token: None 2022-09-09T22:38:18.459 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:38:18 GMT x-amz-acl:public-read /jkcgbj-17/obj-test-zone1?acl 2022-09-09T22:38:18.460 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:K9FzfUzocCZbIyZv4145ys230aw= 2022-09-09T22:38:18.461 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:K9FzfUzocCZbIyZv4145ys230aw=', 'Content-Length': '0'} 2022-09-09T22:38:18.477 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007441ab5fb5af693e-00631bc05a-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.478 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:18.479 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:18.480 DEBUG:boto:Method: HEAD 2022-09-09T22:38:18.481 DEBUG:boto:Path: /jkcgbj-17/ 2022-09-09T22:38:18.482 DEBUG:boto:Data: 2022-09-09T22:38:18.483 DEBUG:boto:Headers: {} 2022-09-09T22:38:18.484 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:18.485 DEBUG:boto:Port: 8000 2022-09-09T22:38:18.486 DEBUG:boto:Params: {} 2022-09-09T22:38:18.487 DEBUG:boto:Token: None 2022-09-09T22:38:18.489 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:18 GMT /jkcgbj-17/ 2022-09-09T22:38:18.490 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8xz7iFPPbHci1Nh0UaTEXnaUkQE= 2022-09-09T22:38:18.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8xz7iFPPbHci1Nh0UaTEXnaUkQE=', 'Content-Length': '0'} 2022-09-09T22:38:18.496 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031b732251d9e23fc-00631bc05a-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.497 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:18.497 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:18.498 DEBUG:boto:path=/jkcgbj-17/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-09T22:38:18.498 DEBUG:boto:auth_path=/jkcgbj-17/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-09T22:38:18.499 DEBUG:boto:Method: GET 2022-09-09T22:38:18.499 DEBUG:boto:Path: /jkcgbj-17/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-09T22:38:18.500 DEBUG:boto:Data: 2022-09-09T22:38:18.502 DEBUG:boto:Headers: {} 2022-09-09T22:38:18.503 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:18.504 DEBUG:boto:Port: 8000 2022-09-09T22:38:18.504 DEBUG:boto:Params: {} 2022-09-09T22:38:18.506 DEBUG:boto:Token: None 2022-09-09T22:38:18.507 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:18 GMT /jkcgbj-17/?versions 2022-09-09T22:38:18.508 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:45jYBtgjGM9hYg8ggb7HrrFsbaw= 2022-09-09T22:38:18.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:45jYBtgjGM9hYg8ggb7HrrFsbaw=', 'Content-Length': '0'} 2022-09-09T22:38:18.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e834dd712c9c95fb-00631bc05a-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.516 DEBUG:boto:b'jkcgbj-17obj-test-zone2999falseobj-test-zone217lUOlN45NLK2DSLxrYnqbOAg6jm2Tgtrue2022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:18.518 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:18.519 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:38:18.519 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?acl 2022-09-09T22:38:18.520 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?acl 2022-09-09T22:38:18.521 DEBUG:boto:Method: PUT 2022-09-09T22:38:18.522 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?acl 2022-09-09T22:38:18.523 DEBUG:boto:Data: 2022-09-09T22:38:18.524 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-09T22:38:18.524 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:18.525 DEBUG:boto:Port: 8000 2022-09-09T22:38:18.526 DEBUG:boto:Params: {} 2022-09-09T22:38:18.527 DEBUG:boto:Token: None 2022-09-09T22:38:18.528 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:38:18 GMT x-amz-acl:public-read /jkcgbj-17/obj-test-zone2?acl 2022-09-09T22:38:18.529 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rmcrFXr4t4dfYXkiSOXnLBYFT5E= 2022-09-09T22:38:18.530 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rmcrFXr4t4dfYXkiSOXnLBYFT5E=', 'Content-Length': '0'} 2022-09-09T22:38:18.546 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000012baf31b14a4f7-00631bc05a-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.547 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:18.548 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:18.549 DEBUG:boto:Method: HEAD 2022-09-09T22:38:18.550 DEBUG:boto:Path: /jkcgbj-17/ 2022-09-09T22:38:18.551 DEBUG:boto:Data: 2022-09-09T22:38:18.552 DEBUG:boto:Headers: {} 2022-09-09T22:38:18.553 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:18.554 DEBUG:boto:Port: 8001 2022-09-09T22:38:18.555 DEBUG:boto:Params: {} 2022-09-09T22:38:18.556 DEBUG:boto:Token: None 2022-09-09T22:38:18.557 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:18 GMT /jkcgbj-17/ 2022-09-09T22:38:18.558 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8xz7iFPPbHci1Nh0UaTEXnaUkQE= 2022-09-09T22:38:18.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8xz7iFPPbHci1Nh0UaTEXnaUkQE=', 'Content-Length': '0'} 2022-09-09T22:38:18.565 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000884f0855581e3eed-00631bc05a-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.566 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:38:18.567 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:38:18.568 DEBUG:boto:path=/jkcgbj-17/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-09T22:38:18.569 DEBUG:boto:auth_path=/jkcgbj-17/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-09T22:38:18.570 DEBUG:boto:Method: GET 2022-09-09T22:38:18.571 DEBUG:boto:Path: /jkcgbj-17/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-09T22:38:18.572 DEBUG:boto:Data: 2022-09-09T22:38:18.573 DEBUG:boto:Headers: {} 2022-09-09T22:38:18.574 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:18.575 DEBUG:boto:Port: 8001 2022-09-09T22:38:18.575 DEBUG:boto:Params: {} 2022-09-09T22:38:18.576 DEBUG:boto:Token: None 2022-09-09T22:38:18.577 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:18 GMT /jkcgbj-17/?versions 2022-09-09T22:38:18.577 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:45jYBtgjGM9hYg8ggb7HrrFsbaw= 2022-09-09T22:38:18.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:45jYBtgjGM9hYg8ggb7HrrFsbaw=', 'Content-Length': '0'} 2022-09-09T22:38:18.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ed5caca30945a71-00631bc05a-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.585 DEBUG:boto:b'jkcgbj-17obj-test-zone3999falseobj-test-zone31RXW239HrAmSBD6KjNRB3qRTyY29OqZtrue2022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:18.586 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:18.587 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:38:18.588 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?acl 2022-09-09T22:38:18.589 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?acl 2022-09-09T22:38:18.589 DEBUG:boto:Method: PUT 2022-09-09T22:38:18.590 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?acl 2022-09-09T22:38:18.590 DEBUG:boto:Data: 2022-09-09T22:38:18.591 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-09T22:38:18.591 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:18.592 DEBUG:boto:Port: 8001 2022-09-09T22:38:18.592 DEBUG:boto:Params: {} 2022-09-09T22:38:18.593 DEBUG:boto:Token: None 2022-09-09T22:38:18.593 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:38:18 GMT x-amz-acl:public-read /jkcgbj-17/obj-test-zone3?acl 2022-09-09T22:38:18.594 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CbvMbSMoXyPEJM9gmr+6AWYjJGI= 2022-09-09T22:38:18.595 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CbvMbSMoXyPEJM9gmr+6AWYjJGI=', 'Content-Length': '0'} 2022-09-09T22:38:18.609 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000134a913dbbe10b5c-00631bc05a-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.610 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:18.611 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:18.613 DEBUG:boto:Method: HEAD 2022-09-09T22:38:18.613 DEBUG:boto:Path: /jkcgbj-18/ 2022-09-09T22:38:18.614 DEBUG:boto:Data: 2022-09-09T22:38:18.615 DEBUG:boto:Headers: {} 2022-09-09T22:38:18.616 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:18.617 DEBUG:boto:Port: 8000 2022-09-09T22:38:18.618 DEBUG:boto:Params: {} 2022-09-09T22:38:18.619 DEBUG:boto:Token: None 2022-09-09T22:38:18.620 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:18 GMT /jkcgbj-18/ 2022-09-09T22:38:18.621 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Sq/uIs7c1p4WV+lqB0/XuLhGFCM= 2022-09-09T22:38:18.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Sq/uIs7c1p4WV+lqB0/XuLhGFCM=', 'Content-Length': '0'} 2022-09-09T22:38:18.626 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000067ae7a6b4115d685-00631bc05a-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.627 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:18.628 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:18.629 DEBUG:boto:path=/jkcgbj-18/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-09T22:38:18.629 DEBUG:boto:auth_path=/jkcgbj-18/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-09T22:38:18.630 DEBUG:boto:Method: GET 2022-09-09T22:38:18.630 DEBUG:boto:Path: /jkcgbj-18/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-09T22:38:18.631 DEBUG:boto:Data: 2022-09-09T22:38:18.631 DEBUG:boto:Headers: {} 2022-09-09T22:38:18.632 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:18.634 DEBUG:boto:Port: 8000 2022-09-09T22:38:18.634 DEBUG:boto:Params: {} 2022-09-09T22:38:18.635 DEBUG:boto:Token: None 2022-09-09T22:38:18.636 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:18 GMT /jkcgbj-18/?versions 2022-09-09T22:38:18.637 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UgD7S0GIhTQb+DyKCwGP5/TCIJ0= 2022-09-09T22:38:18.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UgD7S0GIhTQb+DyKCwGP5/TCIJ0=', 'Content-Length': '0'} 2022-09-09T22:38:18.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000293a2b642f05647d-00631bc05a-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.644 DEBUG:boto:b'jkcgbj-18obj-test-zone1999falseobj-test-zone1a8lrwM74XPlF1mwXf8hNl9Y9ISdV53ltrue2022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:18.645 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:18.646 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:38:18.647 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?acl 2022-09-09T22:38:18.648 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?acl 2022-09-09T22:38:18.649 DEBUG:boto:Method: PUT 2022-09-09T22:38:18.650 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?acl 2022-09-09T22:38:18.650 DEBUG:boto:Data: 2022-09-09T22:38:18.651 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-09T22:38:18.652 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:38:18.653 DEBUG:boto:Port: 8000 2022-09-09T22:38:18.654 DEBUG:boto:Params: {} 2022-09-09T22:38:18.656 DEBUG:boto:Token: None 2022-09-09T22:38:18.657 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:38:18 GMT x-amz-acl:public-read /jkcgbj-18/obj-test-zone1?acl 2022-09-09T22:38:18.658 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UixlvUW1l532p3FRi8GoG9pOWkE= 2022-09-09T22:38:18.659 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UixlvUW1l532p3FRi8GoG9pOWkE=', 'Content-Length': '0'} 2022-09-09T22:38:18.674 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000ec42651244ed735-00631bc05a-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.675 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:18.676 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:18.677 DEBUG:boto:Method: HEAD 2022-09-09T22:38:18.678 DEBUG:boto:Path: /jkcgbj-18/ 2022-09-09T22:38:18.678 DEBUG:boto:Data: 2022-09-09T22:38:18.679 DEBUG:boto:Headers: {} 2022-09-09T22:38:18.680 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:18.681 DEBUG:boto:Port: 8000 2022-09-09T22:38:18.682 DEBUG:boto:Params: {} 2022-09-09T22:38:18.683 DEBUG:boto:Token: None 2022-09-09T22:38:18.684 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:18 GMT /jkcgbj-18/ 2022-09-09T22:38:18.684 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Sq/uIs7c1p4WV+lqB0/XuLhGFCM= 2022-09-09T22:38:18.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Sq/uIs7c1p4WV+lqB0/XuLhGFCM=', 'Content-Length': '0'} 2022-09-09T22:38:18.690 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8adb61786681809-00631bc05a-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.691 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:18.692 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:18.692 DEBUG:boto:path=/jkcgbj-18/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-09T22:38:18.693 DEBUG:boto:auth_path=/jkcgbj-18/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-09T22:38:18.693 DEBUG:boto:Method: GET 2022-09-09T22:38:18.694 DEBUG:boto:Path: /jkcgbj-18/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-09T22:38:18.695 DEBUG:boto:Data: 2022-09-09T22:38:18.695 DEBUG:boto:Headers: {} 2022-09-09T22:38:18.696 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:18.696 DEBUG:boto:Port: 8000 2022-09-09T22:38:18.697 DEBUG:boto:Params: {} 2022-09-09T22:38:18.697 DEBUG:boto:Token: None 2022-09-09T22:38:18.698 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:18 GMT /jkcgbj-18/?versions 2022-09-09T22:38:18.698 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UgD7S0GIhTQb+DyKCwGP5/TCIJ0= 2022-09-09T22:38:18.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UgD7S0GIhTQb+DyKCwGP5/TCIJ0=', 'Content-Length': '0'} 2022-09-09T22:38:18.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcb12a37f3d44f0c-00631bc05a-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.706 DEBUG:boto:b'jkcgbj-18obj-test-zone2999falseobj-test-zone2e.gQ6yFECscD-oc4OV0dmk6aAg9jH.otrue2022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:18.707 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:18.708 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:38:18.710 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?acl 2022-09-09T22:38:18.711 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?acl 2022-09-09T22:38:18.712 DEBUG:boto:Method: PUT 2022-09-09T22:38:18.713 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?acl 2022-09-09T22:38:18.714 DEBUG:boto:Data: 2022-09-09T22:38:18.715 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-09T22:38:18.716 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:38:18.717 DEBUG:boto:Port: 8000 2022-09-09T22:38:18.718 DEBUG:boto:Params: {} 2022-09-09T22:38:18.719 DEBUG:boto:Token: None 2022-09-09T22:38:18.720 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:38:18 GMT x-amz-acl:public-read /jkcgbj-18/obj-test-zone2?acl 2022-09-09T22:38:18.721 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:hA1tRtw6vkpUYpzpvfxIdQSWQ2I= 2022-09-09T22:38:18.722 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:hA1tRtw6vkpUYpzpvfxIdQSWQ2I=', 'Content-Length': '0'} 2022-09-09T22:38:18.738 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000edd5b9829db92d50-00631bc05a-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.739 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:18.739 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:18.740 DEBUG:boto:Method: HEAD 2022-09-09T22:38:18.740 DEBUG:boto:Path: /jkcgbj-18/ 2022-09-09T22:38:18.741 DEBUG:boto:Data: 2022-09-09T22:38:18.741 DEBUG:boto:Headers: {} 2022-09-09T22:38:18.743 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:18.743 DEBUG:boto:Port: 8001 2022-09-09T22:38:18.744 DEBUG:boto:Params: {} 2022-09-09T22:38:18.744 DEBUG:boto:Token: None 2022-09-09T22:38:18.745 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:38:18 GMT /jkcgbj-18/ 2022-09-09T22:38:18.745 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Sq/uIs7c1p4WV+lqB0/XuLhGFCM= 2022-09-09T22:38:18.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Sq/uIs7c1p4WV+lqB0/XuLhGFCM=', 'Content-Length': '0'} 2022-09-09T22:38:18.751 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba1630a4a5514ac3-00631bc05a-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.751 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:38:18.752 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:38:18.753 DEBUG:boto:path=/jkcgbj-18/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-09T22:38:18.753 DEBUG:boto:auth_path=/jkcgbj-18/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-09T22:38:18.754 DEBUG:boto:Method: GET 2022-09-09T22:38:18.754 DEBUG:boto:Path: /jkcgbj-18/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-09T22:38:18.755 DEBUG:boto:Data: 2022-09-09T22:38:18.755 DEBUG:boto:Headers: {} 2022-09-09T22:38:18.756 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:18.756 DEBUG:boto:Port: 8001 2022-09-09T22:38:18.757 DEBUG:boto:Params: {} 2022-09-09T22:38:18.757 DEBUG:boto:Token: None 2022-09-09T22:38:18.758 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:38:18 GMT /jkcgbj-18/?versions 2022-09-09T22:38:18.759 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UgD7S0GIhTQb+DyKCwGP5/TCIJ0= 2022-09-09T22:38:18.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UgD7S0GIhTQb+DyKCwGP5/TCIJ0=', 'Content-Length': '0'} 2022-09-09T22:38:18.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e94ba1bd23c7de3-00631bc05a-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.766 DEBUG:boto:b'jkcgbj-18obj-test-zone3999falseobj-test-zone3qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1true2022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:38:18.766 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:18.767 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:38:18.767 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?acl 2022-09-09T22:38:18.768 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?acl 2022-09-09T22:38:18.768 DEBUG:boto:Method: PUT 2022-09-09T22:38:18.769 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?acl 2022-09-09T22:38:18.769 DEBUG:boto:Data: 2022-09-09T22:38:18.770 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-09T22:38:18.770 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:38:18.771 DEBUG:boto:Port: 8001 2022-09-09T22:38:18.771 DEBUG:boto:Params: {} 2022-09-09T22:38:18.772 DEBUG:boto:Token: None 2022-09-09T22:38:18.772 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:38:18 GMT x-amz-acl:public-read /jkcgbj-18/obj-test-zone3?acl 2022-09-09T22:38:18.773 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:/Yt3lzvD7+Awn7ocRpn3+0Gc7Fo= 2022-09-09T22:38:18.774 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:38:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:/Yt3lzvD7+Awn7ocRpn3+0Gc7Fo=', 'Content-Length': '0'} 2022-09-09T22:38:18.784 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003817d31b67ead548-00631bc05a-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:38:18.784 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jkcgbj-16 2022-09-09T22:38:18.785 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:18.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:18.787 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:38:18.842 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:38:18.842 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:38:18.908 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:38:18.904+0000 7fe6dc63ddc0 1 waiting for incremental sync to catch up: 2022-09-09T22:38:18.909 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000010.24.5, , 00000000003.33.9, , 00000000007.28.9, , , , , ] 2022-09-09T22:38:18.910 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000011.27.5, , 00000000003.33.9, , 00000000007.28.9, , , , , ] 2022-09-09T22:38:23.911 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:38:23.904+0000 7fe6dc63ddc0 1 bucket sync caught up with source: 2022-09-09T22:38:23.912 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000011.27.5, , 00000000003.33.9, , 00000000008.36.5, , , , , ] 2022-09-09T22:38:23.912 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000011.27.5, , 00000000003.33.9, , 00000000007.28.9, , , , , ] 2022-09-09T22:38:23.912 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:38:23.904+0000 7fe6dc63ddc0 0 bucket checkpoint complete 2022-09-09T22:38:23.941 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jkcgbj-16 2022-09-09T22:38:23.942 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:23.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:23.943 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:38:23.998 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:38:23.999 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:38:24.067 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:38:24.060+0000 7f0679515dc0 1 bucket sync caught up with source: 2022-09-09T22:38:24.067 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000011.27.5, , 00000000003.33.9, , 00000000008.36.5, , , , , ] 2022-09-09T22:38:24.068 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000011.27.5, , 00000000003.33.9, , 00000000008.36.5, , , , , ] 2022-09-09T22:38:24.068 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:38:24.060+0000 7f0679515dc0 0 bucket checkpoint complete 2022-09-09T22:38:24.092 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jkcgbj-16 2022-09-09T22:38:24.093 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:24.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:24.094 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:38:24.148 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:38:24.148 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:38:24.216 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:38:24.212+0000 7f413025bdc0 1 waiting for incremental sync to catch up: 2022-09-09T22:38:24.216 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000010.24.5, , 00000000003.33.9, , 00000000007.28.9, , , , , ] 2022-09-09T22:38:24.217 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000011.27.5, , 00000000003.33.9, , 00000000008.36.5, , , , , ] 2022-09-09T22:38:29.219 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:38:29.212+0000 7f413025bdc0 1 bucket sync caught up with source: 2022-09-09T22:38:29.220 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000011.27.5, , 00000000003.33.9, , 00000000008.36.5, , , , , ] 2022-09-09T22:38:29.220 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000011.27.5, , 00000000003.33.9, , 00000000008.36.5, , , , , ] 2022-09-09T22:38:29.220 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:38:29.212+0000 7f413025bdc0 0 bucket checkpoint complete 2022-09-09T22:38:29.245 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jkcgbj-16 2022-09-09T22:38:29.246 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:29.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:38:29.247 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:38:29.301 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:38:29.302 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:38:29.370 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:38:29.364+0000 7f94b5c97dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:38:29.370 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000003.34.9, , 00000000010.49.5, , 00000000008.60.5, , , , , ] 2022-09-09T22:38:29.371 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000004.37.5, , 00000000011.58.5, , 00000000008.60.5, , , , , ] 2022-09-09T22:38:34.373 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:38:34.368+0000 7f94b5c97dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:38:34.373 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000003.34.9, , 00000000010.49.5, , 00000000008.60.5, , , , , ] 2022-09-09T22:38:34.374 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000004.37.5, , 00000000011.58.5, , 00000000008.60.5, , , , , ] 2022-09-09T22:38:39.376 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:38:39.368+0000 7f94b5c97dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:38:39.376 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000003.34.9, , 00000000010.49.5, , 00000000008.60.5, , , , , ] 2022-09-09T22:38:39.377 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000004.37.5, , 00000000011.58.5, , 00000000008.60.5, , , , , ] 2022-09-09T22:38:44.379 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:38:44.372+0000 7f94b5c97dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:38:44.379 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000004.37.5, , 00000000010.49.5, , 00000000008.60.5, , , , , ] 2022-09-09T22:38:44.379 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000004.37.5, , 00000000011.58.5, , 00000000008.60.5, , , , , ] 2022-09-09T22:38:49.381 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:38:49.376+0000 7f94b5c97dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:38:49.382 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000004.37.5, , 00000000010.49.5, , 00000000008.60.5, , , , , ] 2022-09-09T22:38:49.382 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000004.37.5, , 00000000011.58.5, , 00000000008.60.5, , , , , ] 2022-09-09T22:38:54.384 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:38:54.376+0000 7f94b5c97dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:38:54.384 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000004.37.5, , 00000000010.49.5, , 00000000008.60.5, , , , , ] 2022-09-09T22:38:54.385 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000004.37.5, , 00000000011.58.5, , 00000000008.60.5, , , , , ] 2022-09-09T22:38:59.388 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:38:59.380+0000 7f94b5c97dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:38:59.388 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000004.37.5, , 00000000010.49.5, , 00000000008.60.5, , , , , ] 2022-09-09T22:38:59.389 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000004.37.5, , 00000000011.58.5, , 00000000008.60.5, , , , , ] 2022-09-09T22:39:04.390 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:39:04.384+0000 7f94b5c97dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:39:04.390 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000004.37.5, , 00000000010.49.5, , 00000000008.60.5, , , , , ] 2022-09-09T22:39:04.391 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000004.37.5, , 00000000011.58.5, , 00000000008.60.5, , , , , ] 2022-09-09T22:39:09.393 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:39:09.387+0000 7f94b5c97dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:39:09.393 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000004.37.5, , 00000000010.49.5, , 00000000008.60.5, , , , , ] 2022-09-09T22:39:09.394 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000004.37.5, , 00000000011.58.5, , 00000000008.60.5, , , , , ] 2022-09-09T22:39:14.395 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:39:14.387+0000 7f94b5c97dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:39:14.396 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000004.37.5, , 00000000010.49.5, , 00000000008.60.5, , , , , ] 2022-09-09T22:39:14.396 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000004.37.5, , 00000000011.58.5, , 00000000008.60.5, , , , , ] 2022-09-09T22:39:19.398 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:39:19.391+0000 7f94b5c97dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:39:19.399 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000004.37.5, , 00000000010.49.5, , 00000000008.60.5, , , , , ] 2022-09-09T22:39:19.399 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000004.37.5, , 00000000011.58.5, , 00000000008.60.5, , , , , ] 2022-09-09T22:39:24.401 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:39:24.395+0000 7f94b5c97dc0 1 bucket sync caught up with source: 2022-09-09T22:39:24.402 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000004.37.5, , 00000000011.58.5, , 00000000008.60.5, , , , , ] 2022-09-09T22:39:24.402 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000004.37.5, , 00000000011.58.5, , 00000000008.60.5, , , , , ] 2022-09-09T22:39:24.402 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:39:24.395+0000 7f94b5c97dc0 0 bucket checkpoint complete 2022-09-09T22:39:24.426 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jkcgbj-16 2022-09-09T22:39:24.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:24.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:24.428 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:39:24.480 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:39:24.480 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:39:24.548 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:39:24.539+0000 7f2fabe55dc0 1 bucket sync caught up with source: 2022-09-09T22:39:24.548 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000004.37.5, , 00000000011.58.5, , 00000000008.60.5, , , , , ] 2022-09-09T22:39:24.549 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000004.37.5, , 00000000011.58.5, , 00000000008.60.5, , , , , ] 2022-09-09T22:39:24.549 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:39:24.539+0000 7f2fabe55dc0 0 bucket checkpoint complete 2022-09-09T22:39:24.574 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jkcgbj-16 2022-09-09T22:39:24.575 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jkcgbj-16 2022-09-09T22:39:24.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:24.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:24.577 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:39:24.631 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:39:24.632 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:39:24.701 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:39:24.695+0000 7f40adaaadc0 1 bucket sync caught up with source: 2022-09-09T22:39:24.702 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000004.37.5, , 00000000004.62.5, , 00000000015.60.5, , , , , ] 2022-09-09T22:39:24.702 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000004.37.5, , 00000000004.62.5, , 00000000015.60.5, , , , , ] 2022-09-09T22:39:24.702 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:39:24.695+0000 7f40adaaadc0 0 bucket checkpoint complete 2022-09-09T22:39:24.728 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jkcgbj-16 2022-09-09T22:39:24.729 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:24.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:24.730 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:39:24.783 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:39:24.784 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:39:24.862 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:39:24.859+0000 7f3384306dc0 1 bucket sync caught up with source: 2022-09-09T22:39:24.863 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000004.37.5, , 00000000004.62.5, , 00000000015.60.5, , , , , ] 2022-09-09T22:39:24.863 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000004.37.5, , 00000000004.62.5, , 00000000015.60.5, , , , , ] 2022-09-09T22:39:24.863 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:39:24.859+0000 7f3384306dc0 0 bucket checkpoint complete 2022-09-09T22:39:24.893 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jkcgbj-16 2022-09-09T22:39:24.894 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-16 zones={test-zone2, test-zone1} 2022-09-09T22:39:24.895 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:24.895 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:24.896 DEBUG:boto:Method: HEAD 2022-09-09T22:39:24.897 DEBUG:boto:Path: /jkcgbj-16/ 2022-09-09T22:39:24.897 DEBUG:boto:Data: 2022-09-09T22:39:24.898 DEBUG:boto:Headers: {} 2022-09-09T22:39:24.898 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:24.899 DEBUG:boto:Port: 8000 2022-09-09T22:39:24.899 DEBUG:boto:Params: {} 2022-09-09T22:39:24.900 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T22:39:24.901 DEBUG:boto:Token: None 2022-09-09T22:39:24.902 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:24 GMT /jkcgbj-16/ 2022-09-09T22:39:24.902 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:mPKoMZcxQM7Xnj+HAcwIsQNd/fs= 2022-09-09T22:39:24.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:24 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:mPKoMZcxQM7Xnj+HAcwIsQNd/fs=', 'Content-Length': '0'} 2022-09-09T22:39:24.909 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000726c0b655d11f973-00631bc09c-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:39:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:24.910 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:24.910 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:24.911 DEBUG:boto:Method: HEAD 2022-09-09T22:39:24.911 DEBUG:boto:Path: /jkcgbj-16/ 2022-09-09T22:39:24.912 DEBUG:boto:Data: 2022-09-09T22:39:24.912 DEBUG:boto:Headers: {} 2022-09-09T22:39:24.913 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:24.913 DEBUG:boto:Port: 8000 2022-09-09T22:39:24.914 DEBUG:boto:Params: {} 2022-09-09T22:39:24.915 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T22:39:24.915 DEBUG:boto:Token: None 2022-09-09T22:39:24.916 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:24 GMT /jkcgbj-16/ 2022-09-09T22:39:24.916 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:mPKoMZcxQM7Xnj+HAcwIsQNd/fs= 2022-09-09T22:39:24.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:24 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:mPKoMZcxQM7Xnj+HAcwIsQNd/fs=', 'Content-Length': '0'} 2022-09-09T22:39:24.922 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000019ad2a4288ae9ab5-00631bc09c-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:39:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:24.923 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:39:24.923 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:24.924 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:24.924 DEBUG:boto:path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:24.925 DEBUG:boto:auth_path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:24.925 DEBUG:boto:Method: GET 2022-09-09T22:39:24.926 DEBUG:boto:Path: /jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:24.927 DEBUG:boto:Data: 2022-09-09T22:39:24.927 DEBUG:boto:Headers: {} 2022-09-09T22:39:24.928 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:24.928 DEBUG:boto:Port: 8000 2022-09-09T22:39:24.929 DEBUG:boto:Params: {} 2022-09-09T22:39:24.929 DEBUG:boto:Token: None 2022-09-09T22:39:24.930 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:24 GMT /jkcgbj-16/?versions 2022-09-09T22:39:24.931 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0BK1pqCJ1jEKUpqhDlD+DauGmas= 2022-09-09T22:39:24.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:24 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0BK1pqCJ1jEKUpqhDlD+DauGmas=', 'Content-Length': '0'} 2022-09-09T22:39:24.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d143da7d366e8645-00631bc09c-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:24.938 DEBUG:boto:b'jkcgbj-16999falsedummyJ2BEYVUeYOtJqYkdQX3amzkINxiyxtstrue2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1123zMLKlbUHrE3mBcpUCJblKnWDeBXDtrue2022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2MZ.b57oN5HI9SlBmYt.8.2016m6aPMftrue2022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3AesdIyOhh8xrq24fWFi6fMM39PxhXM8true2022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:24.939 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:39:24.940 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:39:24.940 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:39:24.941 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:39:24.941 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:39:24.942 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:24.942 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:24.943 DEBUG:boto:path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:24.943 DEBUG:boto:auth_path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:24.944 DEBUG:boto:Method: GET 2022-09-09T22:39:24.944 DEBUG:boto:Path: /jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:24.945 DEBUG:boto:Data: 2022-09-09T22:39:24.946 DEBUG:boto:Headers: {} 2022-09-09T22:39:24.946 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:24.947 DEBUG:boto:Port: 8000 2022-09-09T22:39:24.947 DEBUG:boto:Params: {} 2022-09-09T22:39:24.948 DEBUG:boto:Token: None 2022-09-09T22:39:24.949 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:24 GMT /jkcgbj-16/?versions 2022-09-09T22:39:24.949 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0BK1pqCJ1jEKUpqhDlD+DauGmas= 2022-09-09T22:39:24.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:24 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0BK1pqCJ1jEKUpqhDlD+DauGmas=', 'Content-Length': '0'} 2022-09-09T22:39:24.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000672668ddf8c59c38-00631bc09c-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:24.956 DEBUG:boto:b'jkcgbj-16999falsedummyJ2BEYVUeYOtJqYkdQX3amzkINxiyxtstrue2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1123zMLKlbUHrE3mBcpUCJblKnWDeBXDtrue2022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2MZ.b57oN5HI9SlBmYt.8.2016m6aPMftrue2022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3AesdIyOhh8xrq24fWFi6fMM39PxhXM8true2022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:24.957 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:39:24.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:39:24.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:39:24.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:39:24.959 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:24.960 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:24.960 DEBUG:boto:path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:24.961 DEBUG:boto:auth_path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:24.961 DEBUG:boto:Method: GET 2022-09-09T22:39:24.962 DEBUG:boto:Path: /jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:24.962 DEBUG:boto:Data: 2022-09-09T22:39:24.963 DEBUG:boto:Headers: {} 2022-09-09T22:39:24.964 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:24.964 DEBUG:boto:Port: 8000 2022-09-09T22:39:24.964 DEBUG:boto:Params: {} 2022-09-09T22:39:24.965 DEBUG:boto:Token: None 2022-09-09T22:39:24.966 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:24 GMT /jkcgbj-16/?versions 2022-09-09T22:39:24.966 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0BK1pqCJ1jEKUpqhDlD+DauGmas= 2022-09-09T22:39:24.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:24 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0BK1pqCJ1jEKUpqhDlD+DauGmas=', 'Content-Length': '0'} 2022-09-09T22:39:24.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002143973f33ef9aa-00631bc09c-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:24.974 DEBUG:boto:b'jkcgbj-16999falsedummyJ2BEYVUeYOtJqYkdQX3amzkINxiyxtstrue2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1123zMLKlbUHrE3mBcpUCJblKnWDeBXDtrue2022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2MZ.b57oN5HI9SlBmYt.8.2016m6aPMftrue2022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3AesdIyOhh8xrq24fWFi6fMM39PxhXM8true2022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:24.975 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:24.976 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:24.976 DEBUG:boto:path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:24.977 DEBUG:boto:auth_path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:24.977 DEBUG:boto:Method: GET 2022-09-09T22:39:24.978 DEBUG:boto:Path: /jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:24.979 DEBUG:boto:Data: 2022-09-09T22:39:24.979 DEBUG:boto:Headers: {} 2022-09-09T22:39:24.980 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:24.980 DEBUG:boto:Port: 8000 2022-09-09T22:39:24.981 DEBUG:boto:Params: {} 2022-09-09T22:39:24.981 DEBUG:boto:Token: None 2022-09-09T22:39:24.982 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:24 GMT /jkcgbj-16/?versions 2022-09-09T22:39:24.983 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0BK1pqCJ1jEKUpqhDlD+DauGmas= 2022-09-09T22:39:24.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:24 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0BK1pqCJ1jEKUpqhDlD+DauGmas=', 'Content-Length': '0'} 2022-09-09T22:39:24.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048d61fd3202f7d4c-00631bc09c-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:24.990 DEBUG:boto:b'jkcgbj-16999falsedummyJ2BEYVUeYOtJqYkdQX3amzkINxiyxtstrue2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1123zMLKlbUHrE3mBcpUCJblKnWDeBXDtrue2022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2MZ.b57oN5HI9SlBmYt.8.2016m6aPMftrue2022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3AesdIyOhh8xrq24fWFi6fMM39PxhXM8true2022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:24.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:39:24.992 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:39:24.992 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:39:24.993 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:24.994 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:24.994 DEBUG:boto:Method: GET 2022-09-09T22:39:24.995 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:24.995 DEBUG:boto:Data: 2022-09-09T22:39:24.996 DEBUG:boto:Headers: {} 2022-09-09T22:39:24.997 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:24.997 DEBUG:boto:Port: 8000 2022-09-09T22:39:24.998 DEBUG:boto:Params: {} 2022-09-09T22:39:24.998 DEBUG:boto:Token: None 2022-09-09T22:39:24.999 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:24 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:25.000 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9ug+XdHn52xlCpPJxS2mrwpR7b4= 2022-09-09T22:39:25.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:24 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9ug+XdHn52xlCpPJxS2mrwpR7b4=', 'Content-Length': '0'} 2022-09-09T22:39:25.004 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cdcc0e26683d2e36-00631bc09c-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.005 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:39:25.005 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:39:25.006 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:25.006 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:25.007 DEBUG:boto:Method: GET 2022-09-09T22:39:25.007 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:25.008 DEBUG:boto:Data: 2022-09-09T22:39:25.009 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.009 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:25.010 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.010 DEBUG:boto:Params: {} 2022-09-09T22:39:25.011 DEBUG:boto:Token: None 2022-09-09T22:39:25.012 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:25.012 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:oqIjoyyQ/ctDdYIxjUEbMgtk8Fs= 2022-09-09T22:39:25.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:oqIjoyyQ/ctDdYIxjUEbMgtk8Fs=', 'Content-Length': '0'} 2022-09-09T22:39:25.016 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000060991dd746895079-00631bc09d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.017 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:39:25.018 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:39:25.018 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:25.019 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:25.020 DEBUG:boto:Method: HEAD 2022-09-09T22:39:25.020 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:25.021 DEBUG:boto:Data: 2022-09-09T22:39:25.021 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.022 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:25.023 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.023 DEBUG:boto:Params: {} 2022-09-09T22:39:25.024 DEBUG:boto:Token: None 2022-09-09T22:39:25.025 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:25.025 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:VuqJe1MAbAiO70Z86ElzwUyJl+0= 2022-09-09T22:39:25.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:VuqJe1MAbAiO70Z86ElzwUyJl+0=', 'Content-Length': '0'} 2022-09-09T22:39:25.029 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001bc4fc6ed41e2338-00631bc09d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.030 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:39:25.030 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:39:25.031 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:25.031 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:25.032 DEBUG:boto:Method: HEAD 2022-09-09T22:39:25.033 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:25.033 DEBUG:boto:Data: 2022-09-09T22:39:25.034 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.035 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:25.036 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.037 DEBUG:boto:Params: {} 2022-09-09T22:39:25.038 DEBUG:boto:Token: None 2022-09-09T22:39:25.039 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:25.040 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:VuqJe1MAbAiO70Z86ElzwUyJl+0= 2022-09-09T22:39:25.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:VuqJe1MAbAiO70Z86ElzwUyJl+0=', 'Content-Length': '0'} 2022-09-09T22:39:25.045 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004bd06dfc928954d9-00631bc09d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:39:25.047 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:39:25.048 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:39:25.049 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:25.050 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:25.051 DEBUG:boto:Method: GET 2022-09-09T22:39:25.051 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:25.052 DEBUG:boto:Data: 2022-09-09T22:39:25.053 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.054 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:25.055 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.056 DEBUG:boto:Params: {} 2022-09-09T22:39:25.057 DEBUG:boto:Token: None 2022-09-09T22:39:25.058 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:25.059 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:oqIjoyyQ/ctDdYIxjUEbMgtk8Fs= 2022-09-09T22:39:25.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:oqIjoyyQ/ctDdYIxjUEbMgtk8Fs=', 'Content-Length': '0'} 2022-09-09T22:39:25.064 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ca714f21004222ae-00631bc09d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.065 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:39:25.066 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:39:25.067 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:25.068 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:25.069 DEBUG:boto:Method: GET 2022-09-09T22:39:25.070 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:25.071 DEBUG:boto:Data: 2022-09-09T22:39:25.072 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.073 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:25.074 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.075 DEBUG:boto:Params: {} 2022-09-09T22:39:25.076 DEBUG:boto:Token: None 2022-09-09T22:39:25.077 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:25.078 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:oqIjoyyQ/ctDdYIxjUEbMgtk8Fs= 2022-09-09T22:39:25.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:oqIjoyyQ/ctDdYIxjUEbMgtk8Fs=', 'Content-Length': '0'} 2022-09-09T22:39:25.083 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000386ca669e89115d0-00631bc09d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.084 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:25.084 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:25.085 DEBUG:boto:path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:39:25.085 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:39:25.086 DEBUG:boto:Method: GET 2022-09-09T22:39:25.087 DEBUG:boto:Path: /jkcgbj-16/?prefix=dummy 2022-09-09T22:39:25.088 DEBUG:boto:Data: 2022-09-09T22:39:25.089 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.090 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:25.091 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.091 DEBUG:boto:Params: {} 2022-09-09T22:39:25.092 DEBUG:boto:Token: None 2022-09-09T22:39:25.092 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/ 2022-09-09T22:39:25.093 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bBVVuiL5JMqXrQOkrp44gZR6k9Q= 2022-09-09T22:39:25.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bBVVuiL5JMqXrQOkrp44gZR6k9Q=', 'Content-Length': '0'} 2022-09-09T22:39:25.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ce87975b15c988a-00631bc09d-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.142 DEBUG:boto:b'jkcgbj-16dummy1000falsedummy2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:25.143 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:25.144 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:25.144 DEBUG:boto:path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:39:25.145 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:39:25.145 DEBUG:boto:Method: GET 2022-09-09T22:39:25.146 DEBUG:boto:Path: /jkcgbj-16/?prefix=dummy 2022-09-09T22:39:25.146 DEBUG:boto:Data: 2022-09-09T22:39:25.147 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.147 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:25.148 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.148 DEBUG:boto:Params: {} 2022-09-09T22:39:25.149 DEBUG:boto:Token: None 2022-09-09T22:39:25.150 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/ 2022-09-09T22:39:25.150 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bBVVuiL5JMqXrQOkrp44gZR6k9Q= 2022-09-09T22:39:25.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bBVVuiL5JMqXrQOkrp44gZR6k9Q=', 'Content-Length': '0'} 2022-09-09T22:39:25.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae86e8e8fc314e8f-00631bc09d-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.158 DEBUG:boto:b'jkcgbj-16dummy1000falsedummy2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:25.159 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:39:25.159 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:39:25.160 DEBUG:boto:Method: HEAD 2022-09-09T22:39:25.160 DEBUG:boto:Path: /jkcgbj-16/dummy 2022-09-09T22:39:25.161 DEBUG:boto:Data: 2022-09-09T22:39:25.161 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.161 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:25.162 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.162 DEBUG:boto:Params: {} 2022-09-09T22:39:25.163 DEBUG:boto:Token: None 2022-09-09T22:39:25.163 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/dummy 2022-09-09T22:39:25.164 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ak98DQETuw7s783emAjrScWMEg0= 2022-09-09T22:39:25.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ak98DQETuw7s783emAjrScWMEg0=', 'Content-Length': '0'} 2022-09-09T22:39:25.169 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f9c7a05e5ac88929-00631bc09d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.169 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:39:25.170 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:39:25.170 DEBUG:boto:Method: HEAD 2022-09-09T22:39:25.171 DEBUG:boto:Path: /jkcgbj-16/dummy 2022-09-09T22:39:25.171 DEBUG:boto:Data: 2022-09-09T22:39:25.172 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.172 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:25.173 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.174 DEBUG:boto:Params: {} 2022-09-09T22:39:25.174 DEBUG:boto:Token: None 2022-09-09T22:39:25.175 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/dummy 2022-09-09T22:39:25.175 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ak98DQETuw7s783emAjrScWMEg0= 2022-09-09T22:39:25.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ak98DQETuw7s783emAjrScWMEg0=', 'Content-Length': '0'} 2022-09-09T22:39:25.180 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003622c2d4efc00b65-00631bc09d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:39:25.181 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:39:25.182 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:39:25.182 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:25.183 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:25.183 DEBUG:boto:Method: GET 2022-09-09T22:39:25.184 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:25.184 DEBUG:boto:Data: 2022-09-09T22:39:25.185 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.186 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:25.186 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.187 DEBUG:boto:Params: {} 2022-09-09T22:39:25.187 DEBUG:boto:Token: None 2022-09-09T22:39:25.188 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:25.188 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:oqIjoyyQ/ctDdYIxjUEbMgtk8Fs= 2022-09-09T22:39:25.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:oqIjoyyQ/ctDdYIxjUEbMgtk8Fs=', 'Content-Length': '0'} 2022-09-09T22:39:25.192 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009a15ff604c0c9cfd-00631bc09d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.193 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:39:25.193 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:39:25.194 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:25.194 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:25.195 DEBUG:boto:Method: GET 2022-09-09T22:39:25.195 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:25.196 DEBUG:boto:Data: 2022-09-09T22:39:25.196 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.197 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:25.197 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.198 DEBUG:boto:Params: {} 2022-09-09T22:39:25.198 DEBUG:boto:Token: None 2022-09-09T22:39:25.199 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:25.200 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:oqIjoyyQ/ctDdYIxjUEbMgtk8Fs= 2022-09-09T22:39:25.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:oqIjoyyQ/ctDdYIxjUEbMgtk8Fs=', 'Content-Length': '0'} 2022-09-09T22:39:25.203 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000352a9d712ce35124-00631bc09d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.204 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:25.204 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:25.205 DEBUG:boto:path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:39:25.205 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:39:25.206 DEBUG:boto:Method: GET 2022-09-09T22:39:25.206 DEBUG:boto:Path: /jkcgbj-16/?prefix=dummy 2022-09-09T22:39:25.207 DEBUG:boto:Data: 2022-09-09T22:39:25.207 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.208 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:25.208 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.209 DEBUG:boto:Params: {} 2022-09-09T22:39:25.209 DEBUG:boto:Token: None 2022-09-09T22:39:25.210 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/ 2022-09-09T22:39:25.210 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bBVVuiL5JMqXrQOkrp44gZR6k9Q= 2022-09-09T22:39:25.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bBVVuiL5JMqXrQOkrp44gZR6k9Q=', 'Content-Length': '0'} 2022-09-09T22:39:25.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000614c468728252411-00631bc09d-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.218 DEBUG:boto:b'jkcgbj-16dummy1000falsedummy2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:25.218 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:25.219 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:25.219 DEBUG:boto:path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:39:25.220 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:39:25.220 DEBUG:boto:Method: GET 2022-09-09T22:39:25.221 DEBUG:boto:Path: /jkcgbj-16/?prefix=dummy 2022-09-09T22:39:25.221 DEBUG:boto:Data: 2022-09-09T22:39:25.222 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.222 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:25.223 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.223 DEBUG:boto:Params: {} 2022-09-09T22:39:25.224 DEBUG:boto:Token: None 2022-09-09T22:39:25.225 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/ 2022-09-09T22:39:25.225 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bBVVuiL5JMqXrQOkrp44gZR6k9Q= 2022-09-09T22:39:25.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bBVVuiL5JMqXrQOkrp44gZR6k9Q=', 'Content-Length': '0'} 2022-09-09T22:39:25.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004057db960ec483f4-00631bc09d-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.274 DEBUG:boto:b'jkcgbj-16dummy1000falsedummy2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:25.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:39:25.275 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:25.276 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:25.276 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.277 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.277 DEBUG:boto:Method: GET 2022-09-09T22:39:25.278 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.278 DEBUG:boto:Data: 2022-09-09T22:39:25.279 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.280 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:25.280 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.281 DEBUG:boto:Params: {} 2022-09-09T22:39:25.281 DEBUG:boto:Token: None 2022-09-09T22:39:25.282 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.282 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:r9hgiOHRlN013xRapuDLVBP49mc= 2022-09-09T22:39:25.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:r9hgiOHRlN013xRapuDLVBP49mc=', 'Content-Length': '0'} 2022-09-09T22:39:25.286 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000036edd47c147b9b7b-00631bc09d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.287 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:25.288 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:25.288 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.289 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.290 DEBUG:boto:Method: GET 2022-09-09T22:39:25.290 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.291 DEBUG:boto:Data: 2022-09-09T22:39:25.291 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.291 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:25.292 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.292 DEBUG:boto:Params: {} 2022-09-09T22:39:25.293 DEBUG:boto:Token: None 2022-09-09T22:39:25.293 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.294 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:r9hgiOHRlN013xRapuDLVBP49mc= 2022-09-09T22:39:25.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:r9hgiOHRlN013xRapuDLVBP49mc=', 'Content-Length': '0'} 2022-09-09T22:39:25.298 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000478056f9579bdee9-00631bc09d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.299 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:25.299 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:25.300 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.300 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.301 DEBUG:boto:Method: HEAD 2022-09-09T22:39:25.301 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.302 DEBUG:boto:Data: 2022-09-09T22:39:25.302 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.303 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:25.303 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.304 DEBUG:boto:Params: {} 2022-09-09T22:39:25.304 DEBUG:boto:Token: None 2022-09-09T22:39:25.305 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.305 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:o26NRW8Asfn19smPbgkFbQ5xIQ0= 2022-09-09T22:39:25.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:o26NRW8Asfn19smPbgkFbQ5xIQ0=', 'Content-Length': '0'} 2022-09-09T22:39:25.309 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009bf665e02b5dbfea-00631bc09d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.310 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:25.310 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:25.311 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.311 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.312 DEBUG:boto:Method: HEAD 2022-09-09T22:39:25.312 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.313 DEBUG:boto:Data: 2022-09-09T22:39:25.313 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.314 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:25.314 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.315 DEBUG:boto:Params: {} 2022-09-09T22:39:25.315 DEBUG:boto:Token: None 2022-09-09T22:39:25.316 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.316 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:o26NRW8Asfn19smPbgkFbQ5xIQ0= 2022-09-09T22:39:25.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:o26NRW8Asfn19smPbgkFbQ5xIQ0=', 'Content-Length': '0'} 2022-09-09T22:39:25.320 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000445385eefdaf50c4-00631bc09d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:39:25.321 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:25.322 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:25.322 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.323 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.323 DEBUG:boto:Method: GET 2022-09-09T22:39:25.324 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.324 DEBUG:boto:Data: 2022-09-09T22:39:25.325 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.325 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:25.326 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.326 DEBUG:boto:Params: {} 2022-09-09T22:39:25.327 DEBUG:boto:Token: None 2022-09-09T22:39:25.327 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.328 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:r9hgiOHRlN013xRapuDLVBP49mc= 2022-09-09T22:39:25.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:r9hgiOHRlN013xRapuDLVBP49mc=', 'Content-Length': '0'} 2022-09-09T22:39:25.331 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a7e73ee45e099351-00631bc09d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.378 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:25.379 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:25.379 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.380 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.380 DEBUG:boto:Method: GET 2022-09-09T22:39:25.381 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.381 DEBUG:boto:Data: 2022-09-09T22:39:25.382 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.383 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:25.383 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.384 DEBUG:boto:Params: {} 2022-09-09T22:39:25.384 DEBUG:boto:Token: None 2022-09-09T22:39:25.385 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.386 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:r9hgiOHRlN013xRapuDLVBP49mc= 2022-09-09T22:39:25.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:r9hgiOHRlN013xRapuDLVBP49mc=', 'Content-Length': '0'} 2022-09-09T22:39:25.390 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e954dbcfdf0f31f-00631bc09d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.391 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:25.391 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:25.392 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:25.392 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:25.393 DEBUG:boto:Method: GET 2022-09-09T22:39:25.393 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:25.394 DEBUG:boto:Data: 2022-09-09T22:39:25.394 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.395 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:25.396 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.396 DEBUG:boto:Params: {} 2022-09-09T22:39:25.397 DEBUG:boto:Token: None 2022-09-09T22:39:25.397 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/ 2022-09-09T22:39:25.398 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bBVVuiL5JMqXrQOkrp44gZR6k9Q= 2022-09-09T22:39:25.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bBVVuiL5JMqXrQOkrp44gZR6k9Q=', 'Content-Length': '0'} 2022-09-09T22:39:25.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adc5f09f855bec0a-00631bc09d-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.405 DEBUG:boto:b'jkcgbj-16obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:25.406 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:25.406 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:25.407 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:25.407 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:25.408 DEBUG:boto:Method: GET 2022-09-09T22:39:25.408 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:25.409 DEBUG:boto:Data: 2022-09-09T22:39:25.409 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.410 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:25.410 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.411 DEBUG:boto:Params: {} 2022-09-09T22:39:25.411 DEBUG:boto:Token: None 2022-09-09T22:39:25.412 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/ 2022-09-09T22:39:25.413 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bBVVuiL5JMqXrQOkrp44gZR6k9Q= 2022-09-09T22:39:25.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bBVVuiL5JMqXrQOkrp44gZR6k9Q=', 'Content-Length': '0'} 2022-09-09T22:39:25.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acf207a8d7e47070-00631bc09d-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.419 DEBUG:boto:b'jkcgbj-16obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:25.420 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:25.421 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:25.421 DEBUG:boto:Method: HEAD 2022-09-09T22:39:25.422 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1 2022-09-09T22:39:25.423 DEBUG:boto:Data: 2022-09-09T22:39:25.423 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.424 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:25.424 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.425 DEBUG:boto:Params: {} 2022-09-09T22:39:25.425 DEBUG:boto:Token: None 2022-09-09T22:39:25.426 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/obj-test-zone1 2022-09-09T22:39:25.427 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sg10LPzPTTEZIesnUPvKS9BK0Hg= 2022-09-09T22:39:25.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sg10LPzPTTEZIesnUPvKS9BK0Hg=', 'Content-Length': '0'} 2022-09-09T22:39:25.431 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c387d66607fd8489-00631bc09d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.432 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:25.432 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:25.433 DEBUG:boto:Method: HEAD 2022-09-09T22:39:25.433 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1 2022-09-09T22:39:25.434 DEBUG:boto:Data: 2022-09-09T22:39:25.435 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.435 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:25.436 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.436 DEBUG:boto:Params: {} 2022-09-09T22:39:25.437 DEBUG:boto:Token: None 2022-09-09T22:39:25.437 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/obj-test-zone1 2022-09-09T22:39:25.438 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sg10LPzPTTEZIesnUPvKS9BK0Hg= 2022-09-09T22:39:25.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sg10LPzPTTEZIesnUPvKS9BK0Hg=', 'Content-Length': '0'} 2022-09-09T22:39:25.442 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000607f09aee6aec329-00631bc09d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:39:25.444 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:25.444 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:25.445 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.446 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.446 DEBUG:boto:Method: GET 2022-09-09T22:39:25.447 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.447 DEBUG:boto:Data: 2022-09-09T22:39:25.448 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.449 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:25.449 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.450 DEBUG:boto:Params: {} 2022-09-09T22:39:25.450 DEBUG:boto:Token: None 2022-09-09T22:39:25.451 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.452 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:r9hgiOHRlN013xRapuDLVBP49mc= 2022-09-09T22:39:25.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:r9hgiOHRlN013xRapuDLVBP49mc=', 'Content-Length': '0'} 2022-09-09T22:39:25.455 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008fe661bafc9d5033-00631bc09d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.498 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:25.499 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:25.499 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.500 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.500 DEBUG:boto:Method: GET 2022-09-09T22:39:25.501 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.501 DEBUG:boto:Data: 2022-09-09T22:39:25.502 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.503 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:25.503 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.504 DEBUG:boto:Params: {} 2022-09-09T22:39:25.504 DEBUG:boto:Token: None 2022-09-09T22:39:25.505 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:25.506 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:r9hgiOHRlN013xRapuDLVBP49mc= 2022-09-09T22:39:25.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:r9hgiOHRlN013xRapuDLVBP49mc=', 'Content-Length': '0'} 2022-09-09T22:39:25.509 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c9860711d66023b-00631bc09d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.510 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:25.511 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:25.511 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:25.512 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:25.513 DEBUG:boto:Method: GET 2022-09-09T22:39:25.513 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:25.514 DEBUG:boto:Data: 2022-09-09T22:39:25.514 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.515 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:25.515 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.516 DEBUG:boto:Params: {} 2022-09-09T22:39:25.516 DEBUG:boto:Token: None 2022-09-09T22:39:25.517 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/ 2022-09-09T22:39:25.518 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bBVVuiL5JMqXrQOkrp44gZR6k9Q= 2022-09-09T22:39:25.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bBVVuiL5JMqXrQOkrp44gZR6k9Q=', 'Content-Length': '0'} 2022-09-09T22:39:25.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005191d9e9b6acd442-00631bc09d-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.525 DEBUG:boto:b'jkcgbj-16obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:25.526 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:25.526 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:25.527 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:25.527 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:25.528 DEBUG:boto:Method: GET 2022-09-09T22:39:25.528 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:25.529 DEBUG:boto:Data: 2022-09-09T22:39:25.529 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.530 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:25.530 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.531 DEBUG:boto:Params: {} 2022-09-09T22:39:25.532 DEBUG:boto:Token: None 2022-09-09T22:39:25.532 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/ 2022-09-09T22:39:25.533 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bBVVuiL5JMqXrQOkrp44gZR6k9Q= 2022-09-09T22:39:25.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bBVVuiL5JMqXrQOkrp44gZR6k9Q=', 'Content-Length': '0'} 2022-09-09T22:39:25.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9eb598d8cdf0d7a-00631bc09d-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.540 DEBUG:boto:b'jkcgbj-16obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:25.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:39:25.541 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:25.542 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:25.542 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.543 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.543 DEBUG:boto:Method: GET 2022-09-09T22:39:25.544 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.544 DEBUG:boto:Data: 2022-09-09T22:39:25.545 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.545 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:25.546 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.546 DEBUG:boto:Params: {} 2022-09-09T22:39:25.547 DEBUG:boto:Token: None 2022-09-09T22:39:25.548 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.548 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2spqQKUmOZz52rFbiI2AYz5G1tk= 2022-09-09T22:39:25.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2spqQKUmOZz52rFbiI2AYz5G1tk=', 'Content-Length': '0'} 2022-09-09T22:39:25.552 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000043c8862eb4a1e110-00631bc09d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.553 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:25.553 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:25.554 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.554 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.555 DEBUG:boto:Method: GET 2022-09-09T22:39:25.555 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.556 DEBUG:boto:Data: 2022-09-09T22:39:25.556 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.557 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:25.558 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.558 DEBUG:boto:Params: {} 2022-09-09T22:39:25.559 DEBUG:boto:Token: None 2022-09-09T22:39:25.559 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.560 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2spqQKUmOZz52rFbiI2AYz5G1tk= 2022-09-09T22:39:25.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2spqQKUmOZz52rFbiI2AYz5G1tk=', 'Content-Length': '0'} 2022-09-09T22:39:25.564 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091f115a58dc65394-00631bc09d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.565 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:25.565 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:25.566 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.566 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.567 DEBUG:boto:Method: HEAD 2022-09-09T22:39:25.567 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.568 DEBUG:boto:Data: 2022-09-09T22:39:25.568 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.569 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:25.569 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.570 DEBUG:boto:Params: {} 2022-09-09T22:39:25.571 DEBUG:boto:Token: None 2022-09-09T22:39:25.571 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.572 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:O0uYR4ZnLUWZhmLBn6jWkK7du80= 2022-09-09T22:39:25.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:O0uYR4ZnLUWZhmLBn6jWkK7du80=', 'Content-Length': '0'} 2022-09-09T22:39:25.575 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9e4605d195785fd-00631bc09d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.576 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:25.576 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:25.577 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.577 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.578 DEBUG:boto:Method: HEAD 2022-09-09T22:39:25.578 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.579 DEBUG:boto:Data: 2022-09-09T22:39:25.579 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.580 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:25.580 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.581 DEBUG:boto:Params: {} 2022-09-09T22:39:25.581 DEBUG:boto:Token: None 2022-09-09T22:39:25.582 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.583 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:O0uYR4ZnLUWZhmLBn6jWkK7du80= 2022-09-09T22:39:25.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:O0uYR4ZnLUWZhmLBn6jWkK7du80=', 'Content-Length': '0'} 2022-09-09T22:39:25.586 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cff8e5627b571bc9-00631bc09d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:39:25.587 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:25.588 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:25.588 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.589 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.589 DEBUG:boto:Method: GET 2022-09-09T22:39:25.590 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.591 DEBUG:boto:Data: 2022-09-09T22:39:25.591 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.592 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:25.592 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.593 DEBUG:boto:Params: {} 2022-09-09T22:39:25.593 DEBUG:boto:Token: None 2022-09-09T22:39:25.594 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.594 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2spqQKUmOZz52rFbiI2AYz5G1tk= 2022-09-09T22:39:25.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2spqQKUmOZz52rFbiI2AYz5G1tk=', 'Content-Length': '0'} 2022-09-09T22:39:25.598 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044c3c72c0f8463d9-00631bc09d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.642 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:25.643 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:25.643 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.644 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.644 DEBUG:boto:Method: GET 2022-09-09T22:39:25.645 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.645 DEBUG:boto:Data: 2022-09-09T22:39:25.646 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.646 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:25.647 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.648 DEBUG:boto:Params: {} 2022-09-09T22:39:25.648 DEBUG:boto:Token: None 2022-09-09T22:39:25.649 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.649 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2spqQKUmOZz52rFbiI2AYz5G1tk= 2022-09-09T22:39:25.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2spqQKUmOZz52rFbiI2AYz5G1tk=', 'Content-Length': '0'} 2022-09-09T22:39:25.653 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a85b9bc60b34131d-00631bc09d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.654 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:25.654 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:25.655 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:25.655 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:25.656 DEBUG:boto:Method: GET 2022-09-09T22:39:25.657 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:25.657 DEBUG:boto:Data: 2022-09-09T22:39:25.658 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.658 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:25.659 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.659 DEBUG:boto:Params: {} 2022-09-09T22:39:25.660 DEBUG:boto:Token: None 2022-09-09T22:39:25.660 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/ 2022-09-09T22:39:25.661 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bBVVuiL5JMqXrQOkrp44gZR6k9Q= 2022-09-09T22:39:25.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bBVVuiL5JMqXrQOkrp44gZR6k9Q=', 'Content-Length': '0'} 2022-09-09T22:39:25.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f10ca9f528af497-00631bc09d-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.666 DEBUG:boto:b'jkcgbj-16obj-test-zone21000falseobj-test-zone22022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:25.667 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:25.668 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:25.668 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:25.669 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:25.669 DEBUG:boto:Method: GET 2022-09-09T22:39:25.670 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:25.670 DEBUG:boto:Data: 2022-09-09T22:39:25.671 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.671 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:25.672 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.673 DEBUG:boto:Params: {} 2022-09-09T22:39:25.673 DEBUG:boto:Token: None 2022-09-09T22:39:25.674 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/ 2022-09-09T22:39:25.674 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bBVVuiL5JMqXrQOkrp44gZR6k9Q= 2022-09-09T22:39:25.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bBVVuiL5JMqXrQOkrp44gZR6k9Q=', 'Content-Length': '0'} 2022-09-09T22:39:25.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2b9473b7904ce36-00631bc09d-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.681 DEBUG:boto:b'jkcgbj-16obj-test-zone21000falseobj-test-zone22022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:25.682 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:25.682 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:25.683 DEBUG:boto:Method: HEAD 2022-09-09T22:39:25.683 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2 2022-09-09T22:39:25.684 DEBUG:boto:Data: 2022-09-09T22:39:25.684 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.685 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:25.685 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.686 DEBUG:boto:Params: {} 2022-09-09T22:39:25.686 DEBUG:boto:Token: None 2022-09-09T22:39:25.687 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/obj-test-zone2 2022-09-09T22:39:25.687 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YY26ItPscPKnQHqZ/v1XFn7CHfs= 2022-09-09T22:39:25.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YY26ItPscPKnQHqZ/v1XFn7CHfs=', 'Content-Length': '0'} 2022-09-09T22:39:25.692 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4d9639ae4d19070-00631bc09d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.692 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:25.693 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:25.693 DEBUG:boto:Method: HEAD 2022-09-09T22:39:25.694 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2 2022-09-09T22:39:25.694 DEBUG:boto:Data: 2022-09-09T22:39:25.695 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.695 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:25.696 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.696 DEBUG:boto:Params: {} 2022-09-09T22:39:25.697 DEBUG:boto:Token: None 2022-09-09T22:39:25.697 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/obj-test-zone2 2022-09-09T22:39:25.698 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YY26ItPscPKnQHqZ/v1XFn7CHfs= 2022-09-09T22:39:25.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YY26ItPscPKnQHqZ/v1XFn7CHfs=', 'Content-Length': '0'} 2022-09-09T22:39:25.703 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b9ec271496459552-00631bc09d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:39:25.704 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:25.705 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:25.705 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.706 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.706 DEBUG:boto:Method: GET 2022-09-09T22:39:25.707 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.707 DEBUG:boto:Data: 2022-09-09T22:39:25.708 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.708 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:25.709 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.709 DEBUG:boto:Params: {} 2022-09-09T22:39:25.710 DEBUG:boto:Token: None 2022-09-09T22:39:25.711 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.711 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2spqQKUmOZz52rFbiI2AYz5G1tk= 2022-09-09T22:39:25.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2spqQKUmOZz52rFbiI2AYz5G1tk=', 'Content-Length': '0'} 2022-09-09T22:39:25.715 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a2de9a36fd68d0a-00631bc09d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.758 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:25.758 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:25.759 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.760 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.760 DEBUG:boto:Method: GET 2022-09-09T22:39:25.761 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.761 DEBUG:boto:Data: 2022-09-09T22:39:25.762 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.762 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:25.763 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.763 DEBUG:boto:Params: {} 2022-09-09T22:39:25.764 DEBUG:boto:Token: None 2022-09-09T22:39:25.765 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:25.765 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2spqQKUmOZz52rFbiI2AYz5G1tk= 2022-09-09T22:39:25.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2spqQKUmOZz52rFbiI2AYz5G1tk=', 'Content-Length': '0'} 2022-09-09T22:39:25.769 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac8b36421d043340-00631bc09d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.770 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:25.770 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:25.771 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:25.772 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:25.772 DEBUG:boto:Method: GET 2022-09-09T22:39:25.773 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:25.773 DEBUG:boto:Data: 2022-09-09T22:39:25.774 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.774 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:25.775 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.775 DEBUG:boto:Params: {} 2022-09-09T22:39:25.776 DEBUG:boto:Token: None 2022-09-09T22:39:25.776 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/ 2022-09-09T22:39:25.777 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bBVVuiL5JMqXrQOkrp44gZR6k9Q= 2022-09-09T22:39:25.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bBVVuiL5JMqXrQOkrp44gZR6k9Q=', 'Content-Length': '0'} 2022-09-09T22:39:25.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d4ca8ee929d3ceb-00631bc09d-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.784 DEBUG:boto:b'jkcgbj-16obj-test-zone21000falseobj-test-zone22022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:25.785 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:25.785 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:25.786 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:25.786 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:25.787 DEBUG:boto:Method: GET 2022-09-09T22:39:25.787 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:25.788 DEBUG:boto:Data: 2022-09-09T22:39:25.789 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.789 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:25.790 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.790 DEBUG:boto:Params: {} 2022-09-09T22:39:25.791 DEBUG:boto:Token: None 2022-09-09T22:39:25.791 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/ 2022-09-09T22:39:25.792 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bBVVuiL5JMqXrQOkrp44gZR6k9Q= 2022-09-09T22:39:25.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bBVVuiL5JMqXrQOkrp44gZR6k9Q=', 'Content-Length': '0'} 2022-09-09T22:39:25.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085e756b2163e7a58-00631bc09d-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.799 DEBUG:boto:b'jkcgbj-16obj-test-zone21000falseobj-test-zone22022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:25.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:39:25.800 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:25.801 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:25.801 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:25.802 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:25.802 DEBUG:boto:Method: GET 2022-09-09T22:39:25.803 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:25.803 DEBUG:boto:Data: 2022-09-09T22:39:25.804 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.805 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:25.805 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.806 DEBUG:boto:Params: {} 2022-09-09T22:39:25.806 DEBUG:boto:Token: None 2022-09-09T22:39:25.807 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:25.807 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:eSCpoJJszaFihsZscIStZ7J2ue0= 2022-09-09T22:39:25.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:eSCpoJJszaFihsZscIStZ7J2ue0=', 'Content-Length': '0'} 2022-09-09T22:39:25.811 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a52355ecd1319270-00631bc09d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.812 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:25.813 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:25.813 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:25.814 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:25.815 DEBUG:boto:Method: GET 2022-09-09T22:39:25.815 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:25.816 DEBUG:boto:Data: 2022-09-09T22:39:25.816 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.817 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:25.817 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.818 DEBUG:boto:Params: {} 2022-09-09T22:39:25.818 DEBUG:boto:Token: None 2022-09-09T22:39:25.819 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:25.820 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:eSCpoJJszaFihsZscIStZ7J2ue0= 2022-09-09T22:39:25.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:eSCpoJJszaFihsZscIStZ7J2ue0=', 'Content-Length': '0'} 2022-09-09T22:39:25.823 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001be098bdfd15bf2a-00631bc09d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.824 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:25.824 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:25.825 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:25.826 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:25.826 DEBUG:boto:Method: HEAD 2022-09-09T22:39:25.827 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:25.827 DEBUG:boto:Data: 2022-09-09T22:39:25.828 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.828 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:25.829 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.829 DEBUG:boto:Params: {} 2022-09-09T22:39:25.830 DEBUG:boto:Token: None 2022-09-09T22:39:25.830 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:25.831 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:EZVvscsh+V3FAWTQZqnVxVbkbBU= 2022-09-09T22:39:25.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:EZVvscsh+V3FAWTQZqnVxVbkbBU=', 'Content-Length': '0'} 2022-09-09T22:39:25.835 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001abcbf3650688654-00631bc09d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.836 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:25.836 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:25.837 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:25.837 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:25.838 DEBUG:boto:Method: HEAD 2022-09-09T22:39:25.838 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:25.839 DEBUG:boto:Data: 2022-09-09T22:39:25.839 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.840 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:25.840 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.841 DEBUG:boto:Params: {} 2022-09-09T22:39:25.841 DEBUG:boto:Token: None 2022-09-09T22:39:25.842 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:25.842 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:EZVvscsh+V3FAWTQZqnVxVbkbBU= 2022-09-09T22:39:25.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:EZVvscsh+V3FAWTQZqnVxVbkbBU=', 'Content-Length': '0'} 2022-09-09T22:39:25.846 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b37d4b64ffdf25b-00631bc09d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:39:25.847 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:25.848 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:25.848 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:25.849 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:25.849 DEBUG:boto:Method: GET 2022-09-09T22:39:25.850 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:25.850 DEBUG:boto:Data: 2022-09-09T22:39:25.851 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.851 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:25.852 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.852 DEBUG:boto:Params: {} 2022-09-09T22:39:25.853 DEBUG:boto:Token: None 2022-09-09T22:39:25.853 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:25.854 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:eSCpoJJszaFihsZscIStZ7J2ue0= 2022-09-09T22:39:25.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:eSCpoJJszaFihsZscIStZ7J2ue0=', 'Content-Length': '0'} 2022-09-09T22:39:25.858 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b4943430898eadef-00631bc09d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.902 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:25.903 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:25.903 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:25.904 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:25.904 DEBUG:boto:Method: GET 2022-09-09T22:39:25.905 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:25.905 DEBUG:boto:Data: 2022-09-09T22:39:25.906 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.906 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:25.907 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.908 DEBUG:boto:Params: {} 2022-09-09T22:39:25.908 DEBUG:boto:Token: None 2022-09-09T22:39:25.909 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:25.909 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:eSCpoJJszaFihsZscIStZ7J2ue0= 2022-09-09T22:39:25.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:eSCpoJJszaFihsZscIStZ7J2ue0=', 'Content-Length': '0'} 2022-09-09T22:39:25.913 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000312726683df9496c-00631bc09d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.914 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:25.915 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:25.915 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:25.916 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:25.917 DEBUG:boto:Method: GET 2022-09-09T22:39:25.917 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:25.918 DEBUG:boto:Data: 2022-09-09T22:39:25.918 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.919 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:25.919 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.920 DEBUG:boto:Params: {} 2022-09-09T22:39:25.921 DEBUG:boto:Token: None 2022-09-09T22:39:25.921 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/ 2022-09-09T22:39:25.922 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bBVVuiL5JMqXrQOkrp44gZR6k9Q= 2022-09-09T22:39:25.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bBVVuiL5JMqXrQOkrp44gZR6k9Q=', 'Content-Length': '0'} 2022-09-09T22:39:25.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000094d1347212532b6-00631bc09d-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.928 DEBUG:boto:b'jkcgbj-16obj-test-zone31000falseobj-test-zone32022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:25.929 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:25.930 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:25.930 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:25.931 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:25.931 DEBUG:boto:Method: GET 2022-09-09T22:39:25.932 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:25.932 DEBUG:boto:Data: 2022-09-09T22:39:25.933 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.933 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:25.934 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.934 DEBUG:boto:Params: {} 2022-09-09T22:39:25.935 DEBUG:boto:Token: None 2022-09-09T22:39:25.935 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/ 2022-09-09T22:39:25.936 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bBVVuiL5JMqXrQOkrp44gZR6k9Q= 2022-09-09T22:39:25.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bBVVuiL5JMqXrQOkrp44gZR6k9Q=', 'Content-Length': '0'} 2022-09-09T22:39:25.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2029b914a6036ef-00631bc09d-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.943 DEBUG:boto:b'jkcgbj-16obj-test-zone31000falseobj-test-zone32022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:25.944 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:25.944 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:25.945 DEBUG:boto:Method: HEAD 2022-09-09T22:39:25.945 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3 2022-09-09T22:39:25.946 DEBUG:boto:Data: 2022-09-09T22:39:25.946 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.947 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:25.947 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.948 DEBUG:boto:Params: {} 2022-09-09T22:39:25.948 DEBUG:boto:Token: None 2022-09-09T22:39:25.949 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/obj-test-zone3 2022-09-09T22:39:25.950 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R4COyfr8ftxJFtBompyziRECoa4= 2022-09-09T22:39:25.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R4COyfr8ftxJFtBompyziRECoa4=', 'Content-Length': '0'} 2022-09-09T22:39:25.955 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000024d01fd0a4718cd3-00631bc09d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.955 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:25.956 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:25.956 DEBUG:boto:Method: HEAD 2022-09-09T22:39:25.957 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3 2022-09-09T22:39:25.957 DEBUG:boto:Data: 2022-09-09T22:39:25.958 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.958 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:25.959 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.959 DEBUG:boto:Params: {} 2022-09-09T22:39:25.960 DEBUG:boto:Token: None 2022-09-09T22:39:25.960 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/obj-test-zone3 2022-09-09T22:39:25.961 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R4COyfr8ftxJFtBompyziRECoa4= 2022-09-09T22:39:25.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R4COyfr8ftxJFtBompyziRECoa4=', 'Content-Length': '0'} 2022-09-09T22:39:25.965 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000094b2238da3b21377-00631bc09d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:25.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:39:25.967 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:25.967 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:25.968 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:25.968 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:25.969 DEBUG:boto:Method: GET 2022-09-09T22:39:25.969 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:25.970 DEBUG:boto:Data: 2022-09-09T22:39:25.970 DEBUG:boto:Headers: {} 2022-09-09T22:39:25.971 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:25.971 DEBUG:boto:Port: 8000 2022-09-09T22:39:25.972 DEBUG:boto:Params: {} 2022-09-09T22:39:25.973 DEBUG:boto:Token: None 2022-09-09T22:39:25.973 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:25 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:25.974 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:eSCpoJJszaFihsZscIStZ7J2ue0= 2022-09-09T22:39:25.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:eSCpoJJszaFihsZscIStZ7J2ue0=', 'Content-Length': '0'} 2022-09-09T22:39:25.977 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ba061b90e6055a6-00631bc09d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.018 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:26.019 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:26.019 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:26.020 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:26.020 DEBUG:boto:Method: GET 2022-09-09T22:39:26.021 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:26.021 DEBUG:boto:Data: 2022-09-09T22:39:26.022 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.023 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:26.023 DEBUG:boto:Port: 8000 2022-09-09T22:39:26.024 DEBUG:boto:Params: {} 2022-09-09T22:39:26.024 DEBUG:boto:Token: None 2022-09-09T22:39:26.025 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:26.026 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:x2P8XIvSqWUE43QVr0HCgOK+Pbo= 2022-09-09T22:39:26.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:x2P8XIvSqWUE43QVr0HCgOK+Pbo=', 'Content-Length': '0'} 2022-09-09T22:39:26.030 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052243dc2ca15fe38-00631bc09e-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.031 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:26.031 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:26.032 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:26.032 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:26.033 DEBUG:boto:Method: GET 2022-09-09T22:39:26.033 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:26.034 DEBUG:boto:Data: 2022-09-09T22:39:26.035 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.035 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:26.035 DEBUG:boto:Port: 8000 2022-09-09T22:39:26.036 DEBUG:boto:Params: {} 2022-09-09T22:39:26.036 DEBUG:boto:Token: None 2022-09-09T22:39:26.037 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/ 2022-09-09T22:39:26.038 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vkGneYak4Rd/at3fi6+Q+fMa/+I= 2022-09-09T22:39:26.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vkGneYak4Rd/at3fi6+Q+fMa/+I=', 'Content-Length': '0'} 2022-09-09T22:39:26.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001161bd0689fb4adb-00631bc09e-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.044 DEBUG:boto:b'jkcgbj-16obj-test-zone31000falseobj-test-zone32022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:26.045 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:26.046 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:26.046 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:26.047 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:26.047 DEBUG:boto:Method: GET 2022-09-09T22:39:26.048 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:26.048 DEBUG:boto:Data: 2022-09-09T22:39:26.049 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.049 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:26.050 DEBUG:boto:Port: 8000 2022-09-09T22:39:26.051 DEBUG:boto:Params: {} 2022-09-09T22:39:26.051 DEBUG:boto:Token: None 2022-09-09T22:39:26.052 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/ 2022-09-09T22:39:26.052 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vkGneYak4Rd/at3fi6+Q+fMa/+I= 2022-09-09T22:39:26.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vkGneYak4Rd/at3fi6+Q+fMa/+I=', 'Content-Length': '0'} 2022-09-09T22:39:26.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000089081946c14bcdc-00631bc09e-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.059 DEBUG:boto:b'jkcgbj-16obj-test-zone31000falseobj-test-zone32022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:26.060 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-16 zones={test-zone2, test-zone1} 2022-09-09T22:39:26.060 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-16 zones={test-zone3, test-zone1} 2022-09-09T22:39:26.061 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:26.061 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:26.062 DEBUG:boto:Method: HEAD 2022-09-09T22:39:26.062 DEBUG:boto:Path: /jkcgbj-16/ 2022-09-09T22:39:26.063 DEBUG:boto:Data: 2022-09-09T22:39:26.063 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.064 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:26.064 DEBUG:boto:Port: 8001 2022-09-09T22:39:26.065 DEBUG:boto:Params: {} 2022-09-09T22:39:26.065 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-09T22:39:26.066 DEBUG:boto:Token: None 2022-09-09T22:39:26.066 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/ 2022-09-09T22:39:26.067 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:IHjQ0FnmQkfgqXWiteYwyLUIc5k= 2022-09-09T22:39:26.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:IHjQ0FnmQkfgqXWiteYwyLUIc5k=', 'Content-Length': '0'} 2022-09-09T22:39:26.073 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d44f5865792ee81a-00631bc09e-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.074 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:26.074 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:26.075 DEBUG:boto:Method: HEAD 2022-09-09T22:39:26.075 DEBUG:boto:Path: /jkcgbj-16/ 2022-09-09T22:39:26.076 DEBUG:boto:Data: 2022-09-09T22:39:26.076 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.076 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:26.077 DEBUG:boto:Port: 8000 2022-09-09T22:39:26.077 DEBUG:boto:Params: {} 2022-09-09T22:39:26.078 DEBUG:boto:Token: None 2022-09-09T22:39:26.079 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/ 2022-09-09T22:39:26.079 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:IHjQ0FnmQkfgqXWiteYwyLUIc5k= 2022-09-09T22:39:26.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:IHjQ0FnmQkfgqXWiteYwyLUIc5k=', 'Content-Length': '0'} 2022-09-09T22:39:26.084 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004431e02ebc23e57c-00631bc09e-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.085 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:39:26.085 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:26.086 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:26.086 DEBUG:boto:path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:26.087 DEBUG:boto:auth_path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:26.087 DEBUG:boto:Method: GET 2022-09-09T22:39:26.088 DEBUG:boto:Path: /jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:26.088 DEBUG:boto:Data: 2022-09-09T22:39:26.089 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.090 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:26.090 DEBUG:boto:Port: 8001 2022-09-09T22:39:26.091 DEBUG:boto:Params: {} 2022-09-09T22:39:26.091 DEBUG:boto:Token: None 2022-09-09T22:39:26.092 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/?versions 2022-09-09T22:39:26.092 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pfhEKHqFW85c63cEzdm+JQy8UZ0= 2022-09-09T22:39:26.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pfhEKHqFW85c63cEzdm+JQy8UZ0=', 'Content-Length': '0'} 2022-09-09T22:39:26.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c1371cd70dd6920-00631bc09e-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.100 DEBUG:boto:b'jkcgbj-16999falsedummyJ2BEYVUeYOtJqYkdQX3amzkINxiyxtstrue2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1123zMLKlbUHrE3mBcpUCJblKnWDeBXDtrue2022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2MZ.b57oN5HI9SlBmYt.8.2016m6aPMftrue2022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3AesdIyOhh8xrq24fWFi6fMM39PxhXM8true2022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:26.101 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:39:26.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:39:26.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:39:26.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:39:26.103 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:39:26.103 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:26.104 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:26.104 DEBUG:boto:path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:26.105 DEBUG:boto:auth_path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:26.105 DEBUG:boto:Method: GET 2022-09-09T22:39:26.106 DEBUG:boto:Path: /jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:26.107 DEBUG:boto:Data: 2022-09-09T22:39:26.107 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.108 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:26.108 DEBUG:boto:Port: 8000 2022-09-09T22:39:26.109 DEBUG:boto:Params: {} 2022-09-09T22:39:26.109 DEBUG:boto:Token: None 2022-09-09T22:39:26.110 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/?versions 2022-09-09T22:39:26.110 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pfhEKHqFW85c63cEzdm+JQy8UZ0= 2022-09-09T22:39:26.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pfhEKHqFW85c63cEzdm+JQy8UZ0=', 'Content-Length': '0'} 2022-09-09T22:39:26.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067a90fbdf9ee797e-00631bc09e-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.117 DEBUG:boto:b'jkcgbj-16999falsedummyJ2BEYVUeYOtJqYkdQX3amzkINxiyxtstrue2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1123zMLKlbUHrE3mBcpUCJblKnWDeBXDtrue2022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2MZ.b57oN5HI9SlBmYt.8.2016m6aPMftrue2022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3AesdIyOhh8xrq24fWFi6fMM39PxhXM8true2022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:26.118 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:39:26.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:39:26.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:39:26.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:39:26.120 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:26.121 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:26.121 DEBUG:boto:path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:26.122 DEBUG:boto:auth_path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:26.122 DEBUG:boto:Method: GET 2022-09-09T22:39:26.123 DEBUG:boto:Path: /jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:26.123 DEBUG:boto:Data: 2022-09-09T22:39:26.124 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.124 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:26.125 DEBUG:boto:Port: 8001 2022-09-09T22:39:26.125 DEBUG:boto:Params: {} 2022-09-09T22:39:26.126 DEBUG:boto:Token: None 2022-09-09T22:39:26.126 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/?versions 2022-09-09T22:39:26.127 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pfhEKHqFW85c63cEzdm+JQy8UZ0= 2022-09-09T22:39:26.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pfhEKHqFW85c63cEzdm+JQy8UZ0=', 'Content-Length': '0'} 2022-09-09T22:39:26.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a47a311012701486-00631bc09e-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.134 DEBUG:boto:b'jkcgbj-16999falsedummyJ2BEYVUeYOtJqYkdQX3amzkINxiyxtstrue2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1123zMLKlbUHrE3mBcpUCJblKnWDeBXDtrue2022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2MZ.b57oN5HI9SlBmYt.8.2016m6aPMftrue2022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3AesdIyOhh8xrq24fWFi6fMM39PxhXM8true2022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:26.134 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:26.135 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:26.135 DEBUG:boto:path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:26.136 DEBUG:boto:auth_path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:26.136 DEBUG:boto:Method: GET 2022-09-09T22:39:26.137 DEBUG:boto:Path: /jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:26.137 DEBUG:boto:Data: 2022-09-09T22:39:26.138 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.138 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:26.139 DEBUG:boto:Port: 8000 2022-09-09T22:39:26.140 DEBUG:boto:Params: {} 2022-09-09T22:39:26.140 DEBUG:boto:Token: None 2022-09-09T22:39:26.141 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/?versions 2022-09-09T22:39:26.141 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pfhEKHqFW85c63cEzdm+JQy8UZ0= 2022-09-09T22:39:26.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pfhEKHqFW85c63cEzdm+JQy8UZ0=', 'Content-Length': '0'} 2022-09-09T22:39:26.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a92da9bf452baa7-00631bc09e-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.149 DEBUG:boto:b'jkcgbj-16999falsedummyJ2BEYVUeYOtJqYkdQX3amzkINxiyxtstrue2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1123zMLKlbUHrE3mBcpUCJblKnWDeBXDtrue2022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2MZ.b57oN5HI9SlBmYt.8.2016m6aPMftrue2022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3AesdIyOhh8xrq24fWFi6fMM39PxhXM8true2022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:26.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:39:26.150 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:39:26.151 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:39:26.151 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.152 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.152 DEBUG:boto:Method: GET 2022-09-09T22:39:26.153 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.153 DEBUG:boto:Data: 2022-09-09T22:39:26.154 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.154 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:26.155 DEBUG:boto:Port: 8001 2022-09-09T22:39:26.155 DEBUG:boto:Params: {} 2022-09-09T22:39:26.156 DEBUG:boto:Token: None 2022-09-09T22:39:26.157 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.157 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:adYLaycRrj4ha0OY5xXmRXSfOuE= 2022-09-09T22:39:26.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:adYLaycRrj4ha0OY5xXmRXSfOuE=', 'Content-Length': '0'} 2022-09-09T22:39:26.161 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000107d9b6f0ad90da6-00631bc09e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.161 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:39:26.162 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:39:26.163 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.163 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.163 DEBUG:boto:Method: GET 2022-09-09T22:39:26.164 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.164 DEBUG:boto:Data: 2022-09-09T22:39:26.165 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.165 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:26.166 DEBUG:boto:Port: 8000 2022-09-09T22:39:26.167 DEBUG:boto:Params: {} 2022-09-09T22:39:26.167 DEBUG:boto:Token: None 2022-09-09T22:39:26.168 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.168 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:adYLaycRrj4ha0OY5xXmRXSfOuE= 2022-09-09T22:39:26.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:adYLaycRrj4ha0OY5xXmRXSfOuE=', 'Content-Length': '0'} 2022-09-09T22:39:26.172 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000020e6963981877e2a-00631bc09e-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.173 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:39:26.173 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:39:26.174 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.175 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.175 DEBUG:boto:Method: HEAD 2022-09-09T22:39:26.176 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.176 DEBUG:boto:Data: 2022-09-09T22:39:26.176 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.177 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:26.177 DEBUG:boto:Port: 8001 2022-09-09T22:39:26.178 DEBUG:boto:Params: {} 2022-09-09T22:39:26.179 DEBUG:boto:Token: None 2022-09-09T22:39:26.179 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.180 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4Ia7J6L3/NgqsLHAngzyX15a2Bg= 2022-09-09T22:39:26.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4Ia7J6L3/NgqsLHAngzyX15a2Bg=', 'Content-Length': '0'} 2022-09-09T22:39:26.183 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cbcd52827a9602c2-00631bc09e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.184 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:39:26.184 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:39:26.185 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.185 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.186 DEBUG:boto:Method: HEAD 2022-09-09T22:39:26.186 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.187 DEBUG:boto:Data: 2022-09-09T22:39:26.188 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.188 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:26.188 DEBUG:boto:Port: 8000 2022-09-09T22:39:26.189 DEBUG:boto:Params: {} 2022-09-09T22:39:26.190 DEBUG:boto:Token: None 2022-09-09T22:39:26.190 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.191 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4Ia7J6L3/NgqsLHAngzyX15a2Bg= 2022-09-09T22:39:26.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4Ia7J6L3/NgqsLHAngzyX15a2Bg=', 'Content-Length': '0'} 2022-09-09T22:39:26.194 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f19d3a544c4fe3e5-00631bc09e-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:39:26.195 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:39:26.196 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:39:26.196 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.197 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.197 DEBUG:boto:Method: GET 2022-09-09T22:39:26.198 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.198 DEBUG:boto:Data: 2022-09-09T22:39:26.199 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.199 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:26.200 DEBUG:boto:Port: 8001 2022-09-09T22:39:26.200 DEBUG:boto:Params: {} 2022-09-09T22:39:26.201 DEBUG:boto:Token: None 2022-09-09T22:39:26.201 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.202 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:adYLaycRrj4ha0OY5xXmRXSfOuE= 2022-09-09T22:39:26.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:adYLaycRrj4ha0OY5xXmRXSfOuE=', 'Content-Length': '0'} 2022-09-09T22:39:26.205 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c33e604d5b529603-00631bc09e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.206 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:39:26.207 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:39:26.207 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.208 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.208 DEBUG:boto:Method: GET 2022-09-09T22:39:26.209 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.209 DEBUG:boto:Data: 2022-09-09T22:39:26.210 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.210 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:26.211 DEBUG:boto:Port: 8000 2022-09-09T22:39:26.211 DEBUG:boto:Params: {} 2022-09-09T22:39:26.212 DEBUG:boto:Token: None 2022-09-09T22:39:26.212 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.213 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:adYLaycRrj4ha0OY5xXmRXSfOuE= 2022-09-09T22:39:26.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:adYLaycRrj4ha0OY5xXmRXSfOuE=', 'Content-Length': '0'} 2022-09-09T22:39:26.216 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c3a0ad73c5379709-00631bc09e-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.217 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:26.218 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:26.218 DEBUG:boto:path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:39:26.219 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:39:26.219 DEBUG:boto:Method: GET 2022-09-09T22:39:26.220 DEBUG:boto:Path: /jkcgbj-16/?prefix=dummy 2022-09-09T22:39:26.220 DEBUG:boto:Data: 2022-09-09T22:39:26.221 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.221 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:26.222 DEBUG:boto:Port: 8001 2022-09-09T22:39:26.222 DEBUG:boto:Params: {} 2022-09-09T22:39:26.223 DEBUG:boto:Token: None 2022-09-09T22:39:26.224 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/ 2022-09-09T22:39:26.224 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vkGneYak4Rd/at3fi6+Q+fMa/+I= 2022-09-09T22:39:26.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vkGneYak4Rd/at3fi6+Q+fMa/+I=', 'Content-Length': '0'} 2022-09-09T22:39:26.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074968882bb603eb3-00631bc09e-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.274 DEBUG:boto:b'jkcgbj-16dummy1000falsedummy2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:26.275 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:26.276 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:26.276 DEBUG:boto:path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:39:26.277 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:39:26.277 DEBUG:boto:Method: GET 2022-09-09T22:39:26.278 DEBUG:boto:Path: /jkcgbj-16/?prefix=dummy 2022-09-09T22:39:26.278 DEBUG:boto:Data: 2022-09-09T22:39:26.279 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.279 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:26.280 DEBUG:boto:Port: 8000 2022-09-09T22:39:26.280 DEBUG:boto:Params: {} 2022-09-09T22:39:26.281 DEBUG:boto:Token: None 2022-09-09T22:39:26.282 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/ 2022-09-09T22:39:26.282 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vkGneYak4Rd/at3fi6+Q+fMa/+I= 2022-09-09T22:39:26.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vkGneYak4Rd/at3fi6+Q+fMa/+I=', 'Content-Length': '0'} 2022-09-09T22:39:26.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aef69ddb4613b49b-00631bc09e-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.289 DEBUG:boto:b'jkcgbj-16dummy1000falsedummy2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:26.290 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:39:26.290 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:39:26.291 DEBUG:boto:Method: HEAD 2022-09-09T22:39:26.292 DEBUG:boto:Path: /jkcgbj-16/dummy 2022-09-09T22:39:26.292 DEBUG:boto:Data: 2022-09-09T22:39:26.293 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.293 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:26.294 DEBUG:boto:Port: 8001 2022-09-09T22:39:26.294 DEBUG:boto:Params: {} 2022-09-09T22:39:26.295 DEBUG:boto:Token: None 2022-09-09T22:39:26.295 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/dummy 2022-09-09T22:39:26.296 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1nxMl5O9st56NgRuiaJX/SVNVYk= 2022-09-09T22:39:26.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1nxMl5O9st56NgRuiaJX/SVNVYk=', 'Content-Length': '0'} 2022-09-09T22:39:26.300 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000515c2d292fc262e6-00631bc09e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.301 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:39:26.301 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:39:26.302 DEBUG:boto:Method: HEAD 2022-09-09T22:39:26.302 DEBUG:boto:Path: /jkcgbj-16/dummy 2022-09-09T22:39:26.303 DEBUG:boto:Data: 2022-09-09T22:39:26.303 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.304 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:26.304 DEBUG:boto:Port: 8000 2022-09-09T22:39:26.305 DEBUG:boto:Params: {} 2022-09-09T22:39:26.305 DEBUG:boto:Token: None 2022-09-09T22:39:26.306 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/dummy 2022-09-09T22:39:26.306 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1nxMl5O9st56NgRuiaJX/SVNVYk= 2022-09-09T22:39:26.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1nxMl5O9st56NgRuiaJX/SVNVYk=', 'Content-Length': '0'} 2022-09-09T22:39:26.311 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b45e0370bef297dd-00631bc09e-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:39:26.312 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:39:26.312 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:39:26.313 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.313 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.314 DEBUG:boto:Method: GET 2022-09-09T22:39:26.314 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.315 DEBUG:boto:Data: 2022-09-09T22:39:26.315 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.316 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:26.316 DEBUG:boto:Port: 8001 2022-09-09T22:39:26.317 DEBUG:boto:Params: {} 2022-09-09T22:39:26.317 DEBUG:boto:Token: None 2022-09-09T22:39:26.318 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.319 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:adYLaycRrj4ha0OY5xXmRXSfOuE= 2022-09-09T22:39:26.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:adYLaycRrj4ha0OY5xXmRXSfOuE=', 'Content-Length': '0'} 2022-09-09T22:39:26.322 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008971af95e7b8dbe2-00631bc09e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.323 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:39:26.323 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:39:26.324 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.324 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.325 DEBUG:boto:Method: GET 2022-09-09T22:39:26.325 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.326 DEBUG:boto:Data: 2022-09-09T22:39:26.327 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.327 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:26.328 DEBUG:boto:Port: 8000 2022-09-09T22:39:26.328 DEBUG:boto:Params: {} 2022-09-09T22:39:26.329 DEBUG:boto:Token: None 2022-09-09T22:39:26.329 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:26.330 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:adYLaycRrj4ha0OY5xXmRXSfOuE= 2022-09-09T22:39:26.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:adYLaycRrj4ha0OY5xXmRXSfOuE=', 'Content-Length': '0'} 2022-09-09T22:39:26.334 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e81936f1fe6b7829-00631bc09e-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.335 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:26.335 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:26.336 DEBUG:boto:path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:39:26.336 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:39:26.337 DEBUG:boto:Method: GET 2022-09-09T22:39:26.337 DEBUG:boto:Path: /jkcgbj-16/?prefix=dummy 2022-09-09T22:39:26.338 DEBUG:boto:Data: 2022-09-09T22:39:26.338 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.339 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:26.339 DEBUG:boto:Port: 8001 2022-09-09T22:39:26.340 DEBUG:boto:Params: {} 2022-09-09T22:39:26.340 DEBUG:boto:Token: None 2022-09-09T22:39:26.341 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/ 2022-09-09T22:39:26.341 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vkGneYak4Rd/at3fi6+Q+fMa/+I= 2022-09-09T22:39:26.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vkGneYak4Rd/at3fi6+Q+fMa/+I=', 'Content-Length': '0'} 2022-09-09T22:39:26.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b533fd9ffbd2f74e-00631bc09e-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.390 DEBUG:boto:b'jkcgbj-16dummy1000falsedummy2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:26.391 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:26.391 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:26.392 DEBUG:boto:path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:39:26.393 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:39:26.393 DEBUG:boto:Method: GET 2022-09-09T22:39:26.394 DEBUG:boto:Path: /jkcgbj-16/?prefix=dummy 2022-09-09T22:39:26.394 DEBUG:boto:Data: 2022-09-09T22:39:26.395 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.395 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:26.396 DEBUG:boto:Port: 8000 2022-09-09T22:39:26.396 DEBUG:boto:Params: {} 2022-09-09T22:39:26.397 DEBUG:boto:Token: None 2022-09-09T22:39:26.397 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/ 2022-09-09T22:39:26.398 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vkGneYak4Rd/at3fi6+Q+fMa/+I= 2022-09-09T22:39:26.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vkGneYak4Rd/at3fi6+Q+fMa/+I=', 'Content-Length': '0'} 2022-09-09T22:39:26.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b2733056871f42f-00631bc09e-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.406 DEBUG:boto:b'jkcgbj-16dummy1000falsedummy2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:26.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:39:26.407 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:26.408 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:26.408 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.409 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.409 DEBUG:boto:Method: GET 2022-09-09T22:39:26.410 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.410 DEBUG:boto:Data: 2022-09-09T22:39:26.411 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.411 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:26.412 DEBUG:boto:Port: 8001 2022-09-09T22:39:26.412 DEBUG:boto:Params: {} 2022-09-09T22:39:26.413 DEBUG:boto:Token: None 2022-09-09T22:39:26.413 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.414 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FDTFt8nzD0Wk0hgoOKZiw7cKpTA= 2022-09-09T22:39:26.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FDTFt8nzD0Wk0hgoOKZiw7cKpTA=', 'Content-Length': '0'} 2022-09-09T22:39:26.418 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000115514b40c306b75-00631bc09e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.419 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:26.419 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:26.420 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.420 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.421 DEBUG:boto:Method: GET 2022-09-09T22:39:26.421 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.422 DEBUG:boto:Data: 2022-09-09T22:39:26.422 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.423 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:26.423 DEBUG:boto:Port: 8000 2022-09-09T22:39:26.424 DEBUG:boto:Params: {} 2022-09-09T22:39:26.424 DEBUG:boto:Token: None 2022-09-09T22:39:26.425 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.426 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FDTFt8nzD0Wk0hgoOKZiw7cKpTA= 2022-09-09T22:39:26.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FDTFt8nzD0Wk0hgoOKZiw7cKpTA=', 'Content-Length': '0'} 2022-09-09T22:39:26.429 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002900914ea485d98b-00631bc09e-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.470 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:26.471 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:26.471 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.472 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.472 DEBUG:boto:Method: HEAD 2022-09-09T22:39:26.473 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.473 DEBUG:boto:Data: 2022-09-09T22:39:26.474 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.474 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:26.475 DEBUG:boto:Port: 8001 2022-09-09T22:39:26.475 DEBUG:boto:Params: {} 2022-09-09T22:39:26.476 DEBUG:boto:Token: None 2022-09-09T22:39:26.476 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.477 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:y1a/HDJiQLg1sYunvvf4/LVyWDk= 2022-09-09T22:39:26.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:y1a/HDJiQLg1sYunvvf4/LVyWDk=', 'Content-Length': '0'} 2022-09-09T22:39:26.481 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e33375b3021e59ed-00631bc09e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.481 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:26.482 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:26.482 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.483 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.483 DEBUG:boto:Method: HEAD 2022-09-09T22:39:26.484 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.484 DEBUG:boto:Data: 2022-09-09T22:39:26.485 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.485 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:26.486 DEBUG:boto:Port: 8000 2022-09-09T22:39:26.486 DEBUG:boto:Params: {} 2022-09-09T22:39:26.487 DEBUG:boto:Token: None 2022-09-09T22:39:26.488 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.488 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:y1a/HDJiQLg1sYunvvf4/LVyWDk= 2022-09-09T22:39:26.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:y1a/HDJiQLg1sYunvvf4/LVyWDk=', 'Content-Length': '0'} 2022-09-09T22:39:26.491 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a346627236d8383-00631bc09e-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:39:26.493 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:26.493 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:26.494 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.494 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.495 DEBUG:boto:Method: GET 2022-09-09T22:39:26.496 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.496 DEBUG:boto:Data: 2022-09-09T22:39:26.497 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.497 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:26.498 DEBUG:boto:Port: 8001 2022-09-09T22:39:26.498 DEBUG:boto:Params: {} 2022-09-09T22:39:26.499 DEBUG:boto:Token: None 2022-09-09T22:39:26.499 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.500 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FDTFt8nzD0Wk0hgoOKZiw7cKpTA= 2022-09-09T22:39:26.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FDTFt8nzD0Wk0hgoOKZiw7cKpTA=', 'Content-Length': '0'} 2022-09-09T22:39:26.503 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b6e102ee0196a6e-00631bc09e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.505 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:26.505 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:26.506 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.506 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.507 DEBUG:boto:Method: GET 2022-09-09T22:39:26.507 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.508 DEBUG:boto:Data: 2022-09-09T22:39:26.508 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.509 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:26.509 DEBUG:boto:Port: 8000 2022-09-09T22:39:26.510 DEBUG:boto:Params: {} 2022-09-09T22:39:26.511 DEBUG:boto:Token: None 2022-09-09T22:39:26.511 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.512 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FDTFt8nzD0Wk0hgoOKZiw7cKpTA= 2022-09-09T22:39:26.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FDTFt8nzD0Wk0hgoOKZiw7cKpTA=', 'Content-Length': '0'} 2022-09-09T22:39:26.515 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003bb5832f249dd38-00631bc09e-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.516 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:26.517 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:26.517 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:26.518 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:26.518 DEBUG:boto:Method: GET 2022-09-09T22:39:26.519 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:26.519 DEBUG:boto:Data: 2022-09-09T22:39:26.520 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.520 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:26.521 DEBUG:boto:Port: 8001 2022-09-09T22:39:26.521 DEBUG:boto:Params: {} 2022-09-09T22:39:26.522 DEBUG:boto:Token: None 2022-09-09T22:39:26.523 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/ 2022-09-09T22:39:26.523 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vkGneYak4Rd/at3fi6+Q+fMa/+I= 2022-09-09T22:39:26.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vkGneYak4Rd/at3fi6+Q+fMa/+I=', 'Content-Length': '0'} 2022-09-09T22:39:26.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1723e2920d7a5a4-00631bc09e-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.530 DEBUG:boto:b'jkcgbj-16obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:26.531 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:26.531 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:26.532 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:26.532 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:26.533 DEBUG:boto:Method: GET 2022-09-09T22:39:26.533 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:26.534 DEBUG:boto:Data: 2022-09-09T22:39:26.534 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.535 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:26.535 DEBUG:boto:Port: 8000 2022-09-09T22:39:26.536 DEBUG:boto:Params: {} 2022-09-09T22:39:26.537 DEBUG:boto:Token: None 2022-09-09T22:39:26.537 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/ 2022-09-09T22:39:26.538 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vkGneYak4Rd/at3fi6+Q+fMa/+I= 2022-09-09T22:39:26.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vkGneYak4Rd/at3fi6+Q+fMa/+I=', 'Content-Length': '0'} 2022-09-09T22:39:26.544 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fc8b235c33784a3-00631bc09e-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.586 DEBUG:boto:b'jkcgbj-16obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:26.587 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:26.587 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:26.588 DEBUG:boto:Method: HEAD 2022-09-09T22:39:26.588 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1 2022-09-09T22:39:26.589 DEBUG:boto:Data: 2022-09-09T22:39:26.589 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.590 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:26.590 DEBUG:boto:Port: 8001 2022-09-09T22:39:26.591 DEBUG:boto:Params: {} 2022-09-09T22:39:26.591 DEBUG:boto:Token: None 2022-09-09T22:39:26.592 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/obj-test-zone1 2022-09-09T22:39:26.592 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YKHYPAVrlzAzIZq5Yk9T7nB4DvU= 2022-09-09T22:39:26.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YKHYPAVrlzAzIZq5Yk9T7nB4DvU=', 'Content-Length': '0'} 2022-09-09T22:39:26.600 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b2d2ae7b803266d0-00631bc09e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.600 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:26.601 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:26.601 DEBUG:boto:Method: HEAD 2022-09-09T22:39:26.602 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1 2022-09-09T22:39:26.602 DEBUG:boto:Data: 2022-09-09T22:39:26.603 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.603 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:26.604 DEBUG:boto:Port: 8000 2022-09-09T22:39:26.604 DEBUG:boto:Params: {} 2022-09-09T22:39:26.605 DEBUG:boto:Token: None 2022-09-09T22:39:26.605 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/obj-test-zone1 2022-09-09T22:39:26.606 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YKHYPAVrlzAzIZq5Yk9T7nB4DvU= 2022-09-09T22:39:26.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YKHYPAVrlzAzIZq5Yk9T7nB4DvU=', 'Content-Length': '0'} 2022-09-09T22:39:26.610 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091c9f79c7e1909d2-00631bc09e-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:39:26.612 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:26.612 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:26.613 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.613 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.614 DEBUG:boto:Method: GET 2022-09-09T22:39:26.614 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.615 DEBUG:boto:Data: 2022-09-09T22:39:26.615 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.616 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:26.616 DEBUG:boto:Port: 8001 2022-09-09T22:39:26.617 DEBUG:boto:Params: {} 2022-09-09T22:39:26.617 DEBUG:boto:Token: None 2022-09-09T22:39:26.618 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.619 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FDTFt8nzD0Wk0hgoOKZiw7cKpTA= 2022-09-09T22:39:26.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FDTFt8nzD0Wk0hgoOKZiw7cKpTA=', 'Content-Length': '0'} 2022-09-09T22:39:26.622 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000055faa9801f35f19-00631bc09e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.666 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:26.667 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:26.667 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.668 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.668 DEBUG:boto:Method: GET 2022-09-09T22:39:26.669 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.669 DEBUG:boto:Data: 2022-09-09T22:39:26.670 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.670 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:26.671 DEBUG:boto:Port: 8000 2022-09-09T22:39:26.671 DEBUG:boto:Params: {} 2022-09-09T22:39:26.672 DEBUG:boto:Token: None 2022-09-09T22:39:26.672 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:26.673 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FDTFt8nzD0Wk0hgoOKZiw7cKpTA= 2022-09-09T22:39:26.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FDTFt8nzD0Wk0hgoOKZiw7cKpTA=', 'Content-Length': '0'} 2022-09-09T22:39:26.677 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d6fef5049d2c535-00631bc09e-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.677 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:26.678 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:26.679 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:26.679 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:26.679 DEBUG:boto:Method: GET 2022-09-09T22:39:26.680 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:26.680 DEBUG:boto:Data: 2022-09-09T22:39:26.681 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.681 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:26.682 DEBUG:boto:Port: 8001 2022-09-09T22:39:26.682 DEBUG:boto:Params: {} 2022-09-09T22:39:26.683 DEBUG:boto:Token: None 2022-09-09T22:39:26.684 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/ 2022-09-09T22:39:26.684 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vkGneYak4Rd/at3fi6+Q+fMa/+I= 2022-09-09T22:39:26.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vkGneYak4Rd/at3fi6+Q+fMa/+I=', 'Content-Length': '0'} 2022-09-09T22:39:26.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b537bcb1e7b4901-00631bc09e-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.691 DEBUG:boto:b'jkcgbj-16obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:26.692 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:26.693 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:26.694 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:26.694 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:26.695 DEBUG:boto:Method: GET 2022-09-09T22:39:26.695 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:26.696 DEBUG:boto:Data: 2022-09-09T22:39:26.696 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.697 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:26.697 DEBUG:boto:Port: 8000 2022-09-09T22:39:26.698 DEBUG:boto:Params: {} 2022-09-09T22:39:26.698 DEBUG:boto:Token: None 2022-09-09T22:39:26.699 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/ 2022-09-09T22:39:26.700 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vkGneYak4Rd/at3fi6+Q+fMa/+I= 2022-09-09T22:39:26.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vkGneYak4Rd/at3fi6+Q+fMa/+I=', 'Content-Length': '0'} 2022-09-09T22:39:26.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e30ab3081adaae9-00631bc09e-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.707 DEBUG:boto:b'jkcgbj-16obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:26.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:39:26.708 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:26.708 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:26.709 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.710 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.710 DEBUG:boto:Method: GET 2022-09-09T22:39:26.711 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.711 DEBUG:boto:Data: 2022-09-09T22:39:26.712 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.713 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:26.714 DEBUG:boto:Port: 8001 2022-09-09T22:39:26.715 DEBUG:boto:Params: {} 2022-09-09T22:39:26.716 DEBUG:boto:Token: None 2022-09-09T22:39:26.717 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.718 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KYfnzjcnBERanWn0sDPLUlJVdL4= 2022-09-09T22:39:26.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KYfnzjcnBERanWn0sDPLUlJVdL4=', 'Content-Length': '0'} 2022-09-09T22:39:26.723 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b88a536120a9c92a-00631bc09e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.724 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:26.724 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:26.725 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.725 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.726 DEBUG:boto:Method: GET 2022-09-09T22:39:26.726 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.727 DEBUG:boto:Data: 2022-09-09T22:39:26.727 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.728 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:26.728 DEBUG:boto:Port: 8000 2022-09-09T22:39:26.729 DEBUG:boto:Params: {} 2022-09-09T22:39:26.729 DEBUG:boto:Token: None 2022-09-09T22:39:26.730 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.731 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KYfnzjcnBERanWn0sDPLUlJVdL4= 2022-09-09T22:39:26.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KYfnzjcnBERanWn0sDPLUlJVdL4=', 'Content-Length': '0'} 2022-09-09T22:39:26.735 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4231861e7a1a63b-00631bc09e-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.736 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:26.737 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:26.737 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.737 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.738 DEBUG:boto:Method: HEAD 2022-09-09T22:39:26.738 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.739 DEBUG:boto:Data: 2022-09-09T22:39:26.739 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.740 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:26.741 DEBUG:boto:Port: 8001 2022-09-09T22:39:26.742 DEBUG:boto:Params: {} 2022-09-09T22:39:26.743 DEBUG:boto:Token: None 2022-09-09T22:39:26.743 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.744 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:f5i0a+BjwOtI7nuCiT1CiKyzHdU= 2022-09-09T22:39:26.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:f5i0a+BjwOtI7nuCiT1CiKyzHdU=', 'Content-Length': '0'} 2022-09-09T22:39:26.747 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bcb427b1bc665e34-00631bc09e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.748 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:26.748 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:26.749 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.750 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.750 DEBUG:boto:Method: HEAD 2022-09-09T22:39:26.751 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.751 DEBUG:boto:Data: 2022-09-09T22:39:26.751 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.752 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:26.752 DEBUG:boto:Port: 8000 2022-09-09T22:39:26.753 DEBUG:boto:Params: {} 2022-09-09T22:39:26.754 DEBUG:boto:Token: None 2022-09-09T22:39:26.754 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.755 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:f5i0a+BjwOtI7nuCiT1CiKyzHdU= 2022-09-09T22:39:26.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:f5i0a+BjwOtI7nuCiT1CiKyzHdU=', 'Content-Length': '0'} 2022-09-09T22:39:26.758 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4e93432412206eb-00631bc09e-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:39:26.759 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:26.760 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:26.760 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.761 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.761 DEBUG:boto:Method: GET 2022-09-09T22:39:26.762 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.762 DEBUG:boto:Data: 2022-09-09T22:39:26.763 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.763 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:26.764 DEBUG:boto:Port: 8001 2022-09-09T22:39:26.764 DEBUG:boto:Params: {} 2022-09-09T22:39:26.765 DEBUG:boto:Token: None 2022-09-09T22:39:26.765 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.766 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KYfnzjcnBERanWn0sDPLUlJVdL4= 2022-09-09T22:39:26.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KYfnzjcnBERanWn0sDPLUlJVdL4=', 'Content-Length': '0'} 2022-09-09T22:39:26.769 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000522ba02893926016-00631bc09e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.810 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:26.811 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:26.811 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.812 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.812 DEBUG:boto:Method: GET 2022-09-09T22:39:26.813 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.813 DEBUG:boto:Data: 2022-09-09T22:39:26.814 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.814 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:26.815 DEBUG:boto:Port: 8000 2022-09-09T22:39:26.815 DEBUG:boto:Params: {} 2022-09-09T22:39:26.816 DEBUG:boto:Token: None 2022-09-09T22:39:26.816 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.817 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KYfnzjcnBERanWn0sDPLUlJVdL4= 2022-09-09T22:39:26.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KYfnzjcnBERanWn0sDPLUlJVdL4=', 'Content-Length': '0'} 2022-09-09T22:39:26.821 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae3069a3a00ff39f-00631bc09e-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.822 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:26.822 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:26.823 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:26.823 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:26.824 DEBUG:boto:Method: GET 2022-09-09T22:39:26.824 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:26.825 DEBUG:boto:Data: 2022-09-09T22:39:26.825 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.826 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:26.826 DEBUG:boto:Port: 8001 2022-09-09T22:39:26.827 DEBUG:boto:Params: {} 2022-09-09T22:39:26.828 DEBUG:boto:Token: None 2022-09-09T22:39:26.828 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/ 2022-09-09T22:39:26.829 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vkGneYak4Rd/at3fi6+Q+fMa/+I= 2022-09-09T22:39:26.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vkGneYak4Rd/at3fi6+Q+fMa/+I=', 'Content-Length': '0'} 2022-09-09T22:39:26.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077423b5bdd86ff2a-00631bc09e-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.837 DEBUG:boto:b'jkcgbj-16obj-test-zone21000falseobj-test-zone22022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:26.837 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:26.838 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:26.839 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:26.839 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:26.840 DEBUG:boto:Method: GET 2022-09-09T22:39:26.840 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:26.841 DEBUG:boto:Data: 2022-09-09T22:39:26.841 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.842 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:26.842 DEBUG:boto:Port: 8000 2022-09-09T22:39:26.843 DEBUG:boto:Params: {} 2022-09-09T22:39:26.843 DEBUG:boto:Token: None 2022-09-09T22:39:26.844 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/ 2022-09-09T22:39:26.844 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vkGneYak4Rd/at3fi6+Q+fMa/+I= 2022-09-09T22:39:26.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vkGneYak4Rd/at3fi6+Q+fMa/+I=', 'Content-Length': '0'} 2022-09-09T22:39:26.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b72b71382d83387-00631bc09e-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.851 DEBUG:boto:b'jkcgbj-16obj-test-zone21000falseobj-test-zone22022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:26.851 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:26.852 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:26.852 DEBUG:boto:Method: HEAD 2022-09-09T22:39:26.853 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2 2022-09-09T22:39:26.854 DEBUG:boto:Data: 2022-09-09T22:39:26.854 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.855 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:26.855 DEBUG:boto:Port: 8001 2022-09-09T22:39:26.856 DEBUG:boto:Params: {} 2022-09-09T22:39:26.856 DEBUG:boto:Token: None 2022-09-09T22:39:26.857 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/obj-test-zone2 2022-09-09T22:39:26.858 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:D2rpHo3lzVW9QUpnUch7UyyNfW4= 2022-09-09T22:39:26.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:D2rpHo3lzVW9QUpnUch7UyyNfW4=', 'Content-Length': '0'} 2022-09-09T22:39:26.862 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7d22c492f6d337d-00631bc09e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.862 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:26.863 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:26.863 DEBUG:boto:Method: HEAD 2022-09-09T22:39:26.864 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2 2022-09-09T22:39:26.864 DEBUG:boto:Data: 2022-09-09T22:39:26.865 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.865 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:26.866 DEBUG:boto:Port: 8000 2022-09-09T22:39:26.866 DEBUG:boto:Params: {} 2022-09-09T22:39:26.867 DEBUG:boto:Token: None 2022-09-09T22:39:26.867 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/obj-test-zone2 2022-09-09T22:39:26.868 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:D2rpHo3lzVW9QUpnUch7UyyNfW4= 2022-09-09T22:39:26.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:D2rpHo3lzVW9QUpnUch7UyyNfW4=', 'Content-Length': '0'} 2022-09-09T22:39:26.872 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004e8364c2ee8c407f-00631bc09e-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:39:26.873 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:26.874 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:26.874 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.875 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.875 DEBUG:boto:Method: GET 2022-09-09T22:39:26.876 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.876 DEBUG:boto:Data: 2022-09-09T22:39:26.877 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.877 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:26.878 DEBUG:boto:Port: 8001 2022-09-09T22:39:26.878 DEBUG:boto:Params: {} 2022-09-09T22:39:26.879 DEBUG:boto:Token: None 2022-09-09T22:39:26.879 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.880 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KYfnzjcnBERanWn0sDPLUlJVdL4= 2022-09-09T22:39:26.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KYfnzjcnBERanWn0sDPLUlJVdL4=', 'Content-Length': '0'} 2022-09-09T22:39:26.884 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000021e9e3eefd3fd62b-00631bc09e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.885 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:26.885 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:26.886 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.886 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.887 DEBUG:boto:Method: GET 2022-09-09T22:39:26.887 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.888 DEBUG:boto:Data: 2022-09-09T22:39:26.888 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.889 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:26.889 DEBUG:boto:Port: 8000 2022-09-09T22:39:26.890 DEBUG:boto:Params: {} 2022-09-09T22:39:26.890 DEBUG:boto:Token: None 2022-09-09T22:39:26.891 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:26.892 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KYfnzjcnBERanWn0sDPLUlJVdL4= 2022-09-09T22:39:26.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KYfnzjcnBERanWn0sDPLUlJVdL4=', 'Content-Length': '0'} 2022-09-09T22:39:26.896 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae59ebf0e9bd46b1-00631bc09e-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.896 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:26.897 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:26.897 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:26.898 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:26.898 DEBUG:boto:Method: GET 2022-09-09T22:39:26.899 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:26.899 DEBUG:boto:Data: 2022-09-09T22:39:26.900 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.900 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:26.901 DEBUG:boto:Port: 8001 2022-09-09T22:39:26.901 DEBUG:boto:Params: {} 2022-09-09T22:39:26.902 DEBUG:boto:Token: None 2022-09-09T22:39:26.903 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/ 2022-09-09T22:39:26.903 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vkGneYak4Rd/at3fi6+Q+fMa/+I= 2022-09-09T22:39:26.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vkGneYak4Rd/at3fi6+Q+fMa/+I=', 'Content-Length': '0'} 2022-09-09T22:39:26.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000515f9374ee16d8e2-00631bc09e-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.950 DEBUG:boto:b'jkcgbj-16obj-test-zone21000falseobj-test-zone22022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:26.951 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:26.951 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:26.952 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:26.952 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:26.953 DEBUG:boto:Method: GET 2022-09-09T22:39:26.954 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:26.954 DEBUG:boto:Data: 2022-09-09T22:39:26.955 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.955 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:26.956 DEBUG:boto:Port: 8000 2022-09-09T22:39:26.956 DEBUG:boto:Params: {} 2022-09-09T22:39:26.957 DEBUG:boto:Token: None 2022-09-09T22:39:26.957 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/ 2022-09-09T22:39:26.958 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vkGneYak4Rd/at3fi6+Q+fMa/+I= 2022-09-09T22:39:26.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vkGneYak4Rd/at3fi6+Q+fMa/+I=', 'Content-Length': '0'} 2022-09-09T22:39:26.965 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a818d72f4d72644-00631bc09e-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.966 DEBUG:boto:b'jkcgbj-16obj-test-zone21000falseobj-test-zone22022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:26.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:39:26.968 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:26.969 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:26.970 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:26.971 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:26.972 DEBUG:boto:Method: GET 2022-09-09T22:39:26.973 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:26.974 DEBUG:boto:Data: 2022-09-09T22:39:26.975 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.976 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:26.977 DEBUG:boto:Port: 8001 2022-09-09T22:39:26.978 DEBUG:boto:Params: {} 2022-09-09T22:39:26.979 DEBUG:boto:Token: None 2022-09-09T22:39:26.980 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:26.981 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:x2P8XIvSqWUE43QVr0HCgOK+Pbo= 2022-09-09T22:39:26.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:x2P8XIvSqWUE43QVr0HCgOK+Pbo=', 'Content-Length': '0'} 2022-09-09T22:39:26.986 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b18473c417af79e-00631bc09e-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:26.987 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:26.988 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:26.989 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:26.990 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:26.991 DEBUG:boto:Method: GET 2022-09-09T22:39:26.992 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:26.993 DEBUG:boto:Data: 2022-09-09T22:39:26.994 DEBUG:boto:Headers: {} 2022-09-09T22:39:26.994 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:26.995 DEBUG:boto:Port: 8000 2022-09-09T22:39:26.995 DEBUG:boto:Params: {} 2022-09-09T22:39:26.996 DEBUG:boto:Token: None 2022-09-09T22:39:26.997 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:26 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:26.998 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:x2P8XIvSqWUE43QVr0HCgOK+Pbo= 2022-09-09T22:39:26.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:x2P8XIvSqWUE43QVr0HCgOK+Pbo=', 'Content-Length': '0'} 2022-09-09T22:39:27.001 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ac46370dba840d4-00631bc09e-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.046 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:27.047 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:27.047 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:27.048 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:27.048 DEBUG:boto:Method: HEAD 2022-09-09T22:39:27.049 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:27.049 DEBUG:boto:Data: 2022-09-09T22:39:27.050 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.050 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:27.051 DEBUG:boto:Port: 8001 2022-09-09T22:39:27.051 DEBUG:boto:Params: {} 2022-09-09T22:39:27.052 DEBUG:boto:Token: None 2022-09-09T22:39:27.052 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:27.053 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1AZYpc/HHdMKbFBEuFsZ3mFVug8= 2022-09-09T22:39:27.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1AZYpc/HHdMKbFBEuFsZ3mFVug8=', 'Content-Length': '0'} 2022-09-09T22:39:27.057 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f5a9a58ed2c8a89-00631bc09f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.057 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:27.058 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:27.058 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:27.059 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:27.059 DEBUG:boto:Method: HEAD 2022-09-09T22:39:27.060 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:27.060 DEBUG:boto:Data: 2022-09-09T22:39:27.061 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.061 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:27.062 DEBUG:boto:Port: 8000 2022-09-09T22:39:27.063 DEBUG:boto:Params: {} 2022-09-09T22:39:27.063 DEBUG:boto:Token: None 2022-09-09T22:39:27.064 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:27.064 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1AZYpc/HHdMKbFBEuFsZ3mFVug8= 2022-09-09T22:39:27.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1AZYpc/HHdMKbFBEuFsZ3mFVug8=', 'Content-Length': '0'} 2022-09-09T22:39:27.068 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000989cdd61e2a38c69-00631bc09f-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:39:27.069 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:27.069 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:27.070 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:27.070 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:27.071 DEBUG:boto:Method: GET 2022-09-09T22:39:27.072 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:27.072 DEBUG:boto:Data: 2022-09-09T22:39:27.073 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.073 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:27.074 DEBUG:boto:Port: 8001 2022-09-09T22:39:27.074 DEBUG:boto:Params: {} 2022-09-09T22:39:27.075 DEBUG:boto:Token: None 2022-09-09T22:39:27.075 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:27.076 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:b5OqEbnzFt4+lUGZQwhjGzDJjjQ= 2022-09-09T22:39:27.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:b5OqEbnzFt4+lUGZQwhjGzDJjjQ=', 'Content-Length': '0'} 2022-09-09T22:39:27.080 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fbbc8883be5ed682-00631bc09f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.080 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:27.081 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:27.081 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:27.082 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:27.083 DEBUG:boto:Method: GET 2022-09-09T22:39:27.083 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:27.084 DEBUG:boto:Data: 2022-09-09T22:39:27.084 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.085 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:27.085 DEBUG:boto:Port: 8000 2022-09-09T22:39:27.086 DEBUG:boto:Params: {} 2022-09-09T22:39:27.086 DEBUG:boto:Token: None 2022-09-09T22:39:27.087 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:27.088 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:b5OqEbnzFt4+lUGZQwhjGzDJjjQ= 2022-09-09T22:39:27.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:b5OqEbnzFt4+lUGZQwhjGzDJjjQ=', 'Content-Length': '0'} 2022-09-09T22:39:27.091 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000163fbba60b728874-00631bc09f-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.092 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:27.093 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:27.093 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:27.094 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:27.094 DEBUG:boto:Method: GET 2022-09-09T22:39:27.095 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:27.095 DEBUG:boto:Data: 2022-09-09T22:39:27.096 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.096 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:27.097 DEBUG:boto:Port: 8001 2022-09-09T22:39:27.097 DEBUG:boto:Params: {} 2022-09-09T22:39:27.098 DEBUG:boto:Token: None 2022-09-09T22:39:27.099 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/ 2022-09-09T22:39:27.099 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dTf3as3nrlDALuzN1eyu7D8Nb/w= 2022-09-09T22:39:27.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dTf3as3nrlDALuzN1eyu7D8Nb/w=', 'Content-Length': '0'} 2022-09-09T22:39:27.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca8e21e2c870133a-00631bc09f-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.106 DEBUG:boto:b'jkcgbj-16obj-test-zone31000falseobj-test-zone32022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:27.107 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:27.107 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:27.108 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:27.108 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:27.109 DEBUG:boto:Method: GET 2022-09-09T22:39:27.109 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:27.110 DEBUG:boto:Data: 2022-09-09T22:39:27.110 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.111 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:27.111 DEBUG:boto:Port: 8000 2022-09-09T22:39:27.112 DEBUG:boto:Params: {} 2022-09-09T22:39:27.112 DEBUG:boto:Token: None 2022-09-09T22:39:27.113 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/ 2022-09-09T22:39:27.114 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dTf3as3nrlDALuzN1eyu7D8Nb/w= 2022-09-09T22:39:27.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dTf3as3nrlDALuzN1eyu7D8Nb/w=', 'Content-Length': '0'} 2022-09-09T22:39:27.120 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5abfd0566336991-00631bc09f-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.162 DEBUG:boto:b'jkcgbj-16obj-test-zone31000falseobj-test-zone32022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:27.163 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:27.163 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:27.164 DEBUG:boto:Method: HEAD 2022-09-09T22:39:27.164 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3 2022-09-09T22:39:27.165 DEBUG:boto:Data: 2022-09-09T22:39:27.165 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.166 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:27.166 DEBUG:boto:Port: 8001 2022-09-09T22:39:27.167 DEBUG:boto:Params: {} 2022-09-09T22:39:27.168 DEBUG:boto:Token: None 2022-09-09T22:39:27.168 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/obj-test-zone3 2022-09-09T22:39:27.169 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:IlydbG6Jju1JhyL9lymj4v1gHxA= 2022-09-09T22:39:27.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:IlydbG6Jju1JhyL9lymj4v1gHxA=', 'Content-Length': '0'} 2022-09-09T22:39:27.173 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ea5e5994073528d-00631bc09f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.174 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:27.175 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:27.175 DEBUG:boto:Method: HEAD 2022-09-09T22:39:27.176 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3 2022-09-09T22:39:27.176 DEBUG:boto:Data: 2022-09-09T22:39:27.177 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.177 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:27.178 DEBUG:boto:Port: 8000 2022-09-09T22:39:27.178 DEBUG:boto:Params: {} 2022-09-09T22:39:27.179 DEBUG:boto:Token: None 2022-09-09T22:39:27.179 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/obj-test-zone3 2022-09-09T22:39:27.180 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:IlydbG6Jju1JhyL9lymj4v1gHxA= 2022-09-09T22:39:27.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:IlydbG6Jju1JhyL9lymj4v1gHxA=', 'Content-Length': '0'} 2022-09-09T22:39:27.184 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055cdd75c22b5b333-00631bc09f-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:39:27.186 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:27.186 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:27.187 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:27.187 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:27.188 DEBUG:boto:Method: GET 2022-09-09T22:39:27.188 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:27.189 DEBUG:boto:Data: 2022-09-09T22:39:27.189 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.190 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:27.190 DEBUG:boto:Port: 8001 2022-09-09T22:39:27.191 DEBUG:boto:Params: {} 2022-09-09T22:39:27.191 DEBUG:boto:Token: None 2022-09-09T22:39:27.192 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:27.193 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:b5OqEbnzFt4+lUGZQwhjGzDJjjQ= 2022-09-09T22:39:27.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:b5OqEbnzFt4+lUGZQwhjGzDJjjQ=', 'Content-Length': '0'} 2022-09-09T22:39:27.197 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097ba7f254afb5884-00631bc09f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.197 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:27.198 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:27.199 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:27.199 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:27.200 DEBUG:boto:Method: GET 2022-09-09T22:39:27.200 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:27.201 DEBUG:boto:Data: 2022-09-09T22:39:27.201 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.202 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:27.202 DEBUG:boto:Port: 8000 2022-09-09T22:39:27.203 DEBUG:boto:Params: {} 2022-09-09T22:39:27.203 DEBUG:boto:Token: None 2022-09-09T22:39:27.204 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:27.204 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:b5OqEbnzFt4+lUGZQwhjGzDJjjQ= 2022-09-09T22:39:27.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:b5OqEbnzFt4+lUGZQwhjGzDJjjQ=', 'Content-Length': '0'} 2022-09-09T22:39:27.208 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000839bd991cb64ab18-00631bc09f-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.209 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:27.210 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:27.210 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:27.211 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:27.211 DEBUG:boto:Method: GET 2022-09-09T22:39:27.212 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:27.212 DEBUG:boto:Data: 2022-09-09T22:39:27.213 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.213 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:27.214 DEBUG:boto:Port: 8001 2022-09-09T22:39:27.214 DEBUG:boto:Params: {} 2022-09-09T22:39:27.215 DEBUG:boto:Token: None 2022-09-09T22:39:27.215 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/ 2022-09-09T22:39:27.216 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dTf3as3nrlDALuzN1eyu7D8Nb/w= 2022-09-09T22:39:27.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dTf3as3nrlDALuzN1eyu7D8Nb/w=', 'Content-Length': '0'} 2022-09-09T22:39:27.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd041d75be46b52e-00631bc09f-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.224 DEBUG:boto:b'jkcgbj-16obj-test-zone31000falseobj-test-zone32022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:27.224 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:27.225 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:27.225 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:27.226 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:27.227 DEBUG:boto:Method: GET 2022-09-09T22:39:27.227 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:27.228 DEBUG:boto:Data: 2022-09-09T22:39:27.228 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.229 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:27.229 DEBUG:boto:Port: 8000 2022-09-09T22:39:27.230 DEBUG:boto:Params: {} 2022-09-09T22:39:27.230 DEBUG:boto:Token: None 2022-09-09T22:39:27.231 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/ 2022-09-09T22:39:27.231 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dTf3as3nrlDALuzN1eyu7D8Nb/w= 2022-09-09T22:39:27.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dTf3as3nrlDALuzN1eyu7D8Nb/w=', 'Content-Length': '0'} 2022-09-09T22:39:27.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087d802804dba8fe1-00631bc09f-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.238 DEBUG:boto:b'jkcgbj-16obj-test-zone31000falseobj-test-zone32022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:27.239 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-16 zones={test-zone3, test-zone1} 2022-09-09T22:39:27.240 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-16 zones={test-zone3, test-zone2} 2022-09-09T22:39:27.240 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:27.241 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:27.241 DEBUG:boto:Method: HEAD 2022-09-09T22:39:27.242 DEBUG:boto:Path: /jkcgbj-16/ 2022-09-09T22:39:27.242 DEBUG:boto:Data: 2022-09-09T22:39:27.243 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.243 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:27.244 DEBUG:boto:Port: 8001 2022-09-09T22:39:27.244 DEBUG:boto:Params: {} 2022-09-09T22:39:27.245 DEBUG:boto:Token: None 2022-09-09T22:39:27.245 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/ 2022-09-09T22:39:27.246 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:zFXpI9WEYxvKIt8RD5KSsCne444= 2022-09-09T22:39:27.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:zFXpI9WEYxvKIt8RD5KSsCne444=', 'Content-Length': '0'} 2022-09-09T22:39:27.251 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000683a85ba935aca90-00631bc09f-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.252 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:27.252 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:27.253 DEBUG:boto:Method: HEAD 2022-09-09T22:39:27.253 DEBUG:boto:Path: /jkcgbj-16/ 2022-09-09T22:39:27.254 DEBUG:boto:Data: 2022-09-09T22:39:27.254 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.255 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:27.255 DEBUG:boto:Port: 8000 2022-09-09T22:39:27.256 DEBUG:boto:Params: {} 2022-09-09T22:39:27.256 DEBUG:boto:Token: None 2022-09-09T22:39:27.257 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/ 2022-09-09T22:39:27.257 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:zFXpI9WEYxvKIt8RD5KSsCne444= 2022-09-09T22:39:27.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:zFXpI9WEYxvKIt8RD5KSsCne444=', 'Content-Length': '0'} 2022-09-09T22:39:27.263 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d6e08ded858cfc0c-00631bc09f-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.264 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:39:27.264 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:27.265 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:27.265 DEBUG:boto:path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:27.266 DEBUG:boto:auth_path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:27.266 DEBUG:boto:Method: GET 2022-09-09T22:39:27.267 DEBUG:boto:Path: /jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:27.267 DEBUG:boto:Data: 2022-09-09T22:39:27.268 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.268 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:27.269 DEBUG:boto:Port: 8001 2022-09-09T22:39:27.269 DEBUG:boto:Params: {} 2022-09-09T22:39:27.270 DEBUG:boto:Token: None 2022-09-09T22:39:27.271 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/?versions 2022-09-09T22:39:27.271 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:x/xfIAtUh36vAseq07MIbgghqvQ= 2022-09-09T22:39:27.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:x/xfIAtUh36vAseq07MIbgghqvQ=', 'Content-Length': '0'} 2022-09-09T22:39:27.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000756036f329effbe1-00631bc09f-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.278 DEBUG:boto:b'jkcgbj-16999falsedummyJ2BEYVUeYOtJqYkdQX3amzkINxiyxtstrue2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1123zMLKlbUHrE3mBcpUCJblKnWDeBXDtrue2022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2MZ.b57oN5HI9SlBmYt.8.2016m6aPMftrue2022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3AesdIyOhh8xrq24fWFi6fMM39PxhXM8true2022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:27.279 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:39:27.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:39:27.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:39:27.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:39:27.281 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:39:27.281 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:27.282 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:27.282 DEBUG:boto:path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:27.283 DEBUG:boto:auth_path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:27.284 DEBUG:boto:Method: GET 2022-09-09T22:39:27.284 DEBUG:boto:Path: /jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:27.285 DEBUG:boto:Data: 2022-09-09T22:39:27.285 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.286 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:27.286 DEBUG:boto:Port: 8000 2022-09-09T22:39:27.287 DEBUG:boto:Params: {} 2022-09-09T22:39:27.287 DEBUG:boto:Token: None 2022-09-09T22:39:27.288 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/?versions 2022-09-09T22:39:27.288 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:x/xfIAtUh36vAseq07MIbgghqvQ= 2022-09-09T22:39:27.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:x/xfIAtUh36vAseq07MIbgghqvQ=', 'Content-Length': '0'} 2022-09-09T22:39:27.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000581586b47a28ae9f-00631bc09f-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.296 DEBUG:boto:b'jkcgbj-16999falsedummyJ2BEYVUeYOtJqYkdQX3amzkINxiyxtstrue2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1123zMLKlbUHrE3mBcpUCJblKnWDeBXDtrue2022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2MZ.b57oN5HI9SlBmYt.8.2016m6aPMftrue2022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3AesdIyOhh8xrq24fWFi6fMM39PxhXM8true2022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:27.297 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:39:27.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:39:27.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:39:27.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:39:27.299 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:27.299 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:27.300 DEBUG:boto:path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:27.300 DEBUG:boto:auth_path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:27.301 DEBUG:boto:Method: GET 2022-09-09T22:39:27.301 DEBUG:boto:Path: /jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:27.302 DEBUG:boto:Data: 2022-09-09T22:39:27.302 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.303 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:27.303 DEBUG:boto:Port: 8001 2022-09-09T22:39:27.304 DEBUG:boto:Params: {} 2022-09-09T22:39:27.304 DEBUG:boto:Token: None 2022-09-09T22:39:27.305 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/?versions 2022-09-09T22:39:27.306 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:x/xfIAtUh36vAseq07MIbgghqvQ= 2022-09-09T22:39:27.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:x/xfIAtUh36vAseq07MIbgghqvQ=', 'Content-Length': '0'} 2022-09-09T22:39:27.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cffc322fac98360e-00631bc09f-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.313 DEBUG:boto:b'jkcgbj-16999falsedummyJ2BEYVUeYOtJqYkdQX3amzkINxiyxtstrue2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1123zMLKlbUHrE3mBcpUCJblKnWDeBXDtrue2022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2MZ.b57oN5HI9SlBmYt.8.2016m6aPMftrue2022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3AesdIyOhh8xrq24fWFi6fMM39PxhXM8true2022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:27.314 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:27.314 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:27.315 DEBUG:boto:path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:27.315 DEBUG:boto:auth_path=/jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:27.316 DEBUG:boto:Method: GET 2022-09-09T22:39:27.316 DEBUG:boto:Path: /jkcgbj-16/?versions&max-keys=999 2022-09-09T22:39:27.317 DEBUG:boto:Data: 2022-09-09T22:39:27.317 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.318 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:27.318 DEBUG:boto:Port: 8000 2022-09-09T22:39:27.319 DEBUG:boto:Params: {} 2022-09-09T22:39:27.319 DEBUG:boto:Token: None 2022-09-09T22:39:27.320 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/?versions 2022-09-09T22:39:27.321 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:x/xfIAtUh36vAseq07MIbgghqvQ= 2022-09-09T22:39:27.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:x/xfIAtUh36vAseq07MIbgghqvQ=', 'Content-Length': '0'} 2022-09-09T22:39:27.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099cb210c66a35c68-00631bc09f-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.328 DEBUG:boto:b'jkcgbj-16999falsedummyJ2BEYVUeYOtJqYkdQX3amzkINxiyxtstrue2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1123zMLKlbUHrE3mBcpUCJblKnWDeBXDtrue2022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2MZ.b57oN5HI9SlBmYt.8.2016m6aPMftrue2022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3AesdIyOhh8xrq24fWFi6fMM39PxhXM8true2022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:27.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:39:27.330 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:39:27.330 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:39:27.331 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.332 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.332 DEBUG:boto:Method: GET 2022-09-09T22:39:27.333 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.333 DEBUG:boto:Data: 2022-09-09T22:39:27.334 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.334 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:27.335 DEBUG:boto:Port: 8001 2022-09-09T22:39:27.335 DEBUG:boto:Params: {} 2022-09-09T22:39:27.336 DEBUG:boto:Token: None 2022-09-09T22:39:27.336 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.337 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:16Y85cS5a27HXQs2yOC2LZ4Hmlo= 2022-09-09T22:39:27.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:16Y85cS5a27HXQs2yOC2LZ4Hmlo=', 'Content-Length': '0'} 2022-09-09T22:39:27.341 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000fea76701461de59-00631bc09f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.341 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:39:27.342 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:39:27.342 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.343 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.343 DEBUG:boto:Method: GET 2022-09-09T22:39:27.344 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.345 DEBUG:boto:Data: 2022-09-09T22:39:27.345 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.346 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:27.346 DEBUG:boto:Port: 8000 2022-09-09T22:39:27.347 DEBUG:boto:Params: {} 2022-09-09T22:39:27.347 DEBUG:boto:Token: None 2022-09-09T22:39:27.348 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.348 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:16Y85cS5a27HXQs2yOC2LZ4Hmlo= 2022-09-09T22:39:27.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:16Y85cS5a27HXQs2yOC2LZ4Hmlo=', 'Content-Length': '0'} 2022-09-09T22:39:27.352 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ad517339c7847a16-00631bc09f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.353 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:39:27.354 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:39:27.354 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.355 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.355 DEBUG:boto:Method: HEAD 2022-09-09T22:39:27.356 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.356 DEBUG:boto:Data: 2022-09-09T22:39:27.357 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.357 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:27.358 DEBUG:boto:Port: 8001 2022-09-09T22:39:27.358 DEBUG:boto:Params: {} 2022-09-09T22:39:27.359 DEBUG:boto:Token: None 2022-09-09T22:39:27.359 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.360 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Yd6dO9mWVeHPwspug2EPq0IM83M= 2022-09-09T22:39:27.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Yd6dO9mWVeHPwspug2EPq0IM83M=', 'Content-Length': '0'} 2022-09-09T22:39:27.363 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000071a8c26fb648ff8e-00631bc09f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.364 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:39:27.365 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:39:27.365 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.366 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.366 DEBUG:boto:Method: HEAD 2022-09-09T22:39:27.367 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.367 DEBUG:boto:Data: 2022-09-09T22:39:27.368 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.368 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:27.369 DEBUG:boto:Port: 8000 2022-09-09T22:39:27.369 DEBUG:boto:Params: {} 2022-09-09T22:39:27.370 DEBUG:boto:Token: None 2022-09-09T22:39:27.370 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.371 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Yd6dO9mWVeHPwspug2EPq0IM83M= 2022-09-09T22:39:27.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Yd6dO9mWVeHPwspug2EPq0IM83M=', 'Content-Length': '0'} 2022-09-09T22:39:27.374 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aa0c974a991196a6-00631bc09f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:39:27.376 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:39:27.376 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:39:27.377 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.377 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.378 DEBUG:boto:Method: GET 2022-09-09T22:39:27.378 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.379 DEBUG:boto:Data: 2022-09-09T22:39:27.379 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.380 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:27.380 DEBUG:boto:Port: 8001 2022-09-09T22:39:27.381 DEBUG:boto:Params: {} 2022-09-09T22:39:27.381 DEBUG:boto:Token: None 2022-09-09T22:39:27.382 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.383 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:16Y85cS5a27HXQs2yOC2LZ4Hmlo= 2022-09-09T22:39:27.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:16Y85cS5a27HXQs2yOC2LZ4Hmlo=', 'Content-Length': '0'} 2022-09-09T22:39:27.386 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f1a8a919294c763d-00631bc09f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.387 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:39:27.394 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:39:27.394 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.395 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.395 DEBUG:boto:Method: GET 2022-09-09T22:39:27.396 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.396 DEBUG:boto:Data: 2022-09-09T22:39:27.397 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.397 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:27.398 DEBUG:boto:Port: 8000 2022-09-09T22:39:27.399 DEBUG:boto:Params: {} 2022-09-09T22:39:27.399 DEBUG:boto:Token: None 2022-09-09T22:39:27.400 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.401 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:16Y85cS5a27HXQs2yOC2LZ4Hmlo= 2022-09-09T22:39:27.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:16Y85cS5a27HXQs2yOC2LZ4Hmlo=', 'Content-Length': '0'} 2022-09-09T22:39:27.405 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dd4178a4f5acc222-00631bc09f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.405 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:27.406 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:27.407 DEBUG:boto:path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:39:27.407 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:39:27.408 DEBUG:boto:Method: GET 2022-09-09T22:39:27.408 DEBUG:boto:Path: /jkcgbj-16/?prefix=dummy 2022-09-09T22:39:27.409 DEBUG:boto:Data: 2022-09-09T22:39:27.409 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.410 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:27.410 DEBUG:boto:Port: 8001 2022-09-09T22:39:27.411 DEBUG:boto:Params: {} 2022-09-09T22:39:27.411 DEBUG:boto:Token: None 2022-09-09T22:39:27.412 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/ 2022-09-09T22:39:27.413 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dTf3as3nrlDALuzN1eyu7D8Nb/w= 2022-09-09T22:39:27.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dTf3as3nrlDALuzN1eyu7D8Nb/w=', 'Content-Length': '0'} 2022-09-09T22:39:27.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d3c835f2ab9a0cf-00631bc09f-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.462 DEBUG:boto:b'jkcgbj-16dummy1000falsedummy2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:27.463 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:27.464 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:27.464 DEBUG:boto:path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:39:27.465 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:39:27.465 DEBUG:boto:Method: GET 2022-09-09T22:39:27.466 DEBUG:boto:Path: /jkcgbj-16/?prefix=dummy 2022-09-09T22:39:27.466 DEBUG:boto:Data: 2022-09-09T22:39:27.467 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.467 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:27.468 DEBUG:boto:Port: 8000 2022-09-09T22:39:27.468 DEBUG:boto:Params: {} 2022-09-09T22:39:27.469 DEBUG:boto:Token: None 2022-09-09T22:39:27.469 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/ 2022-09-09T22:39:27.470 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dTf3as3nrlDALuzN1eyu7D8Nb/w= 2022-09-09T22:39:27.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dTf3as3nrlDALuzN1eyu7D8Nb/w=', 'Content-Length': '0'} 2022-09-09T22:39:27.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f9bb2d9af895e37-00631bc09f-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.477 DEBUG:boto:b'jkcgbj-16dummy1000falsedummy2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:27.478 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:39:27.479 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:39:27.479 DEBUG:boto:Method: HEAD 2022-09-09T22:39:27.480 DEBUG:boto:Path: /jkcgbj-16/dummy 2022-09-09T22:39:27.480 DEBUG:boto:Data: 2022-09-09T22:39:27.481 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.481 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:27.482 DEBUG:boto:Port: 8001 2022-09-09T22:39:27.482 DEBUG:boto:Params: {} 2022-09-09T22:39:27.483 DEBUG:boto:Token: None 2022-09-09T22:39:27.483 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/dummy 2022-09-09T22:39:27.484 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6Md66GFM2yKobUkMBECb2yluAGo= 2022-09-09T22:39:27.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6Md66GFM2yKobUkMBECb2yluAGo=', 'Content-Length': '0'} 2022-09-09T22:39:27.488 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008c22d68355d9a7cc-00631bc09f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.489 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:39:27.490 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:39:27.490 DEBUG:boto:Method: HEAD 2022-09-09T22:39:27.491 DEBUG:boto:Path: /jkcgbj-16/dummy 2022-09-09T22:39:27.491 DEBUG:boto:Data: 2022-09-09T22:39:27.492 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.492 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:27.493 DEBUG:boto:Port: 8000 2022-09-09T22:39:27.493 DEBUG:boto:Params: {} 2022-09-09T22:39:27.494 DEBUG:boto:Token: None 2022-09-09T22:39:27.494 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/dummy 2022-09-09T22:39:27.495 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6Md66GFM2yKobUkMBECb2yluAGo= 2022-09-09T22:39:27.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6Md66GFM2yKobUkMBECb2yluAGo=', 'Content-Length': '0'} 2022-09-09T22:39:27.499 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ba0d5a9d1d3d960f-00631bc09f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:39:27.500 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:39:27.501 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:39:27.501 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.502 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.502 DEBUG:boto:Method: GET 2022-09-09T22:39:27.503 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.504 DEBUG:boto:Data: 2022-09-09T22:39:27.504 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.505 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:27.505 DEBUG:boto:Port: 8001 2022-09-09T22:39:27.506 DEBUG:boto:Params: {} 2022-09-09T22:39:27.506 DEBUG:boto:Token: None 2022-09-09T22:39:27.507 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.508 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:16Y85cS5a27HXQs2yOC2LZ4Hmlo= 2022-09-09T22:39:27.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:16Y85cS5a27HXQs2yOC2LZ4Hmlo=', 'Content-Length': '0'} 2022-09-09T22:39:27.511 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001d4cf8145b329321-00631bc09f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.512 DEBUG:boto:path=/jkcgbj-16/dummy 2022-09-09T22:39:27.512 DEBUG:boto:auth_path=/jkcgbj-16/dummy 2022-09-09T22:39:27.513 DEBUG:boto:path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.514 DEBUG:boto:auth_path=/jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.514 DEBUG:boto:Method: GET 2022-09-09T22:39:27.515 DEBUG:boto:Path: /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.515 DEBUG:boto:Data: 2022-09-09T22:39:27.516 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.516 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:27.517 DEBUG:boto:Port: 8000 2022-09-09T22:39:27.518 DEBUG:boto:Params: {} 2022-09-09T22:39:27.518 DEBUG:boto:Token: None 2022-09-09T22:39:27.519 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/dummy?versionId=J2BEYVUeYOtJqYkdQX3amzkINxiyxts 2022-09-09T22:39:27.519 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:16Y85cS5a27HXQs2yOC2LZ4Hmlo= 2022-09-09T22:39:27.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:16Y85cS5a27HXQs2yOC2LZ4Hmlo=', 'Content-Length': '0'} 2022-09-09T22:39:27.523 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:34:50 GMT'), ('x-amz-version-id', 'J2BEYVUeYOtJqYkdQX3amzkINxiyxts'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f9a8ec749bd80b0b-00631bc09f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.524 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:27.524 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:27.525 DEBUG:boto:path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:39:27.525 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:39:27.526 DEBUG:boto:Method: GET 2022-09-09T22:39:27.526 DEBUG:boto:Path: /jkcgbj-16/?prefix=dummy 2022-09-09T22:39:27.527 DEBUG:boto:Data: 2022-09-09T22:39:27.528 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.528 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:27.529 DEBUG:boto:Port: 8001 2022-09-09T22:39:27.529 DEBUG:boto:Params: {} 2022-09-09T22:39:27.530 DEBUG:boto:Token: None 2022-09-09T22:39:27.531 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/ 2022-09-09T22:39:27.531 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dTf3as3nrlDALuzN1eyu7D8Nb/w= 2022-09-09T22:39:27.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dTf3as3nrlDALuzN1eyu7D8Nb/w=', 'Content-Length': '0'} 2022-09-09T22:39:27.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012a52666e664238c-00631bc09f-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.578 DEBUG:boto:b'jkcgbj-16dummy1000falsedummy2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:27.579 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:27.580 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:27.580 DEBUG:boto:path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:39:27.581 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=dummy 2022-09-09T22:39:27.581 DEBUG:boto:Method: GET 2022-09-09T22:39:27.582 DEBUG:boto:Path: /jkcgbj-16/?prefix=dummy 2022-09-09T22:39:27.582 DEBUG:boto:Data: 2022-09-09T22:39:27.583 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.584 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:27.584 DEBUG:boto:Port: 8000 2022-09-09T22:39:27.585 DEBUG:boto:Params: {} 2022-09-09T22:39:27.585 DEBUG:boto:Token: None 2022-09-09T22:39:27.586 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/ 2022-09-09T22:39:27.587 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dTf3as3nrlDALuzN1eyu7D8Nb/w= 2022-09-09T22:39:27.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dTf3as3nrlDALuzN1eyu7D8Nb/w=', 'Content-Length': '0'} 2022-09-09T22:39:27.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f3c6164ed4b37e2-00631bc09f-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.594 DEBUG:boto:b'jkcgbj-16dummy1000falsedummy2022-09-09T22:34:50.054Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:27.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:39:27.595 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:27.596 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:27.597 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.597 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.598 DEBUG:boto:Method: GET 2022-09-09T22:39:27.598 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.599 DEBUG:boto:Data: 2022-09-09T22:39:27.599 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.600 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:27.600 DEBUG:boto:Port: 8001 2022-09-09T22:39:27.601 DEBUG:boto:Params: {} 2022-09-09T22:39:27.602 DEBUG:boto:Token: None 2022-09-09T22:39:27.602 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.603 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pGHTGk9he0lvrILHojKKeMb8Yw8= 2022-09-09T22:39:27.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pGHTGk9he0lvrILHojKKeMb8Yw8=', 'Content-Length': '0'} 2022-09-09T22:39:27.607 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000530b08e5a6bf5e9d-00631bc09f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.608 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:27.609 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:27.609 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.610 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.610 DEBUG:boto:Method: GET 2022-09-09T22:39:27.611 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.612 DEBUG:boto:Data: 2022-09-09T22:39:27.612 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.613 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:27.613 DEBUG:boto:Port: 8000 2022-09-09T22:39:27.614 DEBUG:boto:Params: {} 2022-09-09T22:39:27.614 DEBUG:boto:Token: None 2022-09-09T22:39:27.615 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.616 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pGHTGk9he0lvrILHojKKeMb8Yw8= 2022-09-09T22:39:27.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pGHTGk9he0lvrILHojKKeMb8Yw8=', 'Content-Length': '0'} 2022-09-09T22:39:27.619 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058fa5194d01c35ef-00631bc09f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.662 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:27.663 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:27.663 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.664 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.664 DEBUG:boto:Method: HEAD 2022-09-09T22:39:27.665 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.666 DEBUG:boto:Data: 2022-09-09T22:39:27.666 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.667 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:27.667 DEBUG:boto:Port: 8001 2022-09-09T22:39:27.668 DEBUG:boto:Params: {} 2022-09-09T22:39:27.668 DEBUG:boto:Token: None 2022-09-09T22:39:27.669 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.670 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:y15rxQmzwHLC0m+VXNxWPzdnLBY= 2022-09-09T22:39:27.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:y15rxQmzwHLC0m+VXNxWPzdnLBY=', 'Content-Length': '0'} 2022-09-09T22:39:27.673 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000051dfcab12f6a84bd-00631bc09f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.674 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:27.674 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:27.675 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.676 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.676 DEBUG:boto:Method: HEAD 2022-09-09T22:39:27.677 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.677 DEBUG:boto:Data: 2022-09-09T22:39:27.678 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.678 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:27.679 DEBUG:boto:Port: 8000 2022-09-09T22:39:27.680 DEBUG:boto:Params: {} 2022-09-09T22:39:27.681 DEBUG:boto:Token: None 2022-09-09T22:39:27.682 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.683 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:y15rxQmzwHLC0m+VXNxWPzdnLBY= 2022-09-09T22:39:27.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:y15rxQmzwHLC0m+VXNxWPzdnLBY=', 'Content-Length': '0'} 2022-09-09T22:39:27.688 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004cc9189919bb4d1-00631bc09f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:39:27.690 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:27.691 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:27.692 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.693 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.694 DEBUG:boto:Method: GET 2022-09-09T22:39:27.695 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.696 DEBUG:boto:Data: 2022-09-09T22:39:27.697 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.698 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:27.700 DEBUG:boto:Port: 8001 2022-09-09T22:39:27.701 DEBUG:boto:Params: {} 2022-09-09T22:39:27.702 DEBUG:boto:Token: None 2022-09-09T22:39:27.703 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.704 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pGHTGk9he0lvrILHojKKeMb8Yw8= 2022-09-09T22:39:27.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pGHTGk9he0lvrILHojKKeMb8Yw8=', 'Content-Length': '0'} 2022-09-09T22:39:27.709 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c05a8a85ef24ec8-00631bc09f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.711 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:27.712 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:27.713 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.714 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.715 DEBUG:boto:Method: GET 2022-09-09T22:39:27.716 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.717 DEBUG:boto:Data: 2022-09-09T22:39:27.717 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.718 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:27.718 DEBUG:boto:Port: 8000 2022-09-09T22:39:27.719 DEBUG:boto:Params: {} 2022-09-09T22:39:27.720 DEBUG:boto:Token: None 2022-09-09T22:39:27.720 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.721 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pGHTGk9he0lvrILHojKKeMb8Yw8= 2022-09-09T22:39:27.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pGHTGk9he0lvrILHojKKeMb8Yw8=', 'Content-Length': '0'} 2022-09-09T22:39:27.725 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086a8260af814f5d8-00631bc09f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.726 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:27.727 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:27.727 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:27.728 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:27.728 DEBUG:boto:Method: GET 2022-09-09T22:39:27.729 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:27.729 DEBUG:boto:Data: 2022-09-09T22:39:27.730 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.731 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:27.731 DEBUG:boto:Port: 8001 2022-09-09T22:39:27.732 DEBUG:boto:Params: {} 2022-09-09T22:39:27.732 DEBUG:boto:Token: None 2022-09-09T22:39:27.733 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/ 2022-09-09T22:39:27.734 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dTf3as3nrlDALuzN1eyu7D8Nb/w= 2022-09-09T22:39:27.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dTf3as3nrlDALuzN1eyu7D8Nb/w=', 'Content-Length': '0'} 2022-09-09T22:39:27.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046a39411a00f109c-00631bc09f-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.743 DEBUG:boto:b'jkcgbj-16obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:27.744 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:27.745 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:27.745 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:27.746 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:27.747 DEBUG:boto:Method: GET 2022-09-09T22:39:27.747 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:27.748 DEBUG:boto:Data: 2022-09-09T22:39:27.748 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.749 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:27.749 DEBUG:boto:Port: 8000 2022-09-09T22:39:27.750 DEBUG:boto:Params: {} 2022-09-09T22:39:27.750 DEBUG:boto:Token: None 2022-09-09T22:39:27.751 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/ 2022-09-09T22:39:27.751 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dTf3as3nrlDALuzN1eyu7D8Nb/w= 2022-09-09T22:39:27.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dTf3as3nrlDALuzN1eyu7D8Nb/w=', 'Content-Length': '0'} 2022-09-09T22:39:27.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000861d5425c68bd872-00631bc09f-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.759 DEBUG:boto:b'jkcgbj-16obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:27.760 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:27.760 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:27.761 DEBUG:boto:Method: HEAD 2022-09-09T22:39:27.762 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1 2022-09-09T22:39:27.762 DEBUG:boto:Data: 2022-09-09T22:39:27.763 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.763 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:27.764 DEBUG:boto:Port: 8001 2022-09-09T22:39:27.764 DEBUG:boto:Params: {} 2022-09-09T22:39:27.765 DEBUG:boto:Token: None 2022-09-09T22:39:27.765 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/obj-test-zone1 2022-09-09T22:39:27.766 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:D79vX8OTxt0CXp2P1dnB8NKVnDs= 2022-09-09T22:39:27.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:D79vX8OTxt0CXp2P1dnB8NKVnDs=', 'Content-Length': '0'} 2022-09-09T22:39:27.772 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6cc262a6e8e2430-00631bc09f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.773 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:27.773 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:27.774 DEBUG:boto:Method: HEAD 2022-09-09T22:39:27.774 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1 2022-09-09T22:39:27.775 DEBUG:boto:Data: 2022-09-09T22:39:27.775 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.776 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:27.776 DEBUG:boto:Port: 8000 2022-09-09T22:39:27.777 DEBUG:boto:Params: {} 2022-09-09T22:39:27.777 DEBUG:boto:Token: None 2022-09-09T22:39:27.778 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/obj-test-zone1 2022-09-09T22:39:27.778 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:D79vX8OTxt0CXp2P1dnB8NKVnDs= 2022-09-09T22:39:27.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:D79vX8OTxt0CXp2P1dnB8NKVnDs=', 'Content-Length': '0'} 2022-09-09T22:39:27.783 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008681249977117661-00631bc09f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:39:27.784 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:27.785 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:27.785 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.786 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.787 DEBUG:boto:Method: GET 2022-09-09T22:39:27.787 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.788 DEBUG:boto:Data: 2022-09-09T22:39:27.788 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.789 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:27.789 DEBUG:boto:Port: 8001 2022-09-09T22:39:27.790 DEBUG:boto:Params: {} 2022-09-09T22:39:27.790 DEBUG:boto:Token: None 2022-09-09T22:39:27.791 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.791 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pGHTGk9he0lvrILHojKKeMb8Yw8= 2022-09-09T22:39:27.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pGHTGk9he0lvrILHojKKeMb8Yw8=', 'Content-Length': '0'} 2022-09-09T22:39:27.795 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e38c9d65bb342b61-00631bc09f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.838 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:27.839 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1 2022-09-09T22:39:27.839 DEBUG:boto:path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.840 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.840 DEBUG:boto:Method: GET 2022-09-09T22:39:27.841 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.841 DEBUG:boto:Data: 2022-09-09T22:39:27.842 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.843 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:27.843 DEBUG:boto:Port: 8000 2022-09-09T22:39:27.844 DEBUG:boto:Params: {} 2022-09-09T22:39:27.844 DEBUG:boto:Token: None 2022-09-09T22:39:27.845 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/obj-test-zone1?versionId=123zMLKlbUHrE3mBcpUCJblKnWDeBXD 2022-09-09T22:39:27.845 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pGHTGk9he0lvrILHojKKeMb8Yw8= 2022-09-09T22:39:27.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pGHTGk9he0lvrILHojKKeMb8Yw8=', 'Content-Length': '0'} 2022-09-09T22:39:27.849 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '123zMLKlbUHrE3mBcpUCJblKnWDeBXD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8c2eacfa6831ad5-00631bc09f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.850 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:27.851 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:27.851 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:27.852 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:27.853 DEBUG:boto:Method: GET 2022-09-09T22:39:27.853 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:27.854 DEBUG:boto:Data: 2022-09-09T22:39:27.854 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.855 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:27.855 DEBUG:boto:Port: 8001 2022-09-09T22:39:27.856 DEBUG:boto:Params: {} 2022-09-09T22:39:27.856 DEBUG:boto:Token: None 2022-09-09T22:39:27.857 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/ 2022-09-09T22:39:27.857 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dTf3as3nrlDALuzN1eyu7D8Nb/w= 2022-09-09T22:39:27.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dTf3as3nrlDALuzN1eyu7D8Nb/w=', 'Content-Length': '0'} 2022-09-09T22:39:27.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cebb14677d9b7c9-00631bc09f-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.865 DEBUG:boto:b'jkcgbj-16obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:27.866 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:27.867 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:27.867 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:27.868 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:27.868 DEBUG:boto:Method: GET 2022-09-09T22:39:27.869 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone1 2022-09-09T22:39:27.870 DEBUG:boto:Data: 2022-09-09T22:39:27.870 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.871 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:27.871 DEBUG:boto:Port: 8000 2022-09-09T22:39:27.872 DEBUG:boto:Params: {} 2022-09-09T22:39:27.872 DEBUG:boto:Token: None 2022-09-09T22:39:27.873 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/ 2022-09-09T22:39:27.874 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dTf3as3nrlDALuzN1eyu7D8Nb/w= 2022-09-09T22:39:27.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dTf3as3nrlDALuzN1eyu7D8Nb/w=', 'Content-Length': '0'} 2022-09-09T22:39:27.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ae784175f85eb31-00631bc09f-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.881 DEBUG:boto:b'jkcgbj-16obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.329Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:27.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:39:27.883 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:27.883 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:27.884 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:27.884 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:27.885 DEBUG:boto:Method: GET 2022-09-09T22:39:27.885 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:27.886 DEBUG:boto:Data: 2022-09-09T22:39:27.886 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.887 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:27.887 DEBUG:boto:Port: 8001 2022-09-09T22:39:27.888 DEBUG:boto:Params: {} 2022-09-09T22:39:27.888 DEBUG:boto:Token: None 2022-09-09T22:39:27.889 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:27.890 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:MRBDy52SXlDd84Z0n66Iq9CLr/A= 2022-09-09T22:39:27.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:MRBDy52SXlDd84Z0n66Iq9CLr/A=', 'Content-Length': '0'} 2022-09-09T22:39:27.893 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9b869b62d6a1002-00631bc09f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.894 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:27.895 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:27.895 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:27.896 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:27.896 DEBUG:boto:Method: GET 2022-09-09T22:39:27.897 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:27.897 DEBUG:boto:Data: 2022-09-09T22:39:27.898 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.899 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:27.899 DEBUG:boto:Port: 8000 2022-09-09T22:39:27.900 DEBUG:boto:Params: {} 2022-09-09T22:39:27.900 DEBUG:boto:Token: None 2022-09-09T22:39:27.901 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:27.901 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:MRBDy52SXlDd84Z0n66Iq9CLr/A= 2022-09-09T22:39:27.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:MRBDy52SXlDd84Z0n66Iq9CLr/A=', 'Content-Length': '0'} 2022-09-09T22:39:27.905 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c33913782379616c-00631bc09f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.906 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:27.906 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:27.907 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:27.907 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:27.908 DEBUG:boto:Method: HEAD 2022-09-09T22:39:27.908 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:27.909 DEBUG:boto:Data: 2022-09-09T22:39:27.909 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.910 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:27.910 DEBUG:boto:Port: 8001 2022-09-09T22:39:27.911 DEBUG:boto:Params: {} 2022-09-09T22:39:27.911 DEBUG:boto:Token: None 2022-09-09T22:39:27.912 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:27.912 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:DxStfecXqspHleNioCXlTmX14To= 2022-09-09T22:39:27.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:DxStfecXqspHleNioCXlTmX14To=', 'Content-Length': '0'} 2022-09-09T22:39:27.916 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fec1b52fc3b9c753-00631bc09f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.918 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:27.918 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:27.919 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:27.919 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:27.920 DEBUG:boto:Method: HEAD 2022-09-09T22:39:27.920 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:27.921 DEBUG:boto:Data: 2022-09-09T22:39:27.922 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.923 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:27.924 DEBUG:boto:Port: 8000 2022-09-09T22:39:27.925 DEBUG:boto:Params: {} 2022-09-09T22:39:27.926 DEBUG:boto:Token: None 2022-09-09T22:39:27.927 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:27.928 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:DxStfecXqspHleNioCXlTmX14To= 2022-09-09T22:39:27.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:DxStfecXqspHleNioCXlTmX14To=', 'Content-Length': '0'} 2022-09-09T22:39:27.932 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f134ab8bad053f1-00631bc09f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:39:27.934 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:27.935 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:27.936 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:27.937 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:27.937 DEBUG:boto:Method: GET 2022-09-09T22:39:27.938 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:27.939 DEBUG:boto:Data: 2022-09-09T22:39:27.940 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.940 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:27.941 DEBUG:boto:Port: 8001 2022-09-09T22:39:27.941 DEBUG:boto:Params: {} 2022-09-09T22:39:27.942 DEBUG:boto:Token: None 2022-09-09T22:39:27.942 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:27.943 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:MRBDy52SXlDd84Z0n66Iq9CLr/A= 2022-09-09T22:39:27.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:MRBDy52SXlDd84Z0n66Iq9CLr/A=', 'Content-Length': '0'} 2022-09-09T22:39:27.947 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef1478bf15b86b94-00631bc09f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.947 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:27.948 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:27.948 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:27.949 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:27.949 DEBUG:boto:Method: GET 2022-09-09T22:39:27.950 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:27.950 DEBUG:boto:Data: 2022-09-09T22:39:27.951 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.952 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:27.952 DEBUG:boto:Port: 8000 2022-09-09T22:39:27.953 DEBUG:boto:Params: {} 2022-09-09T22:39:27.953 DEBUG:boto:Token: None 2022-09-09T22:39:27.954 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:27.955 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:MRBDy52SXlDd84Z0n66Iq9CLr/A= 2022-09-09T22:39:27.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:MRBDy52SXlDd84Z0n66Iq9CLr/A=', 'Content-Length': '0'} 2022-09-09T22:39:27.958 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc70c5644d9e9d18-00631bc09f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:27.959 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:27.960 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:27.960 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:27.961 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:27.961 DEBUG:boto:Method: GET 2022-09-09T22:39:27.962 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:27.963 DEBUG:boto:Data: 2022-09-09T22:39:27.963 DEBUG:boto:Headers: {} 2022-09-09T22:39:27.964 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:27.964 DEBUG:boto:Port: 8001 2022-09-09T22:39:27.965 DEBUG:boto:Params: {} 2022-09-09T22:39:27.965 DEBUG:boto:Token: None 2022-09-09T22:39:27.966 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:27 GMT /jkcgbj-16/ 2022-09-09T22:39:27.967 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dTf3as3nrlDALuzN1eyu7D8Nb/w= 2022-09-09T22:39:27.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:27 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dTf3as3nrlDALuzN1eyu7D8Nb/w=', 'Content-Length': '0'} 2022-09-09T22:39:27.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072d8cd20a3a4aaf0-00631bc09f-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:28.014 DEBUG:boto:b'jkcgbj-16obj-test-zone21000falseobj-test-zone22022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:28.015 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:28.016 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:28.017 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:28.017 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:28.018 DEBUG:boto:Method: GET 2022-09-09T22:39:28.018 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:28.019 DEBUG:boto:Data: 2022-09-09T22:39:28.019 DEBUG:boto:Headers: {} 2022-09-09T22:39:28.020 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:28.020 DEBUG:boto:Port: 8000 2022-09-09T22:39:28.021 DEBUG:boto:Params: {} 2022-09-09T22:39:28.021 DEBUG:boto:Token: None 2022-09-09T22:39:28.022 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:28 GMT /jkcgbj-16/ 2022-09-09T22:39:28.022 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jvB5k6styuPDzoZERrWKrGx5akw= 2022-09-09T22:39:28.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:28 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jvB5k6styuPDzoZERrWKrGx5akw=', 'Content-Length': '0'} 2022-09-09T22:39:28.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d3cca20af02c72f-00631bc0a0-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:28.030 DEBUG:boto:b'jkcgbj-16obj-test-zone21000falseobj-test-zone22022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:28.031 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:28.031 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:28.032 DEBUG:boto:Method: HEAD 2022-09-09T22:39:28.032 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2 2022-09-09T22:39:28.033 DEBUG:boto:Data: 2022-09-09T22:39:28.033 DEBUG:boto:Headers: {} 2022-09-09T22:39:28.034 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:28.034 DEBUG:boto:Port: 8001 2022-09-09T22:39:28.035 DEBUG:boto:Params: {} 2022-09-09T22:39:28.035 DEBUG:boto:Token: None 2022-09-09T22:39:28.036 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:28 GMT /jkcgbj-16/obj-test-zone2 2022-09-09T22:39:28.037 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FGEJ0bLtUbPAe8gKLkDIfJhdIZk= 2022-09-09T22:39:28.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:28 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FGEJ0bLtUbPAe8gKLkDIfJhdIZk=', 'Content-Length': '0'} 2022-09-09T22:39:28.041 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009620b38ca15731a5-00631bc0a0-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:28.042 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:28.043 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:28.043 DEBUG:boto:Method: HEAD 2022-09-09T22:39:28.044 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2 2022-09-09T22:39:28.045 DEBUG:boto:Data: 2022-09-09T22:39:28.045 DEBUG:boto:Headers: {} 2022-09-09T22:39:28.046 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:28.047 DEBUG:boto:Port: 8000 2022-09-09T22:39:28.048 DEBUG:boto:Params: {} 2022-09-09T22:39:28.049 DEBUG:boto:Token: None 2022-09-09T22:39:28.050 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:28 GMT /jkcgbj-16/obj-test-zone2 2022-09-09T22:39:28.051 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FGEJ0bLtUbPAe8gKLkDIfJhdIZk= 2022-09-09T22:39:28.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:28 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FGEJ0bLtUbPAe8gKLkDIfJhdIZk=', 'Content-Length': '0'} 2022-09-09T22:39:28.056 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9c39cd2abcdc7ab-00631bc0a0-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:28.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:39:28.059 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:28.060 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:28.061 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:28.062 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:28.063 DEBUG:boto:Method: GET 2022-09-09T22:39:28.064 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:28.065 DEBUG:boto:Data: 2022-09-09T22:39:28.066 DEBUG:boto:Headers: {} 2022-09-09T22:39:28.067 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:28.068 DEBUG:boto:Port: 8001 2022-09-09T22:39:28.069 DEBUG:boto:Params: {} 2022-09-09T22:39:28.070 DEBUG:boto:Token: None 2022-09-09T22:39:28.071 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:28 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:28.072 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:J4z5FbN/79wNb6+RwAVUdO6MhSM= 2022-09-09T22:39:28.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:28 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:J4z5FbN/79wNb6+RwAVUdO6MhSM=', 'Content-Length': '0'} 2022-09-09T22:39:28.077 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004398817a945868d7-00631bc0a0-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:28.078 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:28.079 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2 2022-09-09T22:39:28.080 DEBUG:boto:path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:28.081 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:28.082 DEBUG:boto:Method: GET 2022-09-09T22:39:28.083 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:28.084 DEBUG:boto:Data: 2022-09-09T22:39:28.085 DEBUG:boto:Headers: {} 2022-09-09T22:39:28.086 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:28.087 DEBUG:boto:Port: 8000 2022-09-09T22:39:28.088 DEBUG:boto:Params: {} 2022-09-09T22:39:28.089 DEBUG:boto:Token: None 2022-09-09T22:39:28.090 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:28 GMT /jkcgbj-16/obj-test-zone2?versionId=MZ.b57oN5HI9SlBmYt.8.2016m6aPMf 2022-09-09T22:39:28.090 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:J4z5FbN/79wNb6+RwAVUdO6MhSM= 2022-09-09T22:39:28.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:28 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:J4z5FbN/79wNb6+RwAVUdO6MhSM=', 'Content-Length': '0'} 2022-09-09T22:39:28.094 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'MZ.b57oN5HI9SlBmYt.8.2016m6aPMf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6db5c549baab6ad-00631bc0a0-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:28.138 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:28.139 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:28.140 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:28.141 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:28.142 DEBUG:boto:Method: GET 2022-09-09T22:39:28.142 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:28.143 DEBUG:boto:Data: 2022-09-09T22:39:28.143 DEBUG:boto:Headers: {} 2022-09-09T22:39:28.144 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:28.144 DEBUG:boto:Port: 8001 2022-09-09T22:39:28.145 DEBUG:boto:Params: {} 2022-09-09T22:39:28.145 DEBUG:boto:Token: None 2022-09-09T22:39:28.146 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:28 GMT /jkcgbj-16/ 2022-09-09T22:39:28.147 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jvB5k6styuPDzoZERrWKrGx5akw= 2022-09-09T22:39:28.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:28 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jvB5k6styuPDzoZERrWKrGx5akw=', 'Content-Length': '0'} 2022-09-09T22:39:28.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2d800dbbae3983b-00631bc0a0-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:28.154 DEBUG:boto:b'jkcgbj-16obj-test-zone21000falseobj-test-zone22022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:28.155 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:28.155 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:28.156 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:28.156 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:28.157 DEBUG:boto:Method: GET 2022-09-09T22:39:28.157 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone2 2022-09-09T22:39:28.158 DEBUG:boto:Data: 2022-09-09T22:39:28.158 DEBUG:boto:Headers: {} 2022-09-09T22:39:28.159 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:28.160 DEBUG:boto:Port: 8000 2022-09-09T22:39:28.160 DEBUG:boto:Params: {} 2022-09-09T22:39:28.161 DEBUG:boto:Token: None 2022-09-09T22:39:28.161 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:28 GMT /jkcgbj-16/ 2022-09-09T22:39:28.162 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jvB5k6styuPDzoZERrWKrGx5akw= 2022-09-09T22:39:28.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:28 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jvB5k6styuPDzoZERrWKrGx5akw=', 'Content-Length': '0'} 2022-09-09T22:39:28.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0e29625b457edad-00631bc0a0-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:28.169 DEBUG:boto:b'jkcgbj-16obj-test-zone21000falseobj-test-zone22022-09-09T22:36:01.532Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:28.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:39:28.170 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:28.170 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:28.171 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.171 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.172 DEBUG:boto:Method: GET 2022-09-09T22:39:28.173 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.173 DEBUG:boto:Data: 2022-09-09T22:39:28.174 DEBUG:boto:Headers: {} 2022-09-09T22:39:28.174 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:28.175 DEBUG:boto:Port: 8001 2022-09-09T22:39:28.175 DEBUG:boto:Params: {} 2022-09-09T22:39:28.176 DEBUG:boto:Token: None 2022-09-09T22:39:28.177 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:28 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.177 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KQx5YyIQVdpydPeWywls+2yPR0g= 2022-09-09T22:39:28.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:28 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KQx5YyIQVdpydPeWywls+2yPR0g=', 'Content-Length': '0'} 2022-09-09T22:39:28.181 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002fba9c5549f1a578-00631bc0a0-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:28.181 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:28.182 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:28.183 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.183 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.184 DEBUG:boto:Method: GET 2022-09-09T22:39:28.184 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.185 DEBUG:boto:Data: 2022-09-09T22:39:28.185 DEBUG:boto:Headers: {} 2022-09-09T22:39:28.186 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:28.186 DEBUG:boto:Port: 8000 2022-09-09T22:39:28.187 DEBUG:boto:Params: {} 2022-09-09T22:39:28.188 DEBUG:boto:Token: None 2022-09-09T22:39:28.188 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:28 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.189 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KQx5YyIQVdpydPeWywls+2yPR0g= 2022-09-09T22:39:28.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:28 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KQx5YyIQVdpydPeWywls+2yPR0g=', 'Content-Length': '0'} 2022-09-09T22:39:28.192 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa0dfcc9706a015f-00631bc0a0-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:28.193 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:28.194 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:28.194 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.195 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.195 DEBUG:boto:Method: HEAD 2022-09-09T22:39:28.196 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.196 DEBUG:boto:Data: 2022-09-09T22:39:28.197 DEBUG:boto:Headers: {} 2022-09-09T22:39:28.197 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:28.198 DEBUG:boto:Port: 8001 2022-09-09T22:39:28.198 DEBUG:boto:Params: {} 2022-09-09T22:39:28.199 DEBUG:boto:Token: None 2022-09-09T22:39:28.200 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:28 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.200 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:yoUtzUr2mgjIEjsgYVrByV6g8X0= 2022-09-09T22:39:28.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:28 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:yoUtzUr2mgjIEjsgYVrByV6g8X0=', 'Content-Length': '0'} 2022-09-09T22:39:28.204 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000913057e134f3ea81-00631bc0a0-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:28.204 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:28.205 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:28.205 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.206 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.207 DEBUG:boto:Method: HEAD 2022-09-09T22:39:28.207 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.208 DEBUG:boto:Data: 2022-09-09T22:39:28.208 DEBUG:boto:Headers: {} 2022-09-09T22:39:28.209 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:28.209 DEBUG:boto:Port: 8000 2022-09-09T22:39:28.210 DEBUG:boto:Params: {} 2022-09-09T22:39:28.210 DEBUG:boto:Token: None 2022-09-09T22:39:28.211 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:28 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.211 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:yoUtzUr2mgjIEjsgYVrByV6g8X0= 2022-09-09T22:39:28.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:28 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:yoUtzUr2mgjIEjsgYVrByV6g8X0=', 'Content-Length': '0'} 2022-09-09T22:39:28.215 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000081449616f6a33623-00631bc0a0-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:28.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:39:28.216 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:28.217 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:28.217 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.218 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.218 DEBUG:boto:Method: GET 2022-09-09T22:39:28.219 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.220 DEBUG:boto:Data: 2022-09-09T22:39:28.220 DEBUG:boto:Headers: {} 2022-09-09T22:39:28.221 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:28.221 DEBUG:boto:Port: 8001 2022-09-09T22:39:28.222 DEBUG:boto:Params: {} 2022-09-09T22:39:28.222 DEBUG:boto:Token: None 2022-09-09T22:39:28.223 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:28 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.223 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KQx5YyIQVdpydPeWywls+2yPR0g= 2022-09-09T22:39:28.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:28 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KQx5YyIQVdpydPeWywls+2yPR0g=', 'Content-Length': '0'} 2022-09-09T22:39:28.227 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db94c58b1a1aee8c-00631bc0a0-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:28.270 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:28.271 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:28.271 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.272 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.272 DEBUG:boto:Method: GET 2022-09-09T22:39:28.273 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.273 DEBUG:boto:Data: 2022-09-09T22:39:28.274 DEBUG:boto:Headers: {} 2022-09-09T22:39:28.274 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:28.275 DEBUG:boto:Port: 8000 2022-09-09T22:39:28.275 DEBUG:boto:Params: {} 2022-09-09T22:39:28.276 DEBUG:boto:Token: None 2022-09-09T22:39:28.277 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:28 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.277 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KQx5YyIQVdpydPeWywls+2yPR0g= 2022-09-09T22:39:28.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:28 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KQx5YyIQVdpydPeWywls+2yPR0g=', 'Content-Length': '0'} 2022-09-09T22:39:28.281 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000075eadc5744a638fd-00631bc0a0-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:28.282 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:28.282 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:28.283 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:28.283 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:28.284 DEBUG:boto:Method: GET 2022-09-09T22:39:28.285 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:28.285 DEBUG:boto:Data: 2022-09-09T22:39:28.286 DEBUG:boto:Headers: {} 2022-09-09T22:39:28.286 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:28.287 DEBUG:boto:Port: 8001 2022-09-09T22:39:28.287 DEBUG:boto:Params: {} 2022-09-09T22:39:28.288 DEBUG:boto:Token: None 2022-09-09T22:39:28.288 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:28 GMT /jkcgbj-16/ 2022-09-09T22:39:28.289 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jvB5k6styuPDzoZERrWKrGx5akw= 2022-09-09T22:39:28.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:28 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jvB5k6styuPDzoZERrWKrGx5akw=', 'Content-Length': '0'} 2022-09-09T22:39:28.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc51785aa02629a9-00631bc0a0-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:28.296 DEBUG:boto:b'jkcgbj-16obj-test-zone31000falseobj-test-zone32022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:28.297 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:28.298 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:28.298 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:28.299 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:28.299 DEBUG:boto:Method: GET 2022-09-09T22:39:28.300 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:28.300 DEBUG:boto:Data: 2022-09-09T22:39:28.301 DEBUG:boto:Headers: {} 2022-09-09T22:39:28.301 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:28.302 DEBUG:boto:Port: 8000 2022-09-09T22:39:28.302 DEBUG:boto:Params: {} 2022-09-09T22:39:28.303 DEBUG:boto:Token: None 2022-09-09T22:39:28.303 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:28 GMT /jkcgbj-16/ 2022-09-09T22:39:28.304 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jvB5k6styuPDzoZERrWKrGx5akw= 2022-09-09T22:39:28.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:28 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jvB5k6styuPDzoZERrWKrGx5akw=', 'Content-Length': '0'} 2022-09-09T22:39:28.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071599be39eae36c5-00631bc0a0-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:28.311 DEBUG:boto:b'jkcgbj-16obj-test-zone31000falseobj-test-zone32022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:28.311 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:28.312 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:28.312 DEBUG:boto:Method: HEAD 2022-09-09T22:39:28.313 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3 2022-09-09T22:39:28.313 DEBUG:boto:Data: 2022-09-09T22:39:28.314 DEBUG:boto:Headers: {} 2022-09-09T22:39:28.314 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:28.315 DEBUG:boto:Port: 8001 2022-09-09T22:39:28.315 DEBUG:boto:Params: {} 2022-09-09T22:39:28.316 DEBUG:boto:Token: None 2022-09-09T22:39:28.317 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:28 GMT /jkcgbj-16/obj-test-zone3 2022-09-09T22:39:28.317 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:kC1ugsa9wHyN2zWIUQ0l5VIea4E= 2022-09-09T22:39:28.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:28 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:kC1ugsa9wHyN2zWIUQ0l5VIea4E=', 'Content-Length': '0'} 2022-09-09T22:39:28.321 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a3592497d5a4f77-00631bc0a0-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:28.322 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:28.323 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:28.323 DEBUG:boto:Method: HEAD 2022-09-09T22:39:28.324 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3 2022-09-09T22:39:28.324 DEBUG:boto:Data: 2022-09-09T22:39:28.325 DEBUG:boto:Headers: {} 2022-09-09T22:39:28.325 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:28.326 DEBUG:boto:Port: 8000 2022-09-09T22:39:28.326 DEBUG:boto:Params: {} 2022-09-09T22:39:28.327 DEBUG:boto:Token: None 2022-09-09T22:39:28.327 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:28 GMT /jkcgbj-16/obj-test-zone3 2022-09-09T22:39:28.328 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:kC1ugsa9wHyN2zWIUQ0l5VIea4E= 2022-09-09T22:39:28.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:28 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:kC1ugsa9wHyN2zWIUQ0l5VIea4E=', 'Content-Length': '0'} 2022-09-09T22:39:28.332 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aee8c2902adaae39-00631bc0a0-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:28.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:39:28.334 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:28.334 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:28.335 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.335 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.336 DEBUG:boto:Method: GET 2022-09-09T22:39:28.336 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.337 DEBUG:boto:Data: 2022-09-09T22:39:28.337 DEBUG:boto:Headers: {} 2022-09-09T22:39:28.338 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:28.338 DEBUG:boto:Port: 8001 2022-09-09T22:39:28.339 DEBUG:boto:Params: {} 2022-09-09T22:39:28.339 DEBUG:boto:Token: None 2022-09-09T22:39:28.340 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:28 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.341 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KQx5YyIQVdpydPeWywls+2yPR0g= 2022-09-09T22:39:28.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:28 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KQx5YyIQVdpydPeWywls+2yPR0g=', 'Content-Length': '0'} 2022-09-09T22:39:28.344 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d22d94c69451fcf4-00631bc0a0-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:28.345 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:28.345 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3 2022-09-09T22:39:28.346 DEBUG:boto:path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.347 DEBUG:boto:auth_path=/jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.347 DEBUG:boto:Method: GET 2022-09-09T22:39:28.348 DEBUG:boto:Path: /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.348 DEBUG:boto:Data: 2022-09-09T22:39:28.349 DEBUG:boto:Headers: {} 2022-09-09T22:39:28.349 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:28.350 DEBUG:boto:Port: 8000 2022-09-09T22:39:28.350 DEBUG:boto:Params: {} 2022-09-09T22:39:28.351 DEBUG:boto:Token: None 2022-09-09T22:39:28.351 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:28 GMT /jkcgbj-16/obj-test-zone3?versionId=AesdIyOhh8xrq24fWFi6fMM39PxhXM8 2022-09-09T22:39:28.352 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KQx5YyIQVdpydPeWywls+2yPR0g= 2022-09-09T22:39:28.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:28 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KQx5YyIQVdpydPeWywls+2yPR0g=', 'Content-Length': '0'} 2022-09-09T22:39:28.355 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'AesdIyOhh8xrq24fWFi6fMM39PxhXM8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b37103e8b3ae219a-00631bc0a0-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:28.398 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:28.399 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:28.400 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:28.400 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:28.401 DEBUG:boto:Method: GET 2022-09-09T22:39:28.401 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:28.402 DEBUG:boto:Data: 2022-09-09T22:39:28.402 DEBUG:boto:Headers: {} 2022-09-09T22:39:28.403 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:28.403 DEBUG:boto:Port: 8001 2022-09-09T22:39:28.404 DEBUG:boto:Params: {} 2022-09-09T22:39:28.405 DEBUG:boto:Token: None 2022-09-09T22:39:28.405 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:28 GMT /jkcgbj-16/ 2022-09-09T22:39:28.406 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jvB5k6styuPDzoZERrWKrGx5akw= 2022-09-09T22:39:28.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:28 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jvB5k6styuPDzoZERrWKrGx5akw=', 'Content-Length': '0'} 2022-09-09T22:39:28.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c31985cdf4dc7028-00631bc0a0-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:28.413 DEBUG:boto:b'jkcgbj-16obj-test-zone31000falseobj-test-zone32022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:28.414 DEBUG:boto:path=/jkcgbj-16/ 2022-09-09T22:39:28.415 DEBUG:boto:auth_path=/jkcgbj-16/ 2022-09-09T22:39:28.415 DEBUG:boto:path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:28.416 DEBUG:boto:auth_path=/jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:28.416 DEBUG:boto:Method: GET 2022-09-09T22:39:28.417 DEBUG:boto:Path: /jkcgbj-16/?prefix=obj-test-zone3 2022-09-09T22:39:28.418 DEBUG:boto:Data: 2022-09-09T22:39:28.418 DEBUG:boto:Headers: {} 2022-09-09T22:39:28.419 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:28.419 DEBUG:boto:Port: 8000 2022-09-09T22:39:28.420 DEBUG:boto:Params: {} 2022-09-09T22:39:28.420 DEBUG:boto:Token: None 2022-09-09T22:39:28.421 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:28 GMT /jkcgbj-16/ 2022-09-09T22:39:28.422 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jvB5k6styuPDzoZERrWKrGx5akw= 2022-09-09T22:39:28.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:28 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jvB5k6styuPDzoZERrWKrGx5akw=', 'Content-Length': '0'} 2022-09-09T22:39:28.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2a524b28587adb6-00631bc0a0-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:28.429 DEBUG:boto:b'jkcgbj-16obj-test-zone31000falseobj-test-zone32022-09-09T22:36:01.738Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:28.429 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-16 zones={test-zone3, test-zone2} 2022-09-09T22:39:28.430 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jkcgbj-17 2022-09-09T22:39:28.431 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:28.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:28.432 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:39:28.484 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:39:28.485 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:39:28.563 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:39:28.563+0000 7fbd0ef11dc0 1 bucket sync caught up with source: 2022-09-09T22:39:28.564 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000011.40.5, , 00000000004.13.5, , 00000000008.42.5, , , , , ] 2022-09-09T22:39:28.564 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000011.40.5, , 00000000004.13.5, , 00000000008.42.5, , , , , ] 2022-09-09T22:39:28.565 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:39:28.563+0000 7fbd0ef11dc0 0 bucket checkpoint complete 2022-09-09T22:39:28.588 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jkcgbj-17 2022-09-09T22:39:28.589 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:28.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:28.590 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:39:28.644 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:39:28.645 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:39:28.706 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:39:28.699+0000 7fdb17494dc0 1 bucket sync caught up with source: 2022-09-09T22:39:28.706 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000011.40.5, , 00000000004.13.5, , 00000000008.42.5, , , , , ] 2022-09-09T22:39:28.706 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000011.40.5, , 00000000004.13.5, , 00000000008.42.5, , , , , ] 2022-09-09T22:39:28.707 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:39:28.699+0000 7fdb17494dc0 0 bucket checkpoint complete 2022-09-09T22:39:28.730 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jkcgbj-17 2022-09-09T22:39:28.731 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:28.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:28.732 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:39:28.787 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:39:28.788 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:39:28.868 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:39:28.867+0000 7fcb17812dc0 1 bucket sync caught up with source: 2022-09-09T22:39:28.868 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000011.40.5, , 00000000004.13.5, , 00000000008.42.5, , , , , ] 2022-09-09T22:39:28.869 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000011.40.5, , 00000000004.13.5, , 00000000008.42.5, , , , , ] 2022-09-09T22:39:28.869 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:39:28.867+0000 7fcb17812dc0 0 bucket checkpoint complete 2022-09-09T22:39:28.902 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jkcgbj-17 2022-09-09T22:39:28.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:28.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:28.904 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:39:28.959 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:39:28.960 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:39:29.029 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:39:29.023+0000 7fcb0a0fedc0 1 bucket sync caught up with source: 2022-09-09T22:39:29.030 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000004.43.5, , 00000000011.42.5, , 00000000009.41.5, , , , , ] 2022-09-09T22:39:29.030 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000004.43.5, , 00000000011.42.5, , 00000000009.41.5, , , , , ] 2022-09-09T22:39:29.030 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:39:29.023+0000 7fcb0a0fedc0 0 bucket checkpoint complete 2022-09-09T22:39:29.057 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jkcgbj-17 2022-09-09T22:39:29.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:29.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:29.059 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:39:29.112 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:39:29.112 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:39:29.176 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:39:29.171+0000 7f89f97c1dc0 1 bucket sync caught up with source: 2022-09-09T22:39:29.177 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000004.43.5, , 00000000011.42.5, , 00000000009.41.5, , , , , ] 2022-09-09T22:39:29.177 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000004.43.5, , 00000000011.42.5, , 00000000009.41.5, , , , , ] 2022-09-09T22:39:29.177 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:39:29.171+0000 7f89f97c1dc0 0 bucket checkpoint complete 2022-09-09T22:39:29.204 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jkcgbj-17 2022-09-09T22:39:29.204 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jkcgbj-17 2022-09-09T22:39:29.205 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:29.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:29.206 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:39:29.261 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:39:29.262 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:39:29.327 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:39:29.319+0000 7f8affef5dc0 1 bucket sync caught up with source: 2022-09-09T22:39:29.328 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000004.56.5, , 00000000004.26.5, , 00000000015.54.5, , , , , ] 2022-09-09T22:39:29.328 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000004.56.5, , 00000000004.26.5, , 00000000015.54.5, , , , , ] 2022-09-09T22:39:29.328 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:39:29.319+0000 7f8affef5dc0 0 bucket checkpoint complete 2022-09-09T22:39:29.352 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jkcgbj-17 2022-09-09T22:39:29.353 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:29.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:29.354 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:39:29.410 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:39:29.411 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:39:29.493 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:39:29.491+0000 7f485ef6adc0 1 bucket sync caught up with source: 2022-09-09T22:39:29.493 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000004.56.5, , 00000000004.26.5, , 00000000015.54.5, , , , , ] 2022-09-09T22:39:29.493 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000004.56.5, , 00000000004.26.5, , 00000000015.54.5, , , , , ] 2022-09-09T22:39:29.494 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:39:29.491+0000 7f485ef6adc0 0 bucket checkpoint complete 2022-09-09T22:39:29.524 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jkcgbj-17 2022-09-09T22:39:29.524 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-17 zones={test-zone2, test-zone1} 2022-09-09T22:39:29.525 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:29.525 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:29.526 DEBUG:boto:Method: HEAD 2022-09-09T22:39:29.526 DEBUG:boto:Path: /jkcgbj-17/ 2022-09-09T22:39:29.527 DEBUG:boto:Data: 2022-09-09T22:39:29.528 DEBUG:boto:Headers: {} 2022-09-09T22:39:29.528 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:29.529 DEBUG:boto:Port: 8000 2022-09-09T22:39:29.529 DEBUG:boto:Params: {} 2022-09-09T22:39:29.530 DEBUG:boto:Token: None 2022-09-09T22:39:29.531 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:29 GMT /jkcgbj-17/ 2022-09-09T22:39:29.531 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:kXfpCr8ytjTyG2Do2HACB/NGFKU= 2022-09-09T22:39:29.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:kXfpCr8ytjTyG2Do2HACB/NGFKU=', 'Content-Length': '0'} 2022-09-09T22:39:29.536 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d6e523a15f7b5e19-00631bc0a1-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:29.537 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:29.537 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:29.538 DEBUG:boto:Method: HEAD 2022-09-09T22:39:29.538 DEBUG:boto:Path: /jkcgbj-17/ 2022-09-09T22:39:29.539 DEBUG:boto:Data: 2022-09-09T22:39:29.539 DEBUG:boto:Headers: {} 2022-09-09T22:39:29.540 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:29.540 DEBUG:boto:Port: 8000 2022-09-09T22:39:29.541 DEBUG:boto:Params: {} 2022-09-09T22:39:29.542 DEBUG:boto:Token: None 2022-09-09T22:39:29.542 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:29 GMT /jkcgbj-17/ 2022-09-09T22:39:29.543 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:kXfpCr8ytjTyG2Do2HACB/NGFKU= 2022-09-09T22:39:29.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:kXfpCr8ytjTyG2Do2HACB/NGFKU=', 'Content-Length': '0'} 2022-09-09T22:39:29.548 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c946b2865bcdcd53-00631bc0a1-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:29.549 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:39:29.550 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:29.550 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:29.551 DEBUG:boto:path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:29.551 DEBUG:boto:auth_path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:29.552 DEBUG:boto:Method: GET 2022-09-09T22:39:29.552 DEBUG:boto:Path: /jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:29.552 DEBUG:boto:Data: 2022-09-09T22:39:29.553 DEBUG:boto:Headers: {} 2022-09-09T22:39:29.553 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:29.554 DEBUG:boto:Port: 8000 2022-09-09T22:39:29.554 DEBUG:boto:Params: {} 2022-09-09T22:39:29.555 DEBUG:boto:Token: None 2022-09-09T22:39:29.556 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:29 GMT /jkcgbj-17/?versions 2022-09-09T22:39:29.556 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:RIja3ccsKvjR4id+2Lro2ilOK08= 2022-09-09T22:39:29.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:RIja3ccsKvjR4id+2Lro2ilOK08=', 'Content-Length': '0'} 2022-09-09T22:39:29.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069d469c649889a07-00631bc0a1-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:29.563 DEBUG:boto:b'jkcgbj-17999falsedummyrcXzglYDyJxE.ArBmS3k944dyN-s9ZTtrue2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1uPhFM-4pEeqszoz0jfwkNTqs81x9txytrue2022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone217lUOlN45NLK2DSLxrYnqbOAg6jm2Tgtrue2022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone31RXW239HrAmSBD6KjNRB3qRTyY29OqZtrue2022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:29.563 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:39:29.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:39:29.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:39:29.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:39:29.566 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:39:29.566 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:29.567 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:29.567 DEBUG:boto:path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:29.568 DEBUG:boto:auth_path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:29.568 DEBUG:boto:Method: GET 2022-09-09T22:39:29.569 DEBUG:boto:Path: /jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:29.569 DEBUG:boto:Data: 2022-09-09T22:39:29.570 DEBUG:boto:Headers: {} 2022-09-09T22:39:29.571 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:29.571 DEBUG:boto:Port: 8000 2022-09-09T22:39:29.572 DEBUG:boto:Params: {} 2022-09-09T22:39:29.572 DEBUG:boto:Token: None 2022-09-09T22:39:29.573 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:29 GMT /jkcgbj-17/?versions 2022-09-09T22:39:29.573 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:RIja3ccsKvjR4id+2Lro2ilOK08= 2022-09-09T22:39:29.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:RIja3ccsKvjR4id+2Lro2ilOK08=', 'Content-Length': '0'} 2022-09-09T22:39:29.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000116aca46ae60df64-00631bc0a1-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:29.581 DEBUG:boto:b'jkcgbj-17999falsedummyrcXzglYDyJxE.ArBmS3k944dyN-s9ZTtrue2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1uPhFM-4pEeqszoz0jfwkNTqs81x9txytrue2022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone217lUOlN45NLK2DSLxrYnqbOAg6jm2Tgtrue2022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone31RXW239HrAmSBD6KjNRB3qRTyY29OqZtrue2022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:29.582 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:39:29.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:39:29.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:39:29.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:39:29.584 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:29.585 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:29.586 DEBUG:boto:path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:29.586 DEBUG:boto:auth_path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:29.587 DEBUG:boto:Method: GET 2022-09-09T22:39:29.587 DEBUG:boto:Path: /jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:29.588 DEBUG:boto:Data: 2022-09-09T22:39:29.588 DEBUG:boto:Headers: {} 2022-09-09T22:39:29.589 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:29.589 DEBUG:boto:Port: 8000 2022-09-09T22:39:29.590 DEBUG:boto:Params: {} 2022-09-09T22:39:29.590 DEBUG:boto:Token: None 2022-09-09T22:39:29.591 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:29 GMT /jkcgbj-17/?versions 2022-09-09T22:39:29.591 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:RIja3ccsKvjR4id+2Lro2ilOK08= 2022-09-09T22:39:29.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:RIja3ccsKvjR4id+2Lro2ilOK08=', 'Content-Length': '0'} 2022-09-09T22:39:29.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004191d44c93cbcd77-00631bc0a1-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:29.598 DEBUG:boto:b'jkcgbj-17999falsedummyrcXzglYDyJxE.ArBmS3k944dyN-s9ZTtrue2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1uPhFM-4pEeqszoz0jfwkNTqs81x9txytrue2022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone217lUOlN45NLK2DSLxrYnqbOAg6jm2Tgtrue2022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone31RXW239HrAmSBD6KjNRB3qRTyY29OqZtrue2022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:29.599 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:29.600 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:29.600 DEBUG:boto:path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:29.601 DEBUG:boto:auth_path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:29.601 DEBUG:boto:Method: GET 2022-09-09T22:39:29.602 DEBUG:boto:Path: /jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:29.602 DEBUG:boto:Data: 2022-09-09T22:39:29.603 DEBUG:boto:Headers: {} 2022-09-09T22:39:29.603 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:29.604 DEBUG:boto:Port: 8000 2022-09-09T22:39:29.604 DEBUG:boto:Params: {} 2022-09-09T22:39:29.605 DEBUG:boto:Token: None 2022-09-09T22:39:29.605 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:29 GMT /jkcgbj-17/?versions 2022-09-09T22:39:29.606 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:RIja3ccsKvjR4id+2Lro2ilOK08= 2022-09-09T22:39:29.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:RIja3ccsKvjR4id+2Lro2ilOK08=', 'Content-Length': '0'} 2022-09-09T22:39:29.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004eac2b71d8013c3f-00631bc0a1-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:29.613 DEBUG:boto:b'jkcgbj-17999falsedummyrcXzglYDyJxE.ArBmS3k944dyN-s9ZTtrue2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1uPhFM-4pEeqszoz0jfwkNTqs81x9txytrue2022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone217lUOlN45NLK2DSLxrYnqbOAg6jm2Tgtrue2022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone31RXW239HrAmSBD6KjNRB3qRTyY29OqZtrue2022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:29.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:39:29.614 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:39:29.615 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:39:29.615 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.616 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.616 DEBUG:boto:Method: GET 2022-09-09T22:39:29.617 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.617 DEBUG:boto:Data: 2022-09-09T22:39:29.618 DEBUG:boto:Headers: {} 2022-09-09T22:39:29.618 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:29.619 DEBUG:boto:Port: 8000 2022-09-09T22:39:29.619 DEBUG:boto:Params: {} 2022-09-09T22:39:29.620 DEBUG:boto:Token: None 2022-09-09T22:39:29.621 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:29 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.621 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0ZNRX9tM5RJELFcaCMg2y/CGbq4= 2022-09-09T22:39:29.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0ZNRX9tM5RJELFcaCMg2y/CGbq4=', 'Content-Length': '0'} 2022-09-09T22:39:29.625 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009cd671a3922631d3-00631bc0a1-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:29.625 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:39:29.626 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:39:29.627 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.627 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.628 DEBUG:boto:Method: GET 2022-09-09T22:39:29.628 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.629 DEBUG:boto:Data: 2022-09-09T22:39:29.629 DEBUG:boto:Headers: {} 2022-09-09T22:39:29.630 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:29.631 DEBUG:boto:Port: 8000 2022-09-09T22:39:29.631 DEBUG:boto:Params: {} 2022-09-09T22:39:29.632 DEBUG:boto:Token: None 2022-09-09T22:39:29.632 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:29 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.633 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0ZNRX9tM5RJELFcaCMg2y/CGbq4= 2022-09-09T22:39:29.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0ZNRX9tM5RJELFcaCMg2y/CGbq4=', 'Content-Length': '0'} 2022-09-09T22:39:29.637 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000001cc50d4126de85-00631bc0a1-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:29.638 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:39:29.638 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:39:29.639 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.639 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.640 DEBUG:boto:Method: HEAD 2022-09-09T22:39:29.640 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.641 DEBUG:boto:Data: 2022-09-09T22:39:29.641 DEBUG:boto:Headers: {} 2022-09-09T22:39:29.642 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:29.642 DEBUG:boto:Port: 8000 2022-09-09T22:39:29.643 DEBUG:boto:Params: {} 2022-09-09T22:39:29.643 DEBUG:boto:Token: None 2022-09-09T22:39:29.644 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:29 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.645 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sHCAk6OO9b/bJU3kfQeNKHnlDh0= 2022-09-09T22:39:29.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sHCAk6OO9b/bJU3kfQeNKHnlDh0=', 'Content-Length': '0'} 2022-09-09T22:39:29.648 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000abc0ceef4fa2db72-00631bc0a1-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:29.649 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:39:29.649 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:39:29.650 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.651 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.651 DEBUG:boto:Method: HEAD 2022-09-09T22:39:29.652 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.652 DEBUG:boto:Data: 2022-09-09T22:39:29.652 DEBUG:boto:Headers: {} 2022-09-09T22:39:29.653 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:29.653 DEBUG:boto:Port: 8000 2022-09-09T22:39:29.654 DEBUG:boto:Params: {} 2022-09-09T22:39:29.654 DEBUG:boto:Token: None 2022-09-09T22:39:29.655 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:29 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.655 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sHCAk6OO9b/bJU3kfQeNKHnlDh0= 2022-09-09T22:39:29.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sHCAk6OO9b/bJU3kfQeNKHnlDh0=', 'Content-Length': '0'} 2022-09-09T22:39:29.658 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000eba0662730a74019-00631bc0a1-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:29.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:39:29.660 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:39:29.660 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:39:29.661 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.662 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.662 DEBUG:boto:Method: GET 2022-09-09T22:39:29.663 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.663 DEBUG:boto:Data: 2022-09-09T22:39:29.664 DEBUG:boto:Headers: {} 2022-09-09T22:39:29.665 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:29.665 DEBUG:boto:Port: 8000 2022-09-09T22:39:29.666 DEBUG:boto:Params: {} 2022-09-09T22:39:29.666 DEBUG:boto:Token: None 2022-09-09T22:39:29.667 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:29 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.668 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0ZNRX9tM5RJELFcaCMg2y/CGbq4= 2022-09-09T22:39:29.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0ZNRX9tM5RJELFcaCMg2y/CGbq4=', 'Content-Length': '0'} 2022-09-09T22:39:29.671 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000014d8ef56092e87e4-00631bc0a1-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:29.672 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:39:29.673 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:39:29.673 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.674 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.674 DEBUG:boto:Method: GET 2022-09-09T22:39:29.675 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.675 DEBUG:boto:Data: 2022-09-09T22:39:29.676 DEBUG:boto:Headers: {} 2022-09-09T22:39:29.676 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:29.677 DEBUG:boto:Port: 8000 2022-09-09T22:39:29.678 DEBUG:boto:Params: {} 2022-09-09T22:39:29.678 DEBUG:boto:Token: None 2022-09-09T22:39:29.679 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:29 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.679 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0ZNRX9tM5RJELFcaCMg2y/CGbq4= 2022-09-09T22:39:29.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0ZNRX9tM5RJELFcaCMg2y/CGbq4=', 'Content-Length': '0'} 2022-09-09T22:39:29.683 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c1e36d20ec988442-00631bc0a1-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:29.684 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:29.685 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:29.686 DEBUG:boto:path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:39:29.686 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:39:29.687 DEBUG:boto:Method: GET 2022-09-09T22:39:29.687 DEBUG:boto:Path: /jkcgbj-17/?prefix=dummy 2022-09-09T22:39:29.688 DEBUG:boto:Data: 2022-09-09T22:39:29.688 DEBUG:boto:Headers: {} 2022-09-09T22:39:29.689 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:29.689 DEBUG:boto:Port: 8000 2022-09-09T22:39:29.690 DEBUG:boto:Params: {} 2022-09-09T22:39:29.690 DEBUG:boto:Token: None 2022-09-09T22:39:29.691 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:29 GMT /jkcgbj-17/ 2022-09-09T22:39:29.692 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:zRysgiE/5QchHfmpla4rybupLb4= 2022-09-09T22:39:29.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:zRysgiE/5QchHfmpla4rybupLb4=', 'Content-Length': '0'} 2022-09-09T22:39:29.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bd3ab875bf649dc-00631bc0a1-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:29.738 DEBUG:boto:b'jkcgbj-17dummy1000falsedummy2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:29.739 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:29.740 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:29.740 DEBUG:boto:path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:39:29.741 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:39:29.741 DEBUG:boto:Method: GET 2022-09-09T22:39:29.742 DEBUG:boto:Path: /jkcgbj-17/?prefix=dummy 2022-09-09T22:39:29.742 DEBUG:boto:Data: 2022-09-09T22:39:29.743 DEBUG:boto:Headers: {} 2022-09-09T22:39:29.744 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:29.744 DEBUG:boto:Port: 8000 2022-09-09T22:39:29.745 DEBUG:boto:Params: {} 2022-09-09T22:39:29.745 DEBUG:boto:Token: None 2022-09-09T22:39:29.746 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:29 GMT /jkcgbj-17/ 2022-09-09T22:39:29.746 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:zRysgiE/5QchHfmpla4rybupLb4= 2022-09-09T22:39:29.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:zRysgiE/5QchHfmpla4rybupLb4=', 'Content-Length': '0'} 2022-09-09T22:39:29.753 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8ca09c6ed6b06f8-00631bc0a1-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:29.754 DEBUG:boto:b'jkcgbj-17dummy1000falsedummy2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:29.755 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:39:29.755 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:39:29.756 DEBUG:boto:Method: HEAD 2022-09-09T22:39:29.756 DEBUG:boto:Path: /jkcgbj-17/dummy 2022-09-09T22:39:29.757 DEBUG:boto:Data: 2022-09-09T22:39:29.757 DEBUG:boto:Headers: {} 2022-09-09T22:39:29.758 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:29.758 DEBUG:boto:Port: 8000 2022-09-09T22:39:29.759 DEBUG:boto:Params: {} 2022-09-09T22:39:29.760 DEBUG:boto:Token: None 2022-09-09T22:39:29.760 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:29 GMT /jkcgbj-17/dummy 2022-09-09T22:39:29.761 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9C0xIsPCJgGkjHX1q+0bxZmP4TI= 2022-09-09T22:39:29.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9C0xIsPCJgGkjHX1q+0bxZmP4TI=', 'Content-Length': '0'} 2022-09-09T22:39:29.765 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f3585e5a8fd76167-00631bc0a1-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:29.766 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:39:29.766 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:39:29.767 DEBUG:boto:Method: HEAD 2022-09-09T22:39:29.767 DEBUG:boto:Path: /jkcgbj-17/dummy 2022-09-09T22:39:29.768 DEBUG:boto:Data: 2022-09-09T22:39:29.768 DEBUG:boto:Headers: {} 2022-09-09T22:39:29.769 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:29.769 DEBUG:boto:Port: 8000 2022-09-09T22:39:29.770 DEBUG:boto:Params: {} 2022-09-09T22:39:29.771 DEBUG:boto:Token: None 2022-09-09T22:39:29.771 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:29 GMT /jkcgbj-17/dummy 2022-09-09T22:39:29.772 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9C0xIsPCJgGkjHX1q+0bxZmP4TI= 2022-09-09T22:39:29.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9C0xIsPCJgGkjHX1q+0bxZmP4TI=', 'Content-Length': '0'} 2022-09-09T22:39:29.776 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000066b9d146c7d2da9-00631bc0a1-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:29.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:39:29.777 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:39:29.778 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:39:29.778 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.779 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.779 DEBUG:boto:Method: GET 2022-09-09T22:39:29.780 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.781 DEBUG:boto:Data: 2022-09-09T22:39:29.781 DEBUG:boto:Headers: {} 2022-09-09T22:39:29.782 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:29.782 DEBUG:boto:Port: 8000 2022-09-09T22:39:29.783 DEBUG:boto:Params: {} 2022-09-09T22:39:29.783 DEBUG:boto:Token: None 2022-09-09T22:39:29.784 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:29 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.784 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0ZNRX9tM5RJELFcaCMg2y/CGbq4= 2022-09-09T22:39:29.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0ZNRX9tM5RJELFcaCMg2y/CGbq4=', 'Content-Length': '0'} 2022-09-09T22:39:29.788 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a3b549a59719af73-00631bc0a1-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:29.789 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:39:29.789 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:39:29.790 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.790 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.791 DEBUG:boto:Method: GET 2022-09-09T22:39:29.792 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.792 DEBUG:boto:Data: 2022-09-09T22:39:29.793 DEBUG:boto:Headers: {} 2022-09-09T22:39:29.793 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:29.794 DEBUG:boto:Port: 8000 2022-09-09T22:39:29.794 DEBUG:boto:Params: {} 2022-09-09T22:39:29.795 DEBUG:boto:Token: None 2022-09-09T22:39:29.795 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:29 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:29.796 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0ZNRX9tM5RJELFcaCMg2y/CGbq4= 2022-09-09T22:39:29.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0ZNRX9tM5RJELFcaCMg2y/CGbq4=', 'Content-Length': '0'} 2022-09-09T22:39:29.799 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000463ab80fd92a719c-00631bc0a1-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:29.800 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:29.801 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:29.801 DEBUG:boto:path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:39:29.802 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:39:29.802 DEBUG:boto:Method: GET 2022-09-09T22:39:29.803 DEBUG:boto:Path: /jkcgbj-17/?prefix=dummy 2022-09-09T22:39:29.803 DEBUG:boto:Data: 2022-09-09T22:39:29.804 DEBUG:boto:Headers: {} 2022-09-09T22:39:29.805 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:29.805 DEBUG:boto:Port: 8000 2022-09-09T22:39:29.806 DEBUG:boto:Params: {} 2022-09-09T22:39:29.806 DEBUG:boto:Token: None 2022-09-09T22:39:29.807 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:29 GMT /jkcgbj-17/ 2022-09-09T22:39:29.807 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:zRysgiE/5QchHfmpla4rybupLb4= 2022-09-09T22:39:29.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:zRysgiE/5QchHfmpla4rybupLb4=', 'Content-Length': '0'} 2022-09-09T22:39:29.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000244bce4748789e0c-00631bc0a1-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:29.814 DEBUG:boto:b'jkcgbj-17dummy1000falsedummy2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:29.816 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:29.816 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:29.817 DEBUG:boto:path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:39:29.817 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:39:29.818 DEBUG:boto:Method: GET 2022-09-09T22:39:29.819 DEBUG:boto:Path: /jkcgbj-17/?prefix=dummy 2022-09-09T22:39:29.819 DEBUG:boto:Data: 2022-09-09T22:39:29.820 DEBUG:boto:Headers: {} 2022-09-09T22:39:29.820 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:29.821 DEBUG:boto:Port: 8000 2022-09-09T22:39:29.821 DEBUG:boto:Params: {} 2022-09-09T22:39:29.822 DEBUG:boto:Token: None 2022-09-09T22:39:29.822 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:29 GMT /jkcgbj-17/ 2022-09-09T22:39:29.823 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:zRysgiE/5QchHfmpla4rybupLb4= 2022-09-09T22:39:29.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:zRysgiE/5QchHfmpla4rybupLb4=', 'Content-Length': '0'} 2022-09-09T22:39:29.829 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c5aa6bbb586d57d-00631bc0a1-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:29.870 DEBUG:boto:b'jkcgbj-17dummy1000falsedummy2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:29.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:39:29.871 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:29.872 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:29.872 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:29.873 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:29.874 DEBUG:boto:Method: GET 2022-09-09T22:39:29.874 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:29.875 DEBUG:boto:Data: 2022-09-09T22:39:29.875 DEBUG:boto:Headers: {} 2022-09-09T22:39:29.876 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:29.876 DEBUG:boto:Port: 8000 2022-09-09T22:39:29.877 DEBUG:boto:Params: {} 2022-09-09T22:39:29.877 DEBUG:boto:Token: None 2022-09-09T22:39:29.878 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:29 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:29.879 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:IMvoT/9p1UwK3Tk50J+pA2HHsA8= 2022-09-09T22:39:29.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:IMvoT/9p1UwK3Tk50J+pA2HHsA8=', 'Content-Length': '0'} 2022-09-09T22:39:29.882 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000685f055b4b36d09a-00631bc0a1-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:29.883 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:29.884 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:29.884 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:29.885 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:29.885 DEBUG:boto:Method: GET 2022-09-09T22:39:29.886 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:29.886 DEBUG:boto:Data: 2022-09-09T22:39:29.887 DEBUG:boto:Headers: {} 2022-09-09T22:39:29.888 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:29.888 DEBUG:boto:Port: 8000 2022-09-09T22:39:29.889 DEBUG:boto:Params: {} 2022-09-09T22:39:29.889 DEBUG:boto:Token: None 2022-09-09T22:39:29.890 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:29 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:29.890 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:IMvoT/9p1UwK3Tk50J+pA2HHsA8= 2022-09-09T22:39:29.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:IMvoT/9p1UwK3Tk50J+pA2HHsA8=', 'Content-Length': '0'} 2022-09-09T22:39:29.894 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000002bd7452bd4a1bce-00631bc0a1-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:29.895 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:29.895 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:29.896 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:29.896 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:29.897 DEBUG:boto:Method: HEAD 2022-09-09T22:39:29.897 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:29.898 DEBUG:boto:Data: 2022-09-09T22:39:29.898 DEBUG:boto:Headers: {} 2022-09-09T22:39:29.899 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:29.899 DEBUG:boto:Port: 8000 2022-09-09T22:39:29.899 DEBUG:boto:Params: {} 2022-09-09T22:39:29.900 DEBUG:boto:Token: None 2022-09-09T22:39:29.900 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:29 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:29.901 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4/mwfa6d6UJJwZ3xn91q0GNfM38= 2022-09-09T22:39:29.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4/mwfa6d6UJJwZ3xn91q0GNfM38=', 'Content-Length': '0'} 2022-09-09T22:39:29.904 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7e778a29ba6d04c-00631bc0a1-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:29.905 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:29.906 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:29.906 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:29.907 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:29.907 DEBUG:boto:Method: HEAD 2022-09-09T22:39:29.907 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:29.908 DEBUG:boto:Data: 2022-09-09T22:39:29.908 DEBUG:boto:Headers: {} 2022-09-09T22:39:29.909 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:29.909 DEBUG:boto:Port: 8000 2022-09-09T22:39:29.910 DEBUG:boto:Params: {} 2022-09-09T22:39:29.910 DEBUG:boto:Token: None 2022-09-09T22:39:29.911 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:29 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:29.912 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4/mwfa6d6UJJwZ3xn91q0GNfM38= 2022-09-09T22:39:29.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4/mwfa6d6UJJwZ3xn91q0GNfM38=', 'Content-Length': '0'} 2022-09-09T22:39:29.915 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017dd033ffe8d0aa0-00631bc0a1-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:29.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:39:29.916 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:29.917 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:29.918 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:29.918 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:29.919 DEBUG:boto:Method: GET 2022-09-09T22:39:29.919 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:29.920 DEBUG:boto:Data: 2022-09-09T22:39:29.920 DEBUG:boto:Headers: {} 2022-09-09T22:39:29.921 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:29.921 DEBUG:boto:Port: 8000 2022-09-09T22:39:29.922 DEBUG:boto:Params: {} 2022-09-09T22:39:29.922 DEBUG:boto:Token: None 2022-09-09T22:39:29.923 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:29 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:29.923 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:IMvoT/9p1UwK3Tk50J+pA2HHsA8= 2022-09-09T22:39:29.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:IMvoT/9p1UwK3Tk50J+pA2HHsA8=', 'Content-Length': '0'} 2022-09-09T22:39:29.927 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df98c8c1d0e8c3eb-00631bc0a1-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:29.928 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:29.929 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:29.929 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:29.930 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:29.930 DEBUG:boto:Method: GET 2022-09-09T22:39:29.931 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:29.931 DEBUG:boto:Data: 2022-09-09T22:39:29.932 DEBUG:boto:Headers: {} 2022-09-09T22:39:29.932 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:29.933 DEBUG:boto:Port: 8000 2022-09-09T22:39:29.933 DEBUG:boto:Params: {} 2022-09-09T22:39:29.934 DEBUG:boto:Token: None 2022-09-09T22:39:29.935 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:29 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:29.935 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:IMvoT/9p1UwK3Tk50J+pA2HHsA8= 2022-09-09T22:39:29.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:IMvoT/9p1UwK3Tk50J+pA2HHsA8=', 'Content-Length': '0'} 2022-09-09T22:39:29.939 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060deb24bf59dd7ed-00631bc0a1-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:29.982 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:29.983 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:29.983 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:29.984 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:29.985 DEBUG:boto:Method: GET 2022-09-09T22:39:29.985 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:29.986 DEBUG:boto:Data: 2022-09-09T22:39:29.986 DEBUG:boto:Headers: {} 2022-09-09T22:39:29.987 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:29.987 DEBUG:boto:Port: 8000 2022-09-09T22:39:29.988 DEBUG:boto:Params: {} 2022-09-09T22:39:29.989 DEBUG:boto:Token: None 2022-09-09T22:39:29.989 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:29 GMT /jkcgbj-17/ 2022-09-09T22:39:29.990 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:zRysgiE/5QchHfmpla4rybupLb4= 2022-09-09T22:39:29.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:zRysgiE/5QchHfmpla4rybupLb4=', 'Content-Length': '0'} 2022-09-09T22:39:29.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5622cd90b57bf26-00631bc0a1-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:29.998 DEBUG:boto:b'jkcgbj-17obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:29.999 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:29.999 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:30.000 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:30.000 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:30.001 DEBUG:boto:Method: GET 2022-09-09T22:39:30.001 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:30.002 DEBUG:boto:Data: 2022-09-09T22:39:30.002 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.003 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:30.004 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.004 DEBUG:boto:Params: {} 2022-09-09T22:39:30.005 DEBUG:boto:Token: None 2022-09-09T22:39:30.005 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/ 2022-09-09T22:39:30.006 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UrNNRpjk5Mb3zByWoS0BU/b2kaw= 2022-09-09T22:39:30.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UrNNRpjk5Mb3zByWoS0BU/b2kaw=', 'Content-Length': '0'} 2022-09-09T22:39:30.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000761207e5db42bbd6-00631bc0a2-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.014 DEBUG:boto:b'jkcgbj-17obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:30.015 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:30.015 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:30.016 DEBUG:boto:Method: HEAD 2022-09-09T22:39:30.016 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1 2022-09-09T22:39:30.017 DEBUG:boto:Data: 2022-09-09T22:39:30.017 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.018 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:30.018 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.019 DEBUG:boto:Params: {} 2022-09-09T22:39:30.019 DEBUG:boto:Token: None 2022-09-09T22:39:30.020 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/obj-test-zone1 2022-09-09T22:39:30.021 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:k+9ut90ssA/udbccJ3+8DG7yLAg= 2022-09-09T22:39:30.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:k+9ut90ssA/udbccJ3+8DG7yLAg=', 'Content-Length': '0'} 2022-09-09T22:39:30.025 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006564244a8d864d47-00631bc0a2-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.026 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:30.026 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:30.027 DEBUG:boto:Method: HEAD 2022-09-09T22:39:30.028 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1 2022-09-09T22:39:30.028 DEBUG:boto:Data: 2022-09-09T22:39:30.029 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.029 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:30.030 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.030 DEBUG:boto:Params: {} 2022-09-09T22:39:30.031 DEBUG:boto:Token: None 2022-09-09T22:39:30.031 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/obj-test-zone1 2022-09-09T22:39:30.032 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:k+9ut90ssA/udbccJ3+8DG7yLAg= 2022-09-09T22:39:30.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:k+9ut90ssA/udbccJ3+8DG7yLAg=', 'Content-Length': '0'} 2022-09-09T22:39:30.037 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f103927a7f00d04-00631bc0a2-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:39:30.038 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:30.039 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:30.039 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:30.040 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:30.040 DEBUG:boto:Method: GET 2022-09-09T22:39:30.041 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:30.041 DEBUG:boto:Data: 2022-09-09T22:39:30.042 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.042 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:30.043 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.043 DEBUG:boto:Params: {} 2022-09-09T22:39:30.044 DEBUG:boto:Token: None 2022-09-09T22:39:30.045 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:30.045 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:GhOqN5otK8T04M9wxqcg7pQFsLw= 2022-09-09T22:39:30.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:GhOqN5otK8T04M9wxqcg7pQFsLw=', 'Content-Length': '0'} 2022-09-09T22:39:30.049 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008268650e1de5b15a-00631bc0a2-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.090 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:30.091 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:30.091 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:30.092 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:30.092 DEBUG:boto:Method: GET 2022-09-09T22:39:30.093 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:30.094 DEBUG:boto:Data: 2022-09-09T22:39:30.094 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.095 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:30.095 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.096 DEBUG:boto:Params: {} 2022-09-09T22:39:30.096 DEBUG:boto:Token: None 2022-09-09T22:39:30.097 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:30.097 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:GhOqN5otK8T04M9wxqcg7pQFsLw= 2022-09-09T22:39:30.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:GhOqN5otK8T04M9wxqcg7pQFsLw=', 'Content-Length': '0'} 2022-09-09T22:39:30.101 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab1b6b147dbb2e20-00631bc0a2-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.102 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:30.103 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:30.103 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:30.104 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:30.104 DEBUG:boto:Method: GET 2022-09-09T22:39:30.105 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:30.105 DEBUG:boto:Data: 2022-09-09T22:39:30.106 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.106 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:30.107 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.107 DEBUG:boto:Params: {} 2022-09-09T22:39:30.108 DEBUG:boto:Token: None 2022-09-09T22:39:30.109 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/ 2022-09-09T22:39:30.109 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UrNNRpjk5Mb3zByWoS0BU/b2kaw= 2022-09-09T22:39:30.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UrNNRpjk5Mb3zByWoS0BU/b2kaw=', 'Content-Length': '0'} 2022-09-09T22:39:30.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054d7f66e89892510-00631bc0a2-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.116 DEBUG:boto:b'jkcgbj-17obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:30.117 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:30.117 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:30.118 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:30.118 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:30.119 DEBUG:boto:Method: GET 2022-09-09T22:39:30.119 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:30.120 DEBUG:boto:Data: 2022-09-09T22:39:30.120 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.121 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:30.121 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.122 DEBUG:boto:Params: {} 2022-09-09T22:39:30.122 DEBUG:boto:Token: None 2022-09-09T22:39:30.123 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/ 2022-09-09T22:39:30.123 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UrNNRpjk5Mb3zByWoS0BU/b2kaw= 2022-09-09T22:39:30.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UrNNRpjk5Mb3zByWoS0BU/b2kaw=', 'Content-Length': '0'} 2022-09-09T22:39:30.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091d0c6647be127a9-00631bc0a2-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.130 DEBUG:boto:b'jkcgbj-17obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:30.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:39:30.132 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:30.132 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:30.133 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.133 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.134 DEBUG:boto:Method: GET 2022-09-09T22:39:30.134 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.135 DEBUG:boto:Data: 2022-09-09T22:39:30.135 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.136 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:30.136 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.137 DEBUG:boto:Params: {} 2022-09-09T22:39:30.137 DEBUG:boto:Token: None 2022-09-09T22:39:30.138 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.138 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:gmYge2wqhEFmmaZQyfXlBKIe1Jw= 2022-09-09T22:39:30.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:gmYge2wqhEFmmaZQyfXlBKIe1Jw=', 'Content-Length': '0'} 2022-09-09T22:39:30.142 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd5d26c9a7ca07f1-00631bc0a2-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.143 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:30.143 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:30.144 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.144 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.145 DEBUG:boto:Method: GET 2022-09-09T22:39:30.146 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.146 DEBUG:boto:Data: 2022-09-09T22:39:30.147 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.147 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:30.148 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.148 DEBUG:boto:Params: {} 2022-09-09T22:39:30.149 DEBUG:boto:Token: None 2022-09-09T22:39:30.149 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.150 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:gmYge2wqhEFmmaZQyfXlBKIe1Jw= 2022-09-09T22:39:30.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:gmYge2wqhEFmmaZQyfXlBKIe1Jw=', 'Content-Length': '0'} 2022-09-09T22:39:30.154 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac0adfaecb9db199-00631bc0a2-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.198 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:30.199 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:30.199 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.200 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.200 DEBUG:boto:Method: HEAD 2022-09-09T22:39:30.201 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.201 DEBUG:boto:Data: 2022-09-09T22:39:30.202 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.202 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:30.203 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.203 DEBUG:boto:Params: {} 2022-09-09T22:39:30.204 DEBUG:boto:Token: None 2022-09-09T22:39:30.204 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.205 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:tv1B1ksi5ghpXxG2xVf8j5DBwtI= 2022-09-09T22:39:30.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:tv1B1ksi5ghpXxG2xVf8j5DBwtI=', 'Content-Length': '0'} 2022-09-09T22:39:30.208 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000496bf18f75d193e3-00631bc0a2-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.209 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:30.210 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:30.210 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.211 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.211 DEBUG:boto:Method: HEAD 2022-09-09T22:39:30.211 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.212 DEBUG:boto:Data: 2022-09-09T22:39:30.212 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.213 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:30.213 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.214 DEBUG:boto:Params: {} 2022-09-09T22:39:30.215 DEBUG:boto:Token: None 2022-09-09T22:39:30.215 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.216 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:tv1B1ksi5ghpXxG2xVf8j5DBwtI= 2022-09-09T22:39:30.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:tv1B1ksi5ghpXxG2xVf8j5DBwtI=', 'Content-Length': '0'} 2022-09-09T22:39:30.219 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002aed9357fb134ec4-00631bc0a2-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:39:30.220 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:30.221 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:30.221 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.222 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.222 DEBUG:boto:Method: GET 2022-09-09T22:39:30.223 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.223 DEBUG:boto:Data: 2022-09-09T22:39:30.224 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.224 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:30.225 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.225 DEBUG:boto:Params: {} 2022-09-09T22:39:30.226 DEBUG:boto:Token: None 2022-09-09T22:39:30.227 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.227 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:gmYge2wqhEFmmaZQyfXlBKIe1Jw= 2022-09-09T22:39:30.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:gmYge2wqhEFmmaZQyfXlBKIe1Jw=', 'Content-Length': '0'} 2022-09-09T22:39:30.231 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009fee33bf1f586372-00631bc0a2-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.232 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:30.232 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:30.233 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.233 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.234 DEBUG:boto:Method: GET 2022-09-09T22:39:30.235 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.235 DEBUG:boto:Data: 2022-09-09T22:39:30.236 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.236 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:30.237 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.237 DEBUG:boto:Params: {} 2022-09-09T22:39:30.238 DEBUG:boto:Token: None 2022-09-09T22:39:30.238 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.239 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:gmYge2wqhEFmmaZQyfXlBKIe1Jw= 2022-09-09T22:39:30.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:gmYge2wqhEFmmaZQyfXlBKIe1Jw=', 'Content-Length': '0'} 2022-09-09T22:39:30.242 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c01c0333ddcaa3c-00631bc0a2-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.243 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:30.244 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:30.244 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:30.245 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:30.245 DEBUG:boto:Method: GET 2022-09-09T22:39:30.246 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:30.246 DEBUG:boto:Data: 2022-09-09T22:39:30.247 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.247 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:30.248 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.249 DEBUG:boto:Params: {} 2022-09-09T22:39:30.249 DEBUG:boto:Token: None 2022-09-09T22:39:30.250 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/ 2022-09-09T22:39:30.250 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UrNNRpjk5Mb3zByWoS0BU/b2kaw= 2022-09-09T22:39:30.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UrNNRpjk5Mb3zByWoS0BU/b2kaw=', 'Content-Length': '0'} 2022-09-09T22:39:30.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe4fcf4eda2b30d4-00631bc0a2-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.302 DEBUG:boto:b'jkcgbj-17obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:30.303 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:30.303 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:30.304 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:30.304 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:30.305 DEBUG:boto:Method: GET 2022-09-09T22:39:30.305 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:30.306 DEBUG:boto:Data: 2022-09-09T22:39:30.306 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.307 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:30.308 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.308 DEBUG:boto:Params: {} 2022-09-09T22:39:30.309 DEBUG:boto:Token: None 2022-09-09T22:39:30.309 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/ 2022-09-09T22:39:30.310 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UrNNRpjk5Mb3zByWoS0BU/b2kaw= 2022-09-09T22:39:30.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UrNNRpjk5Mb3zByWoS0BU/b2kaw=', 'Content-Length': '0'} 2022-09-09T22:39:30.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000781b13f4130b63f8-00631bc0a2-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.318 DEBUG:boto:b'jkcgbj-17obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:30.319 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:30.320 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:30.320 DEBUG:boto:Method: HEAD 2022-09-09T22:39:30.321 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2 2022-09-09T22:39:30.321 DEBUG:boto:Data: 2022-09-09T22:39:30.322 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.322 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:30.323 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.323 DEBUG:boto:Params: {} 2022-09-09T22:39:30.324 DEBUG:boto:Token: None 2022-09-09T22:39:30.324 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/obj-test-zone2 2022-09-09T22:39:30.325 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:aYPIb3EogGYBaieVwisc01bMz+U= 2022-09-09T22:39:30.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:aYPIb3EogGYBaieVwisc01bMz+U=', 'Content-Length': '0'} 2022-09-09T22:39:30.329 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca79e47ae72a5cbb-00631bc0a2-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.330 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:30.330 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:30.331 DEBUG:boto:Method: HEAD 2022-09-09T22:39:30.331 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2 2022-09-09T22:39:30.332 DEBUG:boto:Data: 2022-09-09T22:39:30.332 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.333 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:30.333 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.334 DEBUG:boto:Params: {} 2022-09-09T22:39:30.334 DEBUG:boto:Token: None 2022-09-09T22:39:30.335 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/obj-test-zone2 2022-09-09T22:39:30.335 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:aYPIb3EogGYBaieVwisc01bMz+U= 2022-09-09T22:39:30.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:aYPIb3EogGYBaieVwisc01bMz+U=', 'Content-Length': '0'} 2022-09-09T22:39:30.340 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084797d24762a2d7a-00631bc0a2-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:39:30.341 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:30.341 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:30.342 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.342 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.343 DEBUG:boto:Method: GET 2022-09-09T22:39:30.343 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.344 DEBUG:boto:Data: 2022-09-09T22:39:30.344 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.345 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:30.345 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.346 DEBUG:boto:Params: {} 2022-09-09T22:39:30.347 DEBUG:boto:Token: None 2022-09-09T22:39:30.347 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.348 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:gmYge2wqhEFmmaZQyfXlBKIe1Jw= 2022-09-09T22:39:30.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:gmYge2wqhEFmmaZQyfXlBKIe1Jw=', 'Content-Length': '0'} 2022-09-09T22:39:30.351 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000496ed21cef8ee54d-00631bc0a2-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.352 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:30.352 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:30.353 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.353 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.354 DEBUG:boto:Method: GET 2022-09-09T22:39:30.355 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.355 DEBUG:boto:Data: 2022-09-09T22:39:30.356 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.356 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:30.357 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.357 DEBUG:boto:Params: {} 2022-09-09T22:39:30.358 DEBUG:boto:Token: None 2022-09-09T22:39:30.358 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:30.359 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:gmYge2wqhEFmmaZQyfXlBKIe1Jw= 2022-09-09T22:39:30.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:gmYge2wqhEFmmaZQyfXlBKIe1Jw=', 'Content-Length': '0'} 2022-09-09T22:39:30.362 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000542533445bfe3c35-00631bc0a2-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.406 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:30.407 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:30.407 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:30.408 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:30.408 DEBUG:boto:Method: GET 2022-09-09T22:39:30.409 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:30.409 DEBUG:boto:Data: 2022-09-09T22:39:30.410 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.410 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:30.411 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.411 DEBUG:boto:Params: {} 2022-09-09T22:39:30.412 DEBUG:boto:Token: None 2022-09-09T22:39:30.413 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/ 2022-09-09T22:39:30.413 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UrNNRpjk5Mb3zByWoS0BU/b2kaw= 2022-09-09T22:39:30.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UrNNRpjk5Mb3zByWoS0BU/b2kaw=', 'Content-Length': '0'} 2022-09-09T22:39:30.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebf35411b85f05b7-00631bc0a2-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.420 DEBUG:boto:b'jkcgbj-17obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:30.421 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:30.421 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:30.422 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:30.422 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:30.423 DEBUG:boto:Method: GET 2022-09-09T22:39:30.423 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:30.424 DEBUG:boto:Data: 2022-09-09T22:39:30.424 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.425 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:30.425 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.426 DEBUG:boto:Params: {} 2022-09-09T22:39:30.426 DEBUG:boto:Token: None 2022-09-09T22:39:30.427 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/ 2022-09-09T22:39:30.427 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UrNNRpjk5Mb3zByWoS0BU/b2kaw= 2022-09-09T22:39:30.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UrNNRpjk5Mb3zByWoS0BU/b2kaw=', 'Content-Length': '0'} 2022-09-09T22:39:30.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037943cedec08231a-00631bc0a2-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.434 DEBUG:boto:b'jkcgbj-17obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:30.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:39:30.436 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:30.436 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:30.437 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.437 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.438 DEBUG:boto:Method: GET 2022-09-09T22:39:30.438 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.439 DEBUG:boto:Data: 2022-09-09T22:39:30.439 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.440 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:30.440 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.441 DEBUG:boto:Params: {} 2022-09-09T22:39:30.442 DEBUG:boto:Token: None 2022-09-09T22:39:30.442 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.443 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:AdABbPog6F27FbtXoENHN1qMFrI= 2022-09-09T22:39:30.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:AdABbPog6F27FbtXoENHN1qMFrI=', 'Content-Length': '0'} 2022-09-09T22:39:30.446 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000443e7e1a34990e1c-00631bc0a2-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.447 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:30.448 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:30.448 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.449 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.450 DEBUG:boto:Method: GET 2022-09-09T22:39:30.450 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.451 DEBUG:boto:Data: 2022-09-09T22:39:30.451 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.452 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:30.452 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.453 DEBUG:boto:Params: {} 2022-09-09T22:39:30.453 DEBUG:boto:Token: None 2022-09-09T22:39:30.454 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.455 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:AdABbPog6F27FbtXoENHN1qMFrI= 2022-09-09T22:39:30.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:AdABbPog6F27FbtXoENHN1qMFrI=', 'Content-Length': '0'} 2022-09-09T22:39:30.458 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe214e7f80cfc629-00631bc0a2-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.459 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:30.459 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:30.460 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.460 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.461 DEBUG:boto:Method: HEAD 2022-09-09T22:39:30.461 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.462 DEBUG:boto:Data: 2022-09-09T22:39:30.462 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.463 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:30.463 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.464 DEBUG:boto:Params: {} 2022-09-09T22:39:30.465 DEBUG:boto:Token: None 2022-09-09T22:39:30.465 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.466 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:k5+8/MYeAwHiuqDJQdHxglKMlWk= 2022-09-09T22:39:30.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:k5+8/MYeAwHiuqDJQdHxglKMlWk=', 'Content-Length': '0'} 2022-09-09T22:39:30.469 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a5c267e9e5613d35-00631bc0a2-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.470 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:30.470 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:30.471 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.471 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.472 DEBUG:boto:Method: HEAD 2022-09-09T22:39:30.472 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.473 DEBUG:boto:Data: 2022-09-09T22:39:30.474 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.474 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:30.475 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.475 DEBUG:boto:Params: {} 2022-09-09T22:39:30.476 DEBUG:boto:Token: None 2022-09-09T22:39:30.476 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.477 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:k5+8/MYeAwHiuqDJQdHxglKMlWk= 2022-09-09T22:39:30.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:k5+8/MYeAwHiuqDJQdHxglKMlWk=', 'Content-Length': '0'} 2022-09-09T22:39:30.480 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001d5d534c3e9a9962-00631bc0a2-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:39:30.482 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:30.482 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:30.483 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.483 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.484 DEBUG:boto:Method: GET 2022-09-09T22:39:30.485 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.485 DEBUG:boto:Data: 2022-09-09T22:39:30.486 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.486 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:30.487 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.487 DEBUG:boto:Params: {} 2022-09-09T22:39:30.488 DEBUG:boto:Token: None 2022-09-09T22:39:30.488 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.489 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:AdABbPog6F27FbtXoENHN1qMFrI= 2022-09-09T22:39:30.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:AdABbPog6F27FbtXoENHN1qMFrI=', 'Content-Length': '0'} 2022-09-09T22:39:30.493 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000843de091e3d13ba-00631bc0a2-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.534 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:30.535 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:30.535 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.536 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.536 DEBUG:boto:Method: GET 2022-09-09T22:39:30.537 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.537 DEBUG:boto:Data: 2022-09-09T22:39:30.538 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.538 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:30.539 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.540 DEBUG:boto:Params: {} 2022-09-09T22:39:30.540 DEBUG:boto:Token: None 2022-09-09T22:39:30.541 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.541 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:AdABbPog6F27FbtXoENHN1qMFrI= 2022-09-09T22:39:30.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:AdABbPog6F27FbtXoENHN1qMFrI=', 'Content-Length': '0'} 2022-09-09T22:39:30.545 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6682e9d89fada51-00631bc0a2-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.546 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:30.547 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:30.547 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:30.548 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:30.548 DEBUG:boto:Method: GET 2022-09-09T22:39:30.549 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:30.549 DEBUG:boto:Data: 2022-09-09T22:39:30.550 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.550 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:30.551 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.551 DEBUG:boto:Params: {} 2022-09-09T22:39:30.552 DEBUG:boto:Token: None 2022-09-09T22:39:30.553 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/ 2022-09-09T22:39:30.553 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UrNNRpjk5Mb3zByWoS0BU/b2kaw= 2022-09-09T22:39:30.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UrNNRpjk5Mb3zByWoS0BU/b2kaw=', 'Content-Length': '0'} 2022-09-09T22:39:30.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aadbdf9e66ff7bc3-00631bc0a2-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.560 DEBUG:boto:b'jkcgbj-17obj-test-zone31000falseobj-test-zone32022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:30.560 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:30.561 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:30.561 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:30.562 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:30.562 DEBUG:boto:Method: GET 2022-09-09T22:39:30.563 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:30.564 DEBUG:boto:Data: 2022-09-09T22:39:30.564 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.565 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:30.565 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.566 DEBUG:boto:Params: {} 2022-09-09T22:39:30.566 DEBUG:boto:Token: None 2022-09-09T22:39:30.567 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/ 2022-09-09T22:39:30.567 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UrNNRpjk5Mb3zByWoS0BU/b2kaw= 2022-09-09T22:39:30.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UrNNRpjk5Mb3zByWoS0BU/b2kaw=', 'Content-Length': '0'} 2022-09-09T22:39:30.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f42a8b5c59ba0fa-00631bc0a2-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.575 DEBUG:boto:b'jkcgbj-17obj-test-zone31000falseobj-test-zone32022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:30.576 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:30.577 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:30.577 DEBUG:boto:Method: HEAD 2022-09-09T22:39:30.578 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3 2022-09-09T22:39:30.578 DEBUG:boto:Data: 2022-09-09T22:39:30.579 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.579 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:30.580 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.580 DEBUG:boto:Params: {} 2022-09-09T22:39:30.581 DEBUG:boto:Token: None 2022-09-09T22:39:30.581 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/obj-test-zone3 2022-09-09T22:39:30.582 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:hlbw+j8d9jbFxUosYnR1yv2KUQI= 2022-09-09T22:39:30.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:hlbw+j8d9jbFxUosYnR1yv2KUQI=', 'Content-Length': '0'} 2022-09-09T22:39:30.587 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0da460650f15a91-00631bc0a2-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.587 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:30.588 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:30.589 DEBUG:boto:Method: HEAD 2022-09-09T22:39:30.589 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3 2022-09-09T22:39:30.590 DEBUG:boto:Data: 2022-09-09T22:39:30.590 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.591 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:30.591 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.592 DEBUG:boto:Params: {} 2022-09-09T22:39:30.592 DEBUG:boto:Token: None 2022-09-09T22:39:30.593 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/obj-test-zone3 2022-09-09T22:39:30.593 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:hlbw+j8d9jbFxUosYnR1yv2KUQI= 2022-09-09T22:39:30.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:hlbw+j8d9jbFxUosYnR1yv2KUQI=', 'Content-Length': '0'} 2022-09-09T22:39:30.598 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d7fd0caaecc04a7-00631bc0a2-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:39:30.599 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:30.600 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:30.600 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.601 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.601 DEBUG:boto:Method: GET 2022-09-09T22:39:30.602 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.602 DEBUG:boto:Data: 2022-09-09T22:39:30.603 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.604 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:30.604 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.605 DEBUG:boto:Params: {} 2022-09-09T22:39:30.605 DEBUG:boto:Token: None 2022-09-09T22:39:30.606 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.606 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:AdABbPog6F27FbtXoENHN1qMFrI= 2022-09-09T22:39:30.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:AdABbPog6F27FbtXoENHN1qMFrI=', 'Content-Length': '0'} 2022-09-09T22:39:30.610 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054629beb5a4bf1ba-00631bc0a2-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.654 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:30.655 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:30.655 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.656 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.656 DEBUG:boto:Method: GET 2022-09-09T22:39:30.657 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.657 DEBUG:boto:Data: 2022-09-09T22:39:30.658 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.659 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:30.659 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.660 DEBUG:boto:Params: {} 2022-09-09T22:39:30.660 DEBUG:boto:Token: None 2022-09-09T22:39:30.661 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:30.662 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:AdABbPog6F27FbtXoENHN1qMFrI= 2022-09-09T22:39:30.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:AdABbPog6F27FbtXoENHN1qMFrI=', 'Content-Length': '0'} 2022-09-09T22:39:30.666 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000393afc6f3a3ac4bb-00631bc0a2-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.667 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:30.667 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:30.668 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:30.668 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:30.669 DEBUG:boto:Method: GET 2022-09-09T22:39:30.669 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:30.670 DEBUG:boto:Data: 2022-09-09T22:39:30.670 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.671 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:30.671 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.672 DEBUG:boto:Params: {} 2022-09-09T22:39:30.673 DEBUG:boto:Token: None 2022-09-09T22:39:30.673 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/ 2022-09-09T22:39:30.674 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UrNNRpjk5Mb3zByWoS0BU/b2kaw= 2022-09-09T22:39:30.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UrNNRpjk5Mb3zByWoS0BU/b2kaw=', 'Content-Length': '0'} 2022-09-09T22:39:30.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d58dc647477824d-00631bc0a2-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.681 DEBUG:boto:b'jkcgbj-17obj-test-zone31000falseobj-test-zone32022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:30.682 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:30.683 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:30.683 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:30.684 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:30.684 DEBUG:boto:Method: GET 2022-09-09T22:39:30.685 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:30.685 DEBUG:boto:Data: 2022-09-09T22:39:30.686 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.686 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:30.687 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.688 DEBUG:boto:Params: {} 2022-09-09T22:39:30.688 DEBUG:boto:Token: None 2022-09-09T22:39:30.689 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/ 2022-09-09T22:39:30.689 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UrNNRpjk5Mb3zByWoS0BU/b2kaw= 2022-09-09T22:39:30.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UrNNRpjk5Mb3zByWoS0BU/b2kaw=', 'Content-Length': '0'} 2022-09-09T22:39:30.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006973c510394fdb08-00631bc0a2-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.696 DEBUG:boto:b'jkcgbj-17obj-test-zone31000falseobj-test-zone32022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:30.697 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-17 zones={test-zone2, test-zone1} 2022-09-09T22:39:30.698 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-17 zones={test-zone3, test-zone1} 2022-09-09T22:39:30.698 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:30.699 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:30.699 DEBUG:boto:Method: HEAD 2022-09-09T22:39:30.700 DEBUG:boto:Path: /jkcgbj-17/ 2022-09-09T22:39:30.701 DEBUG:boto:Data: 2022-09-09T22:39:30.701 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.702 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:30.702 DEBUG:boto:Port: 8001 2022-09-09T22:39:30.703 DEBUG:boto:Params: {} 2022-09-09T22:39:30.703 DEBUG:boto:Token: None 2022-09-09T22:39:30.704 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/ 2022-09-09T22:39:30.704 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:f/H53LPrhtVkFl9lCVG7OdFO3Zo= 2022-09-09T22:39:30.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:f/H53LPrhtVkFl9lCVG7OdFO3Zo=', 'Content-Length': '0'} 2022-09-09T22:39:30.709 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b0dd8eff003a9551-00631bc0a2-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.710 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:30.711 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:30.711 DEBUG:boto:Method: HEAD 2022-09-09T22:39:30.712 DEBUG:boto:Path: /jkcgbj-17/ 2022-09-09T22:39:30.712 DEBUG:boto:Data: 2022-09-09T22:39:30.713 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.713 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:30.714 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.714 DEBUG:boto:Params: {} 2022-09-09T22:39:30.715 DEBUG:boto:Token: None 2022-09-09T22:39:30.716 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/ 2022-09-09T22:39:30.716 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:f/H53LPrhtVkFl9lCVG7OdFO3Zo= 2022-09-09T22:39:30.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:f/H53LPrhtVkFl9lCVG7OdFO3Zo=', 'Content-Length': '0'} 2022-09-09T22:39:30.721 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a1a8ff8a35c7297-00631bc0a2-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.722 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:39:30.723 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:30.723 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:30.724 DEBUG:boto:path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:30.724 DEBUG:boto:auth_path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:30.725 DEBUG:boto:Method: GET 2022-09-09T22:39:30.726 DEBUG:boto:Path: /jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:30.726 DEBUG:boto:Data: 2022-09-09T22:39:30.727 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.727 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:30.728 DEBUG:boto:Port: 8001 2022-09-09T22:39:30.728 DEBUG:boto:Params: {} 2022-09-09T22:39:30.729 DEBUG:boto:Token: None 2022-09-09T22:39:30.729 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/?versions 2022-09-09T22:39:30.730 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qj72Xjvl4TwHEksP6XOlAydj3TA= 2022-09-09T22:39:30.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qj72Xjvl4TwHEksP6XOlAydj3TA=', 'Content-Length': '0'} 2022-09-09T22:39:30.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084a4f38dc0522300-00631bc0a2-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.739 DEBUG:boto:b'jkcgbj-17999falsedummyrcXzglYDyJxE.ArBmS3k944dyN-s9ZTtrue2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1uPhFM-4pEeqszoz0jfwkNTqs81x9txytrue2022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone217lUOlN45NLK2DSLxrYnqbOAg6jm2Tgtrue2022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone31RXW239HrAmSBD6KjNRB3qRTyY29OqZtrue2022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:30.740 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:39:30.740 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:39:30.741 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:39:30.741 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:39:30.742 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:39:30.742 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:30.743 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:30.743 DEBUG:boto:path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:30.744 DEBUG:boto:auth_path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:30.744 DEBUG:boto:Method: GET 2022-09-09T22:39:30.745 DEBUG:boto:Path: /jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:30.745 DEBUG:boto:Data: 2022-09-09T22:39:30.746 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.746 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:30.747 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.747 DEBUG:boto:Params: {} 2022-09-09T22:39:30.748 DEBUG:boto:Token: None 2022-09-09T22:39:30.748 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/?versions 2022-09-09T22:39:30.749 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qj72Xjvl4TwHEksP6XOlAydj3TA= 2022-09-09T22:39:30.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qj72Xjvl4TwHEksP6XOlAydj3TA=', 'Content-Length': '0'} 2022-09-09T22:39:30.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdf86f898ae7b289-00631bc0a2-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.758 DEBUG:boto:b'jkcgbj-17999falsedummyrcXzglYDyJxE.ArBmS3k944dyN-s9ZTtrue2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1uPhFM-4pEeqszoz0jfwkNTqs81x9txytrue2022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone217lUOlN45NLK2DSLxrYnqbOAg6jm2Tgtrue2022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone31RXW239HrAmSBD6KjNRB3qRTyY29OqZtrue2022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:30.759 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:39:30.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:39:30.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:39:30.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:39:30.761 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:30.761 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:30.762 DEBUG:boto:path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:30.762 DEBUG:boto:auth_path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:30.763 DEBUG:boto:Method: GET 2022-09-09T22:39:30.764 DEBUG:boto:Path: /jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:30.764 DEBUG:boto:Data: 2022-09-09T22:39:30.765 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.765 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:30.766 DEBUG:boto:Port: 8001 2022-09-09T22:39:30.766 DEBUG:boto:Params: {} 2022-09-09T22:39:30.767 DEBUG:boto:Token: None 2022-09-09T22:39:30.768 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/?versions 2022-09-09T22:39:30.768 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qj72Xjvl4TwHEksP6XOlAydj3TA= 2022-09-09T22:39:30.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qj72Xjvl4TwHEksP6XOlAydj3TA=', 'Content-Length': '0'} 2022-09-09T22:39:30.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000669a91dbc1532d6f-00631bc0a2-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.775 DEBUG:boto:b'jkcgbj-17999falsedummyrcXzglYDyJxE.ArBmS3k944dyN-s9ZTtrue2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1uPhFM-4pEeqszoz0jfwkNTqs81x9txytrue2022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone217lUOlN45NLK2DSLxrYnqbOAg6jm2Tgtrue2022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone31RXW239HrAmSBD6KjNRB3qRTyY29OqZtrue2022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:30.776 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:30.776 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:30.777 DEBUG:boto:path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:30.777 DEBUG:boto:auth_path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:30.778 DEBUG:boto:Method: GET 2022-09-09T22:39:30.778 DEBUG:boto:Path: /jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:30.779 DEBUG:boto:Data: 2022-09-09T22:39:30.780 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.780 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:30.781 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.781 DEBUG:boto:Params: {} 2022-09-09T22:39:30.782 DEBUG:boto:Token: None 2022-09-09T22:39:30.782 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/?versions 2022-09-09T22:39:30.783 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qj72Xjvl4TwHEksP6XOlAydj3TA= 2022-09-09T22:39:30.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qj72Xjvl4TwHEksP6XOlAydj3TA=', 'Content-Length': '0'} 2022-09-09T22:39:30.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009375efe9f9de7a70-00631bc0a2-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.790 DEBUG:boto:b'jkcgbj-17999falsedummyrcXzglYDyJxE.ArBmS3k944dyN-s9ZTtrue2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1uPhFM-4pEeqszoz0jfwkNTqs81x9txytrue2022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone217lUOlN45NLK2DSLxrYnqbOAg6jm2Tgtrue2022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone31RXW239HrAmSBD6KjNRB3qRTyY29OqZtrue2022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:30.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:39:30.791 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:39:30.792 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:39:30.792 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.793 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.793 DEBUG:boto:Method: GET 2022-09-09T22:39:30.794 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.794 DEBUG:boto:Data: 2022-09-09T22:39:30.795 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.795 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:30.796 DEBUG:boto:Port: 8001 2022-09-09T22:39:30.796 DEBUG:boto:Params: {} 2022-09-09T22:39:30.797 DEBUG:boto:Token: None 2022-09-09T22:39:30.797 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.798 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TFqdmrl3PMBGX0LPajhmKp/u2jM= 2022-09-09T22:39:30.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TFqdmrl3PMBGX0LPajhmKp/u2jM=', 'Content-Length': '0'} 2022-09-09T22:39:30.801 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f3b6e3a3095e023d-00631bc0a2-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.802 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:39:30.803 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:39:30.803 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.804 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.804 DEBUG:boto:Method: GET 2022-09-09T22:39:30.805 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.805 DEBUG:boto:Data: 2022-09-09T22:39:30.806 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.806 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:30.807 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.807 DEBUG:boto:Params: {} 2022-09-09T22:39:30.808 DEBUG:boto:Token: None 2022-09-09T22:39:30.808 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.809 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TFqdmrl3PMBGX0LPajhmKp/u2jM= 2022-09-09T22:39:30.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TFqdmrl3PMBGX0LPajhmKp/u2jM=', 'Content-Length': '0'} 2022-09-09T22:39:30.813 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000027234e77615c3ed6-00631bc0a2-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.814 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:39:30.814 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:39:30.815 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.815 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.816 DEBUG:boto:Method: HEAD 2022-09-09T22:39:30.816 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.817 DEBUG:boto:Data: 2022-09-09T22:39:30.817 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.818 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:30.818 DEBUG:boto:Port: 8001 2022-09-09T22:39:30.819 DEBUG:boto:Params: {} 2022-09-09T22:39:30.819 DEBUG:boto:Token: None 2022-09-09T22:39:30.820 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.821 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KC09ThVWw4BLPh8m9AGyDBFGrCY= 2022-09-09T22:39:30.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KC09ThVWw4BLPh8m9AGyDBFGrCY=', 'Content-Length': '0'} 2022-09-09T22:39:30.824 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005e16f0076f769fc6-00631bc0a2-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.825 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:39:30.825 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:39:30.826 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.826 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.827 DEBUG:boto:Method: HEAD 2022-09-09T22:39:30.827 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.828 DEBUG:boto:Data: 2022-09-09T22:39:30.828 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.829 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:30.829 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.830 DEBUG:boto:Params: {} 2022-09-09T22:39:30.831 DEBUG:boto:Token: None 2022-09-09T22:39:30.831 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.832 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KC09ThVWw4BLPh8m9AGyDBFGrCY= 2022-09-09T22:39:30.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KC09ThVWw4BLPh8m9AGyDBFGrCY=', 'Content-Length': '0'} 2022-09-09T22:39:30.836 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008e764d3537b6cdfd-00631bc0a2-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:39:30.837 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:39:30.838 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:39:30.838 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.839 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.839 DEBUG:boto:Method: GET 2022-09-09T22:39:30.840 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.840 DEBUG:boto:Data: 2022-09-09T22:39:30.841 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.841 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:30.842 DEBUG:boto:Port: 8001 2022-09-09T22:39:30.843 DEBUG:boto:Params: {} 2022-09-09T22:39:30.843 DEBUG:boto:Token: None 2022-09-09T22:39:30.844 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.844 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TFqdmrl3PMBGX0LPajhmKp/u2jM= 2022-09-09T22:39:30.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TFqdmrl3PMBGX0LPajhmKp/u2jM=', 'Content-Length': '0'} 2022-09-09T22:39:30.848 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000277b71ff950936e-00631bc0a2-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.849 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:39:30.849 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:39:30.850 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.850 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.851 DEBUG:boto:Method: GET 2022-09-09T22:39:30.851 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.852 DEBUG:boto:Data: 2022-09-09T22:39:30.852 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.853 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:30.853 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.854 DEBUG:boto:Params: {} 2022-09-09T22:39:30.855 DEBUG:boto:Token: None 2022-09-09T22:39:30.855 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.856 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TFqdmrl3PMBGX0LPajhmKp/u2jM= 2022-09-09T22:39:30.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TFqdmrl3PMBGX0LPajhmKp/u2jM=', 'Content-Length': '0'} 2022-09-09T22:39:30.859 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b24873931c18658d-00631bc0a2-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.860 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:30.861 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:30.861 DEBUG:boto:path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:39:30.862 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:39:30.862 DEBUG:boto:Method: GET 2022-09-09T22:39:30.863 DEBUG:boto:Path: /jkcgbj-17/?prefix=dummy 2022-09-09T22:39:30.863 DEBUG:boto:Data: 2022-09-09T22:39:30.864 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.864 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:30.865 DEBUG:boto:Port: 8001 2022-09-09T22:39:30.865 DEBUG:boto:Params: {} 2022-09-09T22:39:30.866 DEBUG:boto:Token: None 2022-09-09T22:39:30.867 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/ 2022-09-09T22:39:30.867 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UrNNRpjk5Mb3zByWoS0BU/b2kaw= 2022-09-09T22:39:30.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UrNNRpjk5Mb3zByWoS0BU/b2kaw=', 'Content-Length': '0'} 2022-09-09T22:39:30.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008952fe522a23d21a-00631bc0a2-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.918 DEBUG:boto:b'jkcgbj-17dummy1000falsedummy2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:30.919 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:30.919 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:30.920 DEBUG:boto:path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:39:30.921 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:39:30.921 DEBUG:boto:Method: GET 2022-09-09T22:39:30.922 DEBUG:boto:Path: /jkcgbj-17/?prefix=dummy 2022-09-09T22:39:30.922 DEBUG:boto:Data: 2022-09-09T22:39:30.923 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.923 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:30.924 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.924 DEBUG:boto:Params: {} 2022-09-09T22:39:30.925 DEBUG:boto:Token: None 2022-09-09T22:39:30.926 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/ 2022-09-09T22:39:30.926 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UrNNRpjk5Mb3zByWoS0BU/b2kaw= 2022-09-09T22:39:30.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UrNNRpjk5Mb3zByWoS0BU/b2kaw=', 'Content-Length': '0'} 2022-09-09T22:39:30.932 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000060e08cb6e8ef52f-00631bc0a2-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.933 DEBUG:boto:b'jkcgbj-17dummy1000falsedummy2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:30.934 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:39:30.935 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:39:30.935 DEBUG:boto:Method: HEAD 2022-09-09T22:39:30.936 DEBUG:boto:Path: /jkcgbj-17/dummy 2022-09-09T22:39:30.936 DEBUG:boto:Data: 2022-09-09T22:39:30.937 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.937 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:30.938 DEBUG:boto:Port: 8001 2022-09-09T22:39:30.938 DEBUG:boto:Params: {} 2022-09-09T22:39:30.939 DEBUG:boto:Token: None 2022-09-09T22:39:30.939 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/dummy 2022-09-09T22:39:30.940 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:fdGi+srmQ4qU5oakOHRpQR9fCH0= 2022-09-09T22:39:30.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:fdGi+srmQ4qU5oakOHRpQR9fCH0=', 'Content-Length': '0'} 2022-09-09T22:39:30.944 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000de66869640855f9e-00631bc0a2-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.945 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:39:30.946 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:39:30.946 DEBUG:boto:Method: HEAD 2022-09-09T22:39:30.947 DEBUG:boto:Path: /jkcgbj-17/dummy 2022-09-09T22:39:30.947 DEBUG:boto:Data: 2022-09-09T22:39:30.948 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.948 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:30.949 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.949 DEBUG:boto:Params: {} 2022-09-09T22:39:30.950 DEBUG:boto:Token: None 2022-09-09T22:39:30.951 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/dummy 2022-09-09T22:39:30.951 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:fdGi+srmQ4qU5oakOHRpQR9fCH0= 2022-09-09T22:39:30.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:fdGi+srmQ4qU5oakOHRpQR9fCH0=', 'Content-Length': '0'} 2022-09-09T22:39:30.956 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000294a0a5052ae98c5-00631bc0a2-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:39:30.957 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:39:30.958 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:39:30.958 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.959 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.959 DEBUG:boto:Method: GET 2022-09-09T22:39:30.960 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.960 DEBUG:boto:Data: 2022-09-09T22:39:30.961 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.961 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:30.962 DEBUG:boto:Port: 8001 2022-09-09T22:39:30.962 DEBUG:boto:Params: {} 2022-09-09T22:39:30.963 DEBUG:boto:Token: None 2022-09-09T22:39:30.964 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.964 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TFqdmrl3PMBGX0LPajhmKp/u2jM= 2022-09-09T22:39:30.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TFqdmrl3PMBGX0LPajhmKp/u2jM=', 'Content-Length': '0'} 2022-09-09T22:39:30.968 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000015316d8ceb592925-00631bc0a2-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.969 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:39:30.969 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:39:30.970 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.970 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.971 DEBUG:boto:Method: GET 2022-09-09T22:39:30.971 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.972 DEBUG:boto:Data: 2022-09-09T22:39:30.972 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.973 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:30.974 DEBUG:boto:Port: 8000 2022-09-09T22:39:30.974 DEBUG:boto:Params: {} 2022-09-09T22:39:30.975 DEBUG:boto:Token: None 2022-09-09T22:39:30.975 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:30.976 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TFqdmrl3PMBGX0LPajhmKp/u2jM= 2022-09-09T22:39:30.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TFqdmrl3PMBGX0LPajhmKp/u2jM=', 'Content-Length': '0'} 2022-09-09T22:39:30.979 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001b45099235db1978-00631bc0a2-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.980 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:30.981 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:30.981 DEBUG:boto:path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:39:30.982 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:39:30.983 DEBUG:boto:Method: GET 2022-09-09T22:39:30.983 DEBUG:boto:Path: /jkcgbj-17/?prefix=dummy 2022-09-09T22:39:30.984 DEBUG:boto:Data: 2022-09-09T22:39:30.984 DEBUG:boto:Headers: {} 2022-09-09T22:39:30.985 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:30.985 DEBUG:boto:Port: 8001 2022-09-09T22:39:30.986 DEBUG:boto:Params: {} 2022-09-09T22:39:30.986 DEBUG:boto:Token: None 2022-09-09T22:39:30.987 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:30 GMT /jkcgbj-17/ 2022-09-09T22:39:30.988 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UrNNRpjk5Mb3zByWoS0BU/b2kaw= 2022-09-09T22:39:30.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:30 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UrNNRpjk5Mb3zByWoS0BU/b2kaw=', 'Content-Length': '0'} 2022-09-09T22:39:30.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008641cb78fa26113b-00631bc0a2-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:30.996 DEBUG:boto:b'jkcgbj-17dummy1000falsedummy2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:30.996 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:30.997 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:30.998 DEBUG:boto:path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:39:30.998 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:39:30.999 DEBUG:boto:Method: GET 2022-09-09T22:39:30.999 DEBUG:boto:Path: /jkcgbj-17/?prefix=dummy 2022-09-09T22:39:31.000 DEBUG:boto:Data: 2022-09-09T22:39:31.000 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.001 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:31.002 DEBUG:boto:Port: 8000 2022-09-09T22:39:31.002 DEBUG:boto:Params: {} 2022-09-09T22:39:31.003 DEBUG:boto:Token: None 2022-09-09T22:39:31.003 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/ 2022-09-09T22:39:31.004 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jyNXMbT3+Cm/MH5OBi0TyUft+cE= 2022-09-09T22:39:31.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jyNXMbT3+Cm/MH5OBi0TyUft+cE=', 'Content-Length': '0'} 2022-09-09T22:39:31.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd3638577083df10-00631bc0a2-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.054 DEBUG:boto:b'jkcgbj-17dummy1000falsedummy2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:31.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:39:31.055 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:31.056 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:31.056 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.057 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.057 DEBUG:boto:Method: GET 2022-09-09T22:39:31.058 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.058 DEBUG:boto:Data: 2022-09-09T22:39:31.059 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.059 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:31.060 DEBUG:boto:Port: 8001 2022-09-09T22:39:31.060 DEBUG:boto:Params: {} 2022-09-09T22:39:31.061 DEBUG:boto:Token: None 2022-09-09T22:39:31.062 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.062 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:zTJzPWL6gvi5Gs9I62DMnhKNGCA= 2022-09-09T22:39:31.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:zTJzPWL6gvi5Gs9I62DMnhKNGCA=', 'Content-Length': '0'} 2022-09-09T22:39:31.066 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003657d829a902b52e-00631bc0a3-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.067 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:31.067 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:31.068 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.068 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.069 DEBUG:boto:Method: GET 2022-09-09T22:39:31.069 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.070 DEBUG:boto:Data: 2022-09-09T22:39:31.070 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.071 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:31.072 DEBUG:boto:Port: 8000 2022-09-09T22:39:31.072 DEBUG:boto:Params: {} 2022-09-09T22:39:31.073 DEBUG:boto:Token: None 2022-09-09T22:39:31.073 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.074 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:zTJzPWL6gvi5Gs9I62DMnhKNGCA= 2022-09-09T22:39:31.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:zTJzPWL6gvi5Gs9I62DMnhKNGCA=', 'Content-Length': '0'} 2022-09-09T22:39:31.078 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000deef115cc701110b-00631bc0a3-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.079 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:31.079 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:31.080 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.080 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.081 DEBUG:boto:Method: HEAD 2022-09-09T22:39:31.081 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.082 DEBUG:boto:Data: 2022-09-09T22:39:31.082 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.083 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:31.083 DEBUG:boto:Port: 8001 2022-09-09T22:39:31.084 DEBUG:boto:Params: {} 2022-09-09T22:39:31.084 DEBUG:boto:Token: None 2022-09-09T22:39:31.085 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.086 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YXcCsm2tUFmDRK/x2pqDFFnJm3k= 2022-09-09T22:39:31.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YXcCsm2tUFmDRK/x2pqDFFnJm3k=', 'Content-Length': '0'} 2022-09-09T22:39:31.089 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f3e6237344c954e-00631bc0a3-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.090 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:31.090 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:31.091 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.091 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.092 DEBUG:boto:Method: HEAD 2022-09-09T22:39:31.092 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.093 DEBUG:boto:Data: 2022-09-09T22:39:31.093 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.094 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:31.094 DEBUG:boto:Port: 8000 2022-09-09T22:39:31.095 DEBUG:boto:Params: {} 2022-09-09T22:39:31.095 DEBUG:boto:Token: None 2022-09-09T22:39:31.096 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.096 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YXcCsm2tUFmDRK/x2pqDFFnJm3k= 2022-09-09T22:39:31.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YXcCsm2tUFmDRK/x2pqDFFnJm3k=', 'Content-Length': '0'} 2022-09-09T22:39:31.100 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000101b13c382f1132e-00631bc0a3-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:39:31.102 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:31.102 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:31.103 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.103 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.104 DEBUG:boto:Method: GET 2022-09-09T22:39:31.104 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.105 DEBUG:boto:Data: 2022-09-09T22:39:31.105 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.106 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:31.107 DEBUG:boto:Port: 8001 2022-09-09T22:39:31.107 DEBUG:boto:Params: {} 2022-09-09T22:39:31.108 DEBUG:boto:Token: None 2022-09-09T22:39:31.108 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.109 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:zTJzPWL6gvi5Gs9I62DMnhKNGCA= 2022-09-09T22:39:31.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:zTJzPWL6gvi5Gs9I62DMnhKNGCA=', 'Content-Length': '0'} 2022-09-09T22:39:31.113 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d78d7c5fb5a69c4-00631bc0a3-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.154 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:31.155 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:31.155 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.156 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.156 DEBUG:boto:Method: GET 2022-09-09T22:39:31.157 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.157 DEBUG:boto:Data: 2022-09-09T22:39:31.158 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.159 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:31.159 DEBUG:boto:Port: 8000 2022-09-09T22:39:31.160 DEBUG:boto:Params: {} 2022-09-09T22:39:31.160 DEBUG:boto:Token: None 2022-09-09T22:39:31.161 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.161 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:zTJzPWL6gvi5Gs9I62DMnhKNGCA= 2022-09-09T22:39:31.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:zTJzPWL6gvi5Gs9I62DMnhKNGCA=', 'Content-Length': '0'} 2022-09-09T22:39:31.165 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007066a53b843af13c-00631bc0a3-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.166 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:31.167 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:31.167 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:31.168 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:31.168 DEBUG:boto:Method: GET 2022-09-09T22:39:31.169 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:31.169 DEBUG:boto:Data: 2022-09-09T22:39:31.170 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.170 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:31.171 DEBUG:boto:Port: 8001 2022-09-09T22:39:31.171 DEBUG:boto:Params: {} 2022-09-09T22:39:31.172 DEBUG:boto:Token: None 2022-09-09T22:39:31.173 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/ 2022-09-09T22:39:31.173 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jyNXMbT3+Cm/MH5OBi0TyUft+cE= 2022-09-09T22:39:31.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jyNXMbT3+Cm/MH5OBi0TyUft+cE=', 'Content-Length': '0'} 2022-09-09T22:39:31.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017a67e2c88b19845-00631bc0a3-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.181 DEBUG:boto:b'jkcgbj-17obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:31.181 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:31.182 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:31.183 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:31.183 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:31.184 DEBUG:boto:Method: GET 2022-09-09T22:39:31.184 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:31.185 DEBUG:boto:Data: 2022-09-09T22:39:31.186 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.186 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:31.187 DEBUG:boto:Port: 8000 2022-09-09T22:39:31.187 DEBUG:boto:Params: {} 2022-09-09T22:39:31.188 DEBUG:boto:Token: None 2022-09-09T22:39:31.189 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/ 2022-09-09T22:39:31.189 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jyNXMbT3+Cm/MH5OBi0TyUft+cE= 2022-09-09T22:39:31.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jyNXMbT3+Cm/MH5OBi0TyUft+cE=', 'Content-Length': '0'} 2022-09-09T22:39:31.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed095396ca7bf246-00631bc0a3-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.196 DEBUG:boto:b'jkcgbj-17obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:31.197 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:31.198 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:31.198 DEBUG:boto:Method: HEAD 2022-09-09T22:39:31.199 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1 2022-09-09T22:39:31.199 DEBUG:boto:Data: 2022-09-09T22:39:31.200 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.200 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:31.201 DEBUG:boto:Port: 8001 2022-09-09T22:39:31.202 DEBUG:boto:Params: {} 2022-09-09T22:39:31.202 DEBUG:boto:Token: None 2022-09-09T22:39:31.203 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/obj-test-zone1 2022-09-09T22:39:31.203 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:uOe8enSobTgL1YCE5Mj3/hZRBxE= 2022-09-09T22:39:31.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:uOe8enSobTgL1YCE5Mj3/hZRBxE=', 'Content-Length': '0'} 2022-09-09T22:39:31.208 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002326ee7f154c9510-00631bc0a3-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.209 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:31.210 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:31.210 DEBUG:boto:Method: HEAD 2022-09-09T22:39:31.211 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1 2022-09-09T22:39:31.211 DEBUG:boto:Data: 2022-09-09T22:39:31.212 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.212 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:31.213 DEBUG:boto:Port: 8000 2022-09-09T22:39:31.213 DEBUG:boto:Params: {} 2022-09-09T22:39:31.214 DEBUG:boto:Token: None 2022-09-09T22:39:31.215 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/obj-test-zone1 2022-09-09T22:39:31.215 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:uOe8enSobTgL1YCE5Mj3/hZRBxE= 2022-09-09T22:39:31.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:uOe8enSobTgL1YCE5Mj3/hZRBxE=', 'Content-Length': '0'} 2022-09-09T22:39:31.220 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a71dfc0826a5aaeb-00631bc0a3-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:39:31.221 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:31.222 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:31.222 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.223 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.223 DEBUG:boto:Method: GET 2022-09-09T22:39:31.224 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.224 DEBUG:boto:Data: 2022-09-09T22:39:31.225 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.225 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:31.226 DEBUG:boto:Port: 8001 2022-09-09T22:39:31.226 DEBUG:boto:Params: {} 2022-09-09T22:39:31.227 DEBUG:boto:Token: None 2022-09-09T22:39:31.227 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.228 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:zTJzPWL6gvi5Gs9I62DMnhKNGCA= 2022-09-09T22:39:31.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:zTJzPWL6gvi5Gs9I62DMnhKNGCA=', 'Content-Length': '0'} 2022-09-09T22:39:31.232 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006f3497270e8e6095-00631bc0a3-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.274 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:31.275 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:31.275 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.276 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.276 DEBUG:boto:Method: GET 2022-09-09T22:39:31.277 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.277 DEBUG:boto:Data: 2022-09-09T22:39:31.278 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.278 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:31.279 DEBUG:boto:Port: 8000 2022-09-09T22:39:31.279 DEBUG:boto:Params: {} 2022-09-09T22:39:31.280 DEBUG:boto:Token: None 2022-09-09T22:39:31.281 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:31.281 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:zTJzPWL6gvi5Gs9I62DMnhKNGCA= 2022-09-09T22:39:31.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:zTJzPWL6gvi5Gs9I62DMnhKNGCA=', 'Content-Length': '0'} 2022-09-09T22:39:31.285 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed9c15c58e462173-00631bc0a3-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.286 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:31.286 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:31.287 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:31.287 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:31.288 DEBUG:boto:Method: GET 2022-09-09T22:39:31.288 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:31.289 DEBUG:boto:Data: 2022-09-09T22:39:31.289 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.290 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:31.290 DEBUG:boto:Port: 8001 2022-09-09T22:39:31.291 DEBUG:boto:Params: {} 2022-09-09T22:39:31.291 DEBUG:boto:Token: None 2022-09-09T22:39:31.292 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/ 2022-09-09T22:39:31.293 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jyNXMbT3+Cm/MH5OBi0TyUft+cE= 2022-09-09T22:39:31.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jyNXMbT3+Cm/MH5OBi0TyUft+cE=', 'Content-Length': '0'} 2022-09-09T22:39:31.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010b6aeddd04a78f7-00631bc0a3-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.300 DEBUG:boto:b'jkcgbj-17obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:31.301 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:31.301 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:31.302 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:31.302 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:31.303 DEBUG:boto:Method: GET 2022-09-09T22:39:31.304 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:31.304 DEBUG:boto:Data: 2022-09-09T22:39:31.305 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.305 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:31.306 DEBUG:boto:Port: 8000 2022-09-09T22:39:31.306 DEBUG:boto:Params: {} 2022-09-09T22:39:31.307 DEBUG:boto:Token: None 2022-09-09T22:39:31.307 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/ 2022-09-09T22:39:31.308 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jyNXMbT3+Cm/MH5OBi0TyUft+cE= 2022-09-09T22:39:31.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jyNXMbT3+Cm/MH5OBi0TyUft+cE=', 'Content-Length': '0'} 2022-09-09T22:39:31.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c77a7049d0dd6ed8-00631bc0a3-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.315 DEBUG:boto:b'jkcgbj-17obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:31.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:39:31.316 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:31.316 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:31.317 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.317 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.318 DEBUG:boto:Method: GET 2022-09-09T22:39:31.318 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.319 DEBUG:boto:Data: 2022-09-09T22:39:31.319 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.320 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:31.320 DEBUG:boto:Port: 8001 2022-09-09T22:39:31.321 DEBUG:boto:Params: {} 2022-09-09T22:39:31.321 DEBUG:boto:Token: None 2022-09-09T22:39:31.322 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.323 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:SVO48b7i+YhEokjm/wImAbbXtR8= 2022-09-09T22:39:31.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:SVO48b7i+YhEokjm/wImAbbXtR8=', 'Content-Length': '0'} 2022-09-09T22:39:31.326 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d39982a4b851198-00631bc0a3-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.327 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:31.328 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:31.328 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.329 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.329 DEBUG:boto:Method: GET 2022-09-09T22:39:31.330 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.330 DEBUG:boto:Data: 2022-09-09T22:39:31.331 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.331 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:31.332 DEBUG:boto:Port: 8000 2022-09-09T22:39:31.333 DEBUG:boto:Params: {} 2022-09-09T22:39:31.333 DEBUG:boto:Token: None 2022-09-09T22:39:31.334 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.334 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:SVO48b7i+YhEokjm/wImAbbXtR8= 2022-09-09T22:39:31.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:SVO48b7i+YhEokjm/wImAbbXtR8=', 'Content-Length': '0'} 2022-09-09T22:39:31.338 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a176416aafab1b17-00631bc0a3-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.339 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:31.339 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:31.340 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.340 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.341 DEBUG:boto:Method: HEAD 2022-09-09T22:39:31.341 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.342 DEBUG:boto:Data: 2022-09-09T22:39:31.342 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.343 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:31.343 DEBUG:boto:Port: 8001 2022-09-09T22:39:31.344 DEBUG:boto:Params: {} 2022-09-09T22:39:31.344 DEBUG:boto:Token: None 2022-09-09T22:39:31.345 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.345 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6ry+cfm9+Ffthh8jLv57xs1bfW8= 2022-09-09T22:39:31.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6ry+cfm9+Ffthh8jLv57xs1bfW8=', 'Content-Length': '0'} 2022-09-09T22:39:31.349 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003d804d166e898e62-00631bc0a3-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.350 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:31.350 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:31.351 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.351 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.351 DEBUG:boto:Method: HEAD 2022-09-09T22:39:31.352 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.352 DEBUG:boto:Data: 2022-09-09T22:39:31.353 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.353 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:31.354 DEBUG:boto:Port: 8000 2022-09-09T22:39:31.354 DEBUG:boto:Params: {} 2022-09-09T22:39:31.355 DEBUG:boto:Token: None 2022-09-09T22:39:31.355 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.356 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6ry+cfm9+Ffthh8jLv57xs1bfW8= 2022-09-09T22:39:31.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6ry+cfm9+Ffthh8jLv57xs1bfW8=', 'Content-Length': '0'} 2022-09-09T22:39:31.359 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072e9ab254a216774-00631bc0a3-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:39:31.360 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:31.361 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:31.361 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.362 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.362 DEBUG:boto:Method: GET 2022-09-09T22:39:31.363 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.363 DEBUG:boto:Data: 2022-09-09T22:39:31.364 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.364 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:31.365 DEBUG:boto:Port: 8001 2022-09-09T22:39:31.365 DEBUG:boto:Params: {} 2022-09-09T22:39:31.366 DEBUG:boto:Token: None 2022-09-09T22:39:31.366 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.367 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:SVO48b7i+YhEokjm/wImAbbXtR8= 2022-09-09T22:39:31.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:SVO48b7i+YhEokjm/wImAbbXtR8=', 'Content-Length': '0'} 2022-09-09T22:39:31.371 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d6c5e18557c1930-00631bc0a3-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.414 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:31.415 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:31.415 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.416 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.416 DEBUG:boto:Method: GET 2022-09-09T22:39:31.417 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.417 DEBUG:boto:Data: 2022-09-09T22:39:31.418 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.418 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:31.426 DEBUG:boto:Port: 8000 2022-09-09T22:39:31.427 DEBUG:boto:Params: {} 2022-09-09T22:39:31.428 DEBUG:boto:Token: None 2022-09-09T22:39:31.428 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.429 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:SVO48b7i+YhEokjm/wImAbbXtR8= 2022-09-09T22:39:31.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:SVO48b7i+YhEokjm/wImAbbXtR8=', 'Content-Length': '0'} 2022-09-09T22:39:31.433 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a305bce78f55c5d3-00631bc0a3-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.434 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:31.434 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:31.435 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:31.435 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:31.436 DEBUG:boto:Method: GET 2022-09-09T22:39:31.436 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:31.437 DEBUG:boto:Data: 2022-09-09T22:39:31.437 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.438 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:31.438 DEBUG:boto:Port: 8001 2022-09-09T22:39:31.439 DEBUG:boto:Params: {} 2022-09-09T22:39:31.439 DEBUG:boto:Token: None 2022-09-09T22:39:31.440 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/ 2022-09-09T22:39:31.441 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jyNXMbT3+Cm/MH5OBi0TyUft+cE= 2022-09-09T22:39:31.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jyNXMbT3+Cm/MH5OBi0TyUft+cE=', 'Content-Length': '0'} 2022-09-09T22:39:31.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1e55df46b645be7-00631bc0a3-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.448 DEBUG:boto:b'jkcgbj-17obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:31.449 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:31.450 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:31.450 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:31.451 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:31.451 DEBUG:boto:Method: GET 2022-09-09T22:39:31.452 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:31.452 DEBUG:boto:Data: 2022-09-09T22:39:31.453 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.453 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:31.454 DEBUG:boto:Port: 8000 2022-09-09T22:39:31.454 DEBUG:boto:Params: {} 2022-09-09T22:39:31.455 DEBUG:boto:Token: None 2022-09-09T22:39:31.455 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/ 2022-09-09T22:39:31.456 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jyNXMbT3+Cm/MH5OBi0TyUft+cE= 2022-09-09T22:39:31.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jyNXMbT3+Cm/MH5OBi0TyUft+cE=', 'Content-Length': '0'} 2022-09-09T22:39:31.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019e7b1b78ceaa1c3-00631bc0a3-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.464 DEBUG:boto:b'jkcgbj-17obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:31.465 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:31.465 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:31.466 DEBUG:boto:Method: HEAD 2022-09-09T22:39:31.466 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2 2022-09-09T22:39:31.467 DEBUG:boto:Data: 2022-09-09T22:39:31.467 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.468 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:31.468 DEBUG:boto:Port: 8001 2022-09-09T22:39:31.469 DEBUG:boto:Params: {} 2022-09-09T22:39:31.469 DEBUG:boto:Token: None 2022-09-09T22:39:31.470 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/obj-test-zone2 2022-09-09T22:39:31.470 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JhofL1VnYOPOIdCoZHdMNPCtTSU= 2022-09-09T22:39:31.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JhofL1VnYOPOIdCoZHdMNPCtTSU=', 'Content-Length': '0'} 2022-09-09T22:39:31.474 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000533ce25b376f9ced-00631bc0a3-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.475 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:31.476 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:31.476 DEBUG:boto:Method: HEAD 2022-09-09T22:39:31.476 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2 2022-09-09T22:39:31.477 DEBUG:boto:Data: 2022-09-09T22:39:31.477 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.478 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:31.478 DEBUG:boto:Port: 8000 2022-09-09T22:39:31.479 DEBUG:boto:Params: {} 2022-09-09T22:39:31.479 DEBUG:boto:Token: None 2022-09-09T22:39:31.480 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/obj-test-zone2 2022-09-09T22:39:31.481 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JhofL1VnYOPOIdCoZHdMNPCtTSU= 2022-09-09T22:39:31.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JhofL1VnYOPOIdCoZHdMNPCtTSU=', 'Content-Length': '0'} 2022-09-09T22:39:31.485 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004fdb6234d8acdad4-00631bc0a3-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:39:31.486 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:31.487 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:31.487 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.488 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.488 DEBUG:boto:Method: GET 2022-09-09T22:39:31.489 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.489 DEBUG:boto:Data: 2022-09-09T22:39:31.490 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.490 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:31.491 DEBUG:boto:Port: 8001 2022-09-09T22:39:31.492 DEBUG:boto:Params: {} 2022-09-09T22:39:31.492 DEBUG:boto:Token: None 2022-09-09T22:39:31.493 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.493 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:SVO48b7i+YhEokjm/wImAbbXtR8= 2022-09-09T22:39:31.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:SVO48b7i+YhEokjm/wImAbbXtR8=', 'Content-Length': '0'} 2022-09-09T22:39:31.497 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000062eae4facb140d9d-00631bc0a3-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.538 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:31.539 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:31.539 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.539 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.540 DEBUG:boto:Method: GET 2022-09-09T22:39:31.540 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.541 DEBUG:boto:Data: 2022-09-09T22:39:31.542 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.542 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:31.543 DEBUG:boto:Port: 8000 2022-09-09T22:39:31.543 DEBUG:boto:Params: {} 2022-09-09T22:39:31.544 DEBUG:boto:Token: None 2022-09-09T22:39:31.544 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:31.545 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:SVO48b7i+YhEokjm/wImAbbXtR8= 2022-09-09T22:39:31.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:SVO48b7i+YhEokjm/wImAbbXtR8=', 'Content-Length': '0'} 2022-09-09T22:39:31.549 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dde6b6b03430949f-00631bc0a3-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.550 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:31.550 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:31.551 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:31.552 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:31.552 DEBUG:boto:Method: GET 2022-09-09T22:39:31.553 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:31.553 DEBUG:boto:Data: 2022-09-09T22:39:31.554 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.554 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:31.555 DEBUG:boto:Port: 8001 2022-09-09T22:39:31.555 DEBUG:boto:Params: {} 2022-09-09T22:39:31.556 DEBUG:boto:Token: None 2022-09-09T22:39:31.556 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/ 2022-09-09T22:39:31.557 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jyNXMbT3+Cm/MH5OBi0TyUft+cE= 2022-09-09T22:39:31.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jyNXMbT3+Cm/MH5OBi0TyUft+cE=', 'Content-Length': '0'} 2022-09-09T22:39:31.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000177c28e1d2cfd111-00631bc0a3-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.564 DEBUG:boto:b'jkcgbj-17obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:31.565 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:31.566 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:31.566 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:31.567 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:31.567 DEBUG:boto:Method: GET 2022-09-09T22:39:31.568 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:31.568 DEBUG:boto:Data: 2022-09-09T22:39:31.569 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.569 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:31.570 DEBUG:boto:Port: 8000 2022-09-09T22:39:31.570 DEBUG:boto:Params: {} 2022-09-09T22:39:31.571 DEBUG:boto:Token: None 2022-09-09T22:39:31.572 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/ 2022-09-09T22:39:31.572 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jyNXMbT3+Cm/MH5OBi0TyUft+cE= 2022-09-09T22:39:31.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jyNXMbT3+Cm/MH5OBi0TyUft+cE=', 'Content-Length': '0'} 2022-09-09T22:39:31.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd421ecb2ae22ab5-00631bc0a3-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.579 DEBUG:boto:b'jkcgbj-17obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:31.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:39:31.581 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:31.581 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:31.582 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.582 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.583 DEBUG:boto:Method: GET 2022-09-09T22:39:31.583 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.584 DEBUG:boto:Data: 2022-09-09T22:39:31.584 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.585 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:31.585 DEBUG:boto:Port: 8001 2022-09-09T22:39:31.586 DEBUG:boto:Params: {} 2022-09-09T22:39:31.586 DEBUG:boto:Token: None 2022-09-09T22:39:31.587 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.588 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:G6Z4fXpCLkxGkGuESDzlI/ynyX4= 2022-09-09T22:39:31.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:G6Z4fXpCLkxGkGuESDzlI/ynyX4=', 'Content-Length': '0'} 2022-09-09T22:39:31.591 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ef6d89de4342b24-00631bc0a3-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.592 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:31.592 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:31.593 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.593 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.594 DEBUG:boto:Method: GET 2022-09-09T22:39:31.595 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.595 DEBUG:boto:Data: 2022-09-09T22:39:31.596 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.596 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:31.597 DEBUG:boto:Port: 8000 2022-09-09T22:39:31.597 DEBUG:boto:Params: {} 2022-09-09T22:39:31.598 DEBUG:boto:Token: None 2022-09-09T22:39:31.598 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.599 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:G6Z4fXpCLkxGkGuESDzlI/ynyX4= 2022-09-09T22:39:31.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:G6Z4fXpCLkxGkGuESDzlI/ynyX4=', 'Content-Length': '0'} 2022-09-09T22:39:31.602 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004932602defc8a8f9-00631bc0a3-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.603 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:31.604 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:31.604 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.605 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.606 DEBUG:boto:Method: HEAD 2022-09-09T22:39:31.607 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.608 DEBUG:boto:Data: 2022-09-09T22:39:31.609 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.610 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:31.611 DEBUG:boto:Port: 8001 2022-09-09T22:39:31.612 DEBUG:boto:Params: {} 2022-09-09T22:39:31.613 DEBUG:boto:Token: None 2022-09-09T22:39:31.614 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.615 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+L/NUWufIlgXZG9PC7886bOvgdo= 2022-09-09T22:39:31.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+L/NUWufIlgXZG9PC7886bOvgdo=', 'Content-Length': '0'} 2022-09-09T22:39:31.619 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000068481b178ab68e80-00631bc0a3-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.620 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:31.621 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:31.621 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.621 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.622 DEBUG:boto:Method: HEAD 2022-09-09T22:39:31.622 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.623 DEBUG:boto:Data: 2022-09-09T22:39:31.623 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.624 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:31.624 DEBUG:boto:Port: 8000 2022-09-09T22:39:31.625 DEBUG:boto:Params: {} 2022-09-09T22:39:31.625 DEBUG:boto:Token: None 2022-09-09T22:39:31.626 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.626 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+L/NUWufIlgXZG9PC7886bOvgdo= 2022-09-09T22:39:31.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+L/NUWufIlgXZG9PC7886bOvgdo=', 'Content-Length': '0'} 2022-09-09T22:39:31.630 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d1e312cbde453abf-00631bc0a3-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:39:31.631 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:31.631 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:31.632 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.632 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.633 DEBUG:boto:Method: GET 2022-09-09T22:39:31.633 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.634 DEBUG:boto:Data: 2022-09-09T22:39:31.635 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.635 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:31.636 DEBUG:boto:Port: 8001 2022-09-09T22:39:31.636 DEBUG:boto:Params: {} 2022-09-09T22:39:31.637 DEBUG:boto:Token: None 2022-09-09T22:39:31.638 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.638 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:G6Z4fXpCLkxGkGuESDzlI/ynyX4= 2022-09-09T22:39:31.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:G6Z4fXpCLkxGkGuESDzlI/ynyX4=', 'Content-Length': '0'} 2022-09-09T22:39:31.642 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e10cacf68a4879f-00631bc0a3-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.686 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:31.686 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:31.687 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.688 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.688 DEBUG:boto:Method: GET 2022-09-09T22:39:31.689 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.689 DEBUG:boto:Data: 2022-09-09T22:39:31.690 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.690 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:31.691 DEBUG:boto:Port: 8000 2022-09-09T22:39:31.691 DEBUG:boto:Params: {} 2022-09-09T22:39:31.692 DEBUG:boto:Token: None 2022-09-09T22:39:31.693 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.693 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:G6Z4fXpCLkxGkGuESDzlI/ynyX4= 2022-09-09T22:39:31.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:G6Z4fXpCLkxGkGuESDzlI/ynyX4=', 'Content-Length': '0'} 2022-09-09T22:39:31.696 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003cc33447f42eb66f-00631bc0a3-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.697 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:31.698 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:31.698 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:31.699 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:31.699 DEBUG:boto:Method: GET 2022-09-09T22:39:31.700 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:31.700 DEBUG:boto:Data: 2022-09-09T22:39:31.701 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.701 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:31.702 DEBUG:boto:Port: 8001 2022-09-09T22:39:31.702 DEBUG:boto:Params: {} 2022-09-09T22:39:31.703 DEBUG:boto:Token: None 2022-09-09T22:39:31.703 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/ 2022-09-09T22:39:31.704 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jyNXMbT3+Cm/MH5OBi0TyUft+cE= 2022-09-09T22:39:31.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jyNXMbT3+Cm/MH5OBi0TyUft+cE=', 'Content-Length': '0'} 2022-09-09T22:39:31.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fd5b7b910cd779d-00631bc0a3-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.710 DEBUG:boto:b'jkcgbj-17obj-test-zone31000falseobj-test-zone32022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:31.711 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:31.711 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:31.712 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:31.712 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:31.713 DEBUG:boto:Method: GET 2022-09-09T22:39:31.714 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:31.714 DEBUG:boto:Data: 2022-09-09T22:39:31.715 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.715 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:31.716 DEBUG:boto:Port: 8000 2022-09-09T22:39:31.716 DEBUG:boto:Params: {} 2022-09-09T22:39:31.717 DEBUG:boto:Token: None 2022-09-09T22:39:31.717 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/ 2022-09-09T22:39:31.718 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jyNXMbT3+Cm/MH5OBi0TyUft+cE= 2022-09-09T22:39:31.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jyNXMbT3+Cm/MH5OBi0TyUft+cE=', 'Content-Length': '0'} 2022-09-09T22:39:31.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051b8308be01706e3-00631bc0a3-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.725 DEBUG:boto:b'jkcgbj-17obj-test-zone31000falseobj-test-zone32022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:31.726 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:31.726 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:31.727 DEBUG:boto:Method: HEAD 2022-09-09T22:39:31.727 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3 2022-09-09T22:39:31.727 DEBUG:boto:Data: 2022-09-09T22:39:31.728 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.728 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:31.729 DEBUG:boto:Port: 8001 2022-09-09T22:39:31.729 DEBUG:boto:Params: {} 2022-09-09T22:39:31.730 DEBUG:boto:Token: None 2022-09-09T22:39:31.731 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/obj-test-zone3 2022-09-09T22:39:31.731 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:krSIAIrULtDwS0cL5EQHWJpx9Os= 2022-09-09T22:39:31.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:krSIAIrULtDwS0cL5EQHWJpx9Os=', 'Content-Length': '0'} 2022-09-09T22:39:31.735 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000039715bc49cbb142a-00631bc0a3-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.736 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:31.737 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:31.737 DEBUG:boto:Method: HEAD 2022-09-09T22:39:31.738 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3 2022-09-09T22:39:31.738 DEBUG:boto:Data: 2022-09-09T22:39:31.739 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.739 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:31.740 DEBUG:boto:Port: 8000 2022-09-09T22:39:31.740 DEBUG:boto:Params: {} 2022-09-09T22:39:31.741 DEBUG:boto:Token: None 2022-09-09T22:39:31.741 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/obj-test-zone3 2022-09-09T22:39:31.742 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:krSIAIrULtDwS0cL5EQHWJpx9Os= 2022-09-09T22:39:31.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:krSIAIrULtDwS0cL5EQHWJpx9Os=', 'Content-Length': '0'} 2022-09-09T22:39:31.746 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6266421ac59d464-00631bc0a3-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:39:31.748 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:31.748 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:31.749 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.749 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.750 DEBUG:boto:Method: GET 2022-09-09T22:39:31.750 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.751 DEBUG:boto:Data: 2022-09-09T22:39:31.751 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.752 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:31.752 DEBUG:boto:Port: 8001 2022-09-09T22:39:31.753 DEBUG:boto:Params: {} 2022-09-09T22:39:31.754 DEBUG:boto:Token: None 2022-09-09T22:39:31.754 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.755 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:G6Z4fXpCLkxGkGuESDzlI/ynyX4= 2022-09-09T22:39:31.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:G6Z4fXpCLkxGkGuESDzlI/ynyX4=', 'Content-Length': '0'} 2022-09-09T22:39:31.758 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054af951b540f89e9-00631bc0a3-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.802 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:31.803 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:31.803 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.804 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.804 DEBUG:boto:Method: GET 2022-09-09T22:39:31.805 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.805 DEBUG:boto:Data: 2022-09-09T22:39:31.806 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.806 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:31.807 DEBUG:boto:Port: 8000 2022-09-09T22:39:31.808 DEBUG:boto:Params: {} 2022-09-09T22:39:31.808 DEBUG:boto:Token: None 2022-09-09T22:39:31.809 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:31.809 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:G6Z4fXpCLkxGkGuESDzlI/ynyX4= 2022-09-09T22:39:31.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:G6Z4fXpCLkxGkGuESDzlI/ynyX4=', 'Content-Length': '0'} 2022-09-09T22:39:31.813 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a21fe1c52dc1c56e-00631bc0a3-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.814 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:31.814 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:31.815 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:31.815 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:31.816 DEBUG:boto:Method: GET 2022-09-09T22:39:31.816 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:31.817 DEBUG:boto:Data: 2022-09-09T22:39:31.817 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.818 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:31.819 DEBUG:boto:Port: 8001 2022-09-09T22:39:31.819 DEBUG:boto:Params: {} 2022-09-09T22:39:31.820 DEBUG:boto:Token: None 2022-09-09T22:39:31.820 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/ 2022-09-09T22:39:31.821 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jyNXMbT3+Cm/MH5OBi0TyUft+cE= 2022-09-09T22:39:31.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jyNXMbT3+Cm/MH5OBi0TyUft+cE=', 'Content-Length': '0'} 2022-09-09T22:39:31.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c917d37d521f77e3-00631bc0a3-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.828 DEBUG:boto:b'jkcgbj-17obj-test-zone31000falseobj-test-zone32022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:31.829 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:31.829 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:31.830 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:31.830 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:31.831 DEBUG:boto:Method: GET 2022-09-09T22:39:31.832 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:31.832 DEBUG:boto:Data: 2022-09-09T22:39:31.833 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.833 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:31.834 DEBUG:boto:Port: 8000 2022-09-09T22:39:31.834 DEBUG:boto:Params: {} 2022-09-09T22:39:31.835 DEBUG:boto:Token: None 2022-09-09T22:39:31.835 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/ 2022-09-09T22:39:31.836 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jyNXMbT3+Cm/MH5OBi0TyUft+cE= 2022-09-09T22:39:31.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jyNXMbT3+Cm/MH5OBi0TyUft+cE=', 'Content-Length': '0'} 2022-09-09T22:39:31.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7755d5996b491bd-00631bc0a3-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.843 DEBUG:boto:b'jkcgbj-17obj-test-zone31000falseobj-test-zone32022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:31.844 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-17 zones={test-zone3, test-zone1} 2022-09-09T22:39:31.845 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-17 zones={test-zone3, test-zone2} 2022-09-09T22:39:31.845 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:31.845 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:31.846 DEBUG:boto:Method: HEAD 2022-09-09T22:39:31.847 DEBUG:boto:Path: /jkcgbj-17/ 2022-09-09T22:39:31.847 DEBUG:boto:Data: 2022-09-09T22:39:31.848 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.848 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:31.849 DEBUG:boto:Port: 8001 2022-09-09T22:39:31.849 DEBUG:boto:Params: {} 2022-09-09T22:39:31.850 DEBUG:boto:Token: None 2022-09-09T22:39:31.850 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/ 2022-09-09T22:39:31.851 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:y3djCuUouJWrY9ogHUGzm+hLnHg= 2022-09-09T22:39:31.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:y3djCuUouJWrY9ogHUGzm+hLnHg=', 'Content-Length': '0'} 2022-09-09T22:39:31.856 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c98fc6ccd657ff6-00631bc0a3-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.856 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:31.857 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:31.857 DEBUG:boto:Method: HEAD 2022-09-09T22:39:31.858 DEBUG:boto:Path: /jkcgbj-17/ 2022-09-09T22:39:31.858 DEBUG:boto:Data: 2022-09-09T22:39:31.859 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.859 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:31.860 DEBUG:boto:Port: 8000 2022-09-09T22:39:31.860 DEBUG:boto:Params: {} 2022-09-09T22:39:31.861 DEBUG:boto:Token: None 2022-09-09T22:39:31.862 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/ 2022-09-09T22:39:31.862 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:y3djCuUouJWrY9ogHUGzm+hLnHg= 2022-09-09T22:39:31.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:y3djCuUouJWrY9ogHUGzm+hLnHg=', 'Content-Length': '0'} 2022-09-09T22:39:31.868 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000efc4b25376200eb6-00631bc0a3-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.868 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:39:31.869 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:31.870 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:31.870 DEBUG:boto:path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:31.871 DEBUG:boto:auth_path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:31.871 DEBUG:boto:Method: GET 2022-09-09T22:39:31.872 DEBUG:boto:Path: /jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:31.872 DEBUG:boto:Data: 2022-09-09T22:39:31.873 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.873 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:31.874 DEBUG:boto:Port: 8001 2022-09-09T22:39:31.874 DEBUG:boto:Params: {} 2022-09-09T22:39:31.875 DEBUG:boto:Token: None 2022-09-09T22:39:31.875 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/?versions 2022-09-09T22:39:31.876 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:NaN7Gkqcb2CamlCaSkGEXiEnqR0= 2022-09-09T22:39:31.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:NaN7Gkqcb2CamlCaSkGEXiEnqR0=', 'Content-Length': '0'} 2022-09-09T22:39:31.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a3be163d9a60186-00631bc0a3-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.884 DEBUG:boto:b'jkcgbj-17999falsedummyrcXzglYDyJxE.ArBmS3k944dyN-s9ZTtrue2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1uPhFM-4pEeqszoz0jfwkNTqs81x9txytrue2022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone217lUOlN45NLK2DSLxrYnqbOAg6jm2Tgtrue2022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone31RXW239HrAmSBD6KjNRB3qRTyY29OqZtrue2022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:31.885 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:39:31.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:39:31.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:39:31.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:39:31.886 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:39:31.887 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:31.887 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:31.888 DEBUG:boto:path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:31.888 DEBUG:boto:auth_path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:31.889 DEBUG:boto:Method: GET 2022-09-09T22:39:31.889 DEBUG:boto:Path: /jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:31.890 DEBUG:boto:Data: 2022-09-09T22:39:31.891 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.891 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:31.891 DEBUG:boto:Port: 8000 2022-09-09T22:39:31.892 DEBUG:boto:Params: {} 2022-09-09T22:39:31.893 DEBUG:boto:Token: None 2022-09-09T22:39:31.893 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/?versions 2022-09-09T22:39:31.894 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:NaN7Gkqcb2CamlCaSkGEXiEnqR0= 2022-09-09T22:39:31.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:NaN7Gkqcb2CamlCaSkGEXiEnqR0=', 'Content-Length': '0'} 2022-09-09T22:39:31.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d50e90c60a4e9643-00631bc0a3-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.901 DEBUG:boto:b'jkcgbj-17999falsedummyrcXzglYDyJxE.ArBmS3k944dyN-s9ZTtrue2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1uPhFM-4pEeqszoz0jfwkNTqs81x9txytrue2022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone217lUOlN45NLK2DSLxrYnqbOAg6jm2Tgtrue2022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone31RXW239HrAmSBD6KjNRB3qRTyY29OqZtrue2022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:31.902 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:39:31.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:39:31.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:39:31.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:39:31.904 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:31.905 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:31.905 DEBUG:boto:path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:31.906 DEBUG:boto:auth_path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:31.906 DEBUG:boto:Method: GET 2022-09-09T22:39:31.907 DEBUG:boto:Path: /jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:31.907 DEBUG:boto:Data: 2022-09-09T22:39:31.908 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.908 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:31.909 DEBUG:boto:Port: 8001 2022-09-09T22:39:31.909 DEBUG:boto:Params: {} 2022-09-09T22:39:31.910 DEBUG:boto:Token: None 2022-09-09T22:39:31.910 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/?versions 2022-09-09T22:39:31.911 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:NaN7Gkqcb2CamlCaSkGEXiEnqR0= 2022-09-09T22:39:31.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:NaN7Gkqcb2CamlCaSkGEXiEnqR0=', 'Content-Length': '0'} 2022-09-09T22:39:31.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000118031c3a885c09c-00631bc0a3-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.918 DEBUG:boto:b'jkcgbj-17999falsedummyrcXzglYDyJxE.ArBmS3k944dyN-s9ZTtrue2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1uPhFM-4pEeqszoz0jfwkNTqs81x9txytrue2022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone217lUOlN45NLK2DSLxrYnqbOAg6jm2Tgtrue2022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone31RXW239HrAmSBD6KjNRB3qRTyY29OqZtrue2022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:31.919 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:31.919 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:31.920 DEBUG:boto:path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:31.920 DEBUG:boto:auth_path=/jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:31.921 DEBUG:boto:Method: GET 2022-09-09T22:39:31.921 DEBUG:boto:Path: /jkcgbj-17/?versions&max-keys=999 2022-09-09T22:39:31.922 DEBUG:boto:Data: 2022-09-09T22:39:31.922 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.923 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:31.923 DEBUG:boto:Port: 8000 2022-09-09T22:39:31.924 DEBUG:boto:Params: {} 2022-09-09T22:39:31.924 DEBUG:boto:Token: None 2022-09-09T22:39:31.925 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/?versions 2022-09-09T22:39:31.925 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:NaN7Gkqcb2CamlCaSkGEXiEnqR0= 2022-09-09T22:39:31.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:NaN7Gkqcb2CamlCaSkGEXiEnqR0=', 'Content-Length': '0'} 2022-09-09T22:39:31.932 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061e633ae98fee89a-00631bc0a3-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.933 DEBUG:boto:b'jkcgbj-17999falsedummyrcXzglYDyJxE.ArBmS3k944dyN-s9ZTtrue2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1uPhFM-4pEeqszoz0jfwkNTqs81x9txytrue2022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone217lUOlN45NLK2DSLxrYnqbOAg6jm2Tgtrue2022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone31RXW239HrAmSBD6KjNRB3qRTyY29OqZtrue2022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:31.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:39:31.934 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:39:31.935 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:39:31.935 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:31.936 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:31.936 DEBUG:boto:Method: GET 2022-09-09T22:39:31.937 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:31.937 DEBUG:boto:Data: 2022-09-09T22:39:31.938 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.938 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:31.939 DEBUG:boto:Port: 8001 2022-09-09T22:39:31.939 DEBUG:boto:Params: {} 2022-09-09T22:39:31.940 DEBUG:boto:Token: None 2022-09-09T22:39:31.941 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:31.941 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KlMjjaa/LukMYooHpZERX9k7/WA= 2022-09-09T22:39:31.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KlMjjaa/LukMYooHpZERX9k7/WA=', 'Content-Length': '0'} 2022-09-09T22:39:31.945 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c5fd981b80276606-00631bc0a3-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.946 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:39:31.946 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:39:31.947 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:31.947 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:31.948 DEBUG:boto:Method: GET 2022-09-09T22:39:31.948 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:31.949 DEBUG:boto:Data: 2022-09-09T22:39:31.949 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.950 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:31.950 DEBUG:boto:Port: 8000 2022-09-09T22:39:31.951 DEBUG:boto:Params: {} 2022-09-09T22:39:31.951 DEBUG:boto:Token: None 2022-09-09T22:39:31.952 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:31.953 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KlMjjaa/LukMYooHpZERX9k7/WA= 2022-09-09T22:39:31.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KlMjjaa/LukMYooHpZERX9k7/WA=', 'Content-Length': '0'} 2022-09-09T22:39:31.956 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001053618c63056a82-00631bc0a3-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.957 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:39:31.957 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:39:31.958 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:31.958 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:31.959 DEBUG:boto:Method: HEAD 2022-09-09T22:39:31.959 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:31.960 DEBUG:boto:Data: 2022-09-09T22:39:31.960 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.961 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:31.961 DEBUG:boto:Port: 8001 2022-09-09T22:39:31.962 DEBUG:boto:Params: {} 2022-09-09T22:39:31.962 DEBUG:boto:Token: None 2022-09-09T22:39:31.963 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:31.963 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:efwL9F2TE6Jbyu8kK39GY92jvlI= 2022-09-09T22:39:31.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:efwL9F2TE6Jbyu8kK39GY92jvlI=', 'Content-Length': '0'} 2022-09-09T22:39:31.967 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000773eae5d0b59c070-00631bc0a3-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.967 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:39:31.968 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:39:31.968 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:31.969 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:31.969 DEBUG:boto:Method: HEAD 2022-09-09T22:39:31.970 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:31.970 DEBUG:boto:Data: 2022-09-09T22:39:31.971 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.971 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:31.972 DEBUG:boto:Port: 8000 2022-09-09T22:39:31.972 DEBUG:boto:Params: {} 2022-09-09T22:39:31.973 DEBUG:boto:Token: None 2022-09-09T22:39:31.973 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:31.974 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:efwL9F2TE6Jbyu8kK39GY92jvlI= 2022-09-09T22:39:31.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:efwL9F2TE6Jbyu8kK39GY92jvlI=', 'Content-Length': '0'} 2022-09-09T22:39:31.977 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a35c04154eb2a1d2-00631bc0a3-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:39:31.978 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:39:31.979 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:39:31.979 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:31.980 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:31.980 DEBUG:boto:Method: GET 2022-09-09T22:39:31.981 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:31.981 DEBUG:boto:Data: 2022-09-09T22:39:31.982 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.982 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:31.983 DEBUG:boto:Port: 8001 2022-09-09T22:39:31.983 DEBUG:boto:Params: {} 2022-09-09T22:39:31.984 DEBUG:boto:Token: None 2022-09-09T22:39:31.985 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:31.985 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KlMjjaa/LukMYooHpZERX9k7/WA= 2022-09-09T22:39:31.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KlMjjaa/LukMYooHpZERX9k7/WA=', 'Content-Length': '0'} 2022-09-09T22:39:31.988 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000737ef7025b1b6cef-00631bc0a3-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:31.989 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:39:31.990 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:39:31.990 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:31.991 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:31.991 DEBUG:boto:Method: GET 2022-09-09T22:39:31.992 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:31.992 DEBUG:boto:Data: 2022-09-09T22:39:31.993 DEBUG:boto:Headers: {} 2022-09-09T22:39:31.993 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:31.994 DEBUG:boto:Port: 8000 2022-09-09T22:39:31.994 DEBUG:boto:Params: {} 2022-09-09T22:39:31.995 DEBUG:boto:Token: None 2022-09-09T22:39:31.996 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:31 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:31.996 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KlMjjaa/LukMYooHpZERX9k7/WA= 2022-09-09T22:39:31.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:31 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KlMjjaa/LukMYooHpZERX9k7/WA=', 'Content-Length': '0'} 2022-09-09T22:39:32.000 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000df6b26cbe9d710a0-00631bc0a3-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.000 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:32.001 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:32.001 DEBUG:boto:path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:39:32.002 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:39:32.002 DEBUG:boto:Method: GET 2022-09-09T22:39:32.003 DEBUG:boto:Path: /jkcgbj-17/?prefix=dummy 2022-09-09T22:39:32.003 DEBUG:boto:Data: 2022-09-09T22:39:32.004 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.004 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:32.005 DEBUG:boto:Port: 8001 2022-09-09T22:39:32.005 DEBUG:boto:Params: {} 2022-09-09T22:39:32.006 DEBUG:boto:Token: None 2022-09-09T22:39:32.007 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/ 2022-09-09T22:39:32.007 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o= 2022-09-09T22:39:32.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o=', 'Content-Length': '0'} 2022-09-09T22:39:32.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c36d708d7e47440-00631bc0a4-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.062 DEBUG:boto:b'jkcgbj-17dummy1000falsedummy2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:32.063 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:32.064 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:32.064 DEBUG:boto:path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:39:32.065 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:39:32.065 DEBUG:boto:Method: GET 2022-09-09T22:39:32.066 DEBUG:boto:Path: /jkcgbj-17/?prefix=dummy 2022-09-09T22:39:32.067 DEBUG:boto:Data: 2022-09-09T22:39:32.067 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.068 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:32.068 DEBUG:boto:Port: 8000 2022-09-09T22:39:32.069 DEBUG:boto:Params: {} 2022-09-09T22:39:32.070 DEBUG:boto:Token: None 2022-09-09T22:39:32.071 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/ 2022-09-09T22:39:32.072 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o= 2022-09-09T22:39:32.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o=', 'Content-Length': '0'} 2022-09-09T22:39:32.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d68c85896cf762ce-00631bc0a4-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.081 DEBUG:boto:b'jkcgbj-17dummy1000falsedummy2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:32.081 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:39:32.082 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:39:32.083 DEBUG:boto:Method: HEAD 2022-09-09T22:39:32.083 DEBUG:boto:Path: /jkcgbj-17/dummy 2022-09-09T22:39:32.084 DEBUG:boto:Data: 2022-09-09T22:39:32.084 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.085 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:32.085 DEBUG:boto:Port: 8001 2022-09-09T22:39:32.086 DEBUG:boto:Params: {} 2022-09-09T22:39:32.086 DEBUG:boto:Token: None 2022-09-09T22:39:32.087 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/dummy 2022-09-09T22:39:32.088 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:t4rxOsbqF0IRYGJWsi+eRNJQxFw= 2022-09-09T22:39:32.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:t4rxOsbqF0IRYGJWsi+eRNJQxFw=', 'Content-Length': '0'} 2022-09-09T22:39:32.093 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c8f7bd165d628aa2-00631bc0a4-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.094 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:39:32.095 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:39:32.095 DEBUG:boto:Method: HEAD 2022-09-09T22:39:32.096 DEBUG:boto:Path: /jkcgbj-17/dummy 2022-09-09T22:39:32.097 DEBUG:boto:Data: 2022-09-09T22:39:32.097 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.098 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:32.098 DEBUG:boto:Port: 8000 2022-09-09T22:39:32.099 DEBUG:boto:Params: {} 2022-09-09T22:39:32.099 DEBUG:boto:Token: None 2022-09-09T22:39:32.100 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/dummy 2022-09-09T22:39:32.100 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:t4rxOsbqF0IRYGJWsi+eRNJQxFw= 2022-09-09T22:39:32.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:t4rxOsbqF0IRYGJWsi+eRNJQxFw=', 'Content-Length': '0'} 2022-09-09T22:39:32.105 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007e3249bfb6485b30-00631bc0a4-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:39:32.106 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:39:32.107 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:39:32.107 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:32.108 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:32.108 DEBUG:boto:Method: GET 2022-09-09T22:39:32.109 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:32.109 DEBUG:boto:Data: 2022-09-09T22:39:32.110 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.110 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:32.111 DEBUG:boto:Port: 8001 2022-09-09T22:39:32.111 DEBUG:boto:Params: {} 2022-09-09T22:39:32.112 DEBUG:boto:Token: None 2022-09-09T22:39:32.113 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:32.113 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:htnzYrM8hu9twZfWTslly0A8LJc= 2022-09-09T22:39:32.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:htnzYrM8hu9twZfWTslly0A8LJc=', 'Content-Length': '0'} 2022-09-09T22:39:32.117 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008d35bd55d16e193f-00631bc0a4-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.118 DEBUG:boto:path=/jkcgbj-17/dummy 2022-09-09T22:39:32.118 DEBUG:boto:auth_path=/jkcgbj-17/dummy 2022-09-09T22:39:32.119 DEBUG:boto:path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:32.120 DEBUG:boto:auth_path=/jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:32.120 DEBUG:boto:Method: GET 2022-09-09T22:39:32.121 DEBUG:boto:Path: /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:32.121 DEBUG:boto:Data: 2022-09-09T22:39:32.122 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.122 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:32.123 DEBUG:boto:Port: 8000 2022-09-09T22:39:32.123 DEBUG:boto:Params: {} 2022-09-09T22:39:32.124 DEBUG:boto:Token: None 2022-09-09T22:39:32.124 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/dummy?versionId=rcXzglYDyJxE.ArBmS3k944dyN-s9ZT 2022-09-09T22:39:32.125 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:htnzYrM8hu9twZfWTslly0A8LJc= 2022-09-09T22:39:32.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:htnzYrM8hu9twZfWTslly0A8LJc=', 'Content-Length': '0'} 2022-09-09T22:39:32.129 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:05 GMT'), ('x-amz-version-id', 'rcXzglYDyJxE.ArBmS3k944dyN-s9ZT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000063849ef1a1ad34a1-00631bc0a4-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.130 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:32.130 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:32.131 DEBUG:boto:path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:39:32.131 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:39:32.132 DEBUG:boto:Method: GET 2022-09-09T22:39:32.132 DEBUG:boto:Path: /jkcgbj-17/?prefix=dummy 2022-09-09T22:39:32.133 DEBUG:boto:Data: 2022-09-09T22:39:32.133 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.134 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:32.135 DEBUG:boto:Port: 8001 2022-09-09T22:39:32.135 DEBUG:boto:Params: {} 2022-09-09T22:39:32.136 DEBUG:boto:Token: None 2022-09-09T22:39:32.136 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/ 2022-09-09T22:39:32.137 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o= 2022-09-09T22:39:32.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o=', 'Content-Length': '0'} 2022-09-09T22:39:32.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac6e36113d8911f8-00631bc0a4-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.143 DEBUG:boto:b'jkcgbj-17dummy1000falsedummy2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:32.144 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:32.145 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:32.145 DEBUG:boto:path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:39:32.146 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=dummy 2022-09-09T22:39:32.147 DEBUG:boto:Method: GET 2022-09-09T22:39:32.147 DEBUG:boto:Path: /jkcgbj-17/?prefix=dummy 2022-09-09T22:39:32.148 DEBUG:boto:Data: 2022-09-09T22:39:32.148 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.149 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:32.149 DEBUG:boto:Port: 8000 2022-09-09T22:39:32.150 DEBUG:boto:Params: {} 2022-09-09T22:39:32.150 DEBUG:boto:Token: None 2022-09-09T22:39:32.151 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/ 2022-09-09T22:39:32.151 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o= 2022-09-09T22:39:32.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o=', 'Content-Length': '0'} 2022-09-09T22:39:32.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000599ab97c13a57459-00631bc0a4-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.158 DEBUG:boto:b'jkcgbj-17dummy1000falsedummy2022-09-09T22:35:05.583Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:32.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:39:32.160 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:32.160 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:32.161 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.161 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.162 DEBUG:boto:Method: GET 2022-09-09T22:39:32.162 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.163 DEBUG:boto:Data: 2022-09-09T22:39:32.163 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.164 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:32.164 DEBUG:boto:Port: 8001 2022-09-09T22:39:32.165 DEBUG:boto:Params: {} 2022-09-09T22:39:32.165 DEBUG:boto:Token: None 2022-09-09T22:39:32.166 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.167 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:EL0yjVuP63I1KJHCs8bsGXpr4Vw= 2022-09-09T22:39:32.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:EL0yjVuP63I1KJHCs8bsGXpr4Vw=', 'Content-Length': '0'} 2022-09-09T22:39:32.170 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f31055291e50644e-00631bc0a4-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.171 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:32.172 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:32.172 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.173 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.173 DEBUG:boto:Method: GET 2022-09-09T22:39:32.174 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.175 DEBUG:boto:Data: 2022-09-09T22:39:32.175 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.176 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:32.176 DEBUG:boto:Port: 8000 2022-09-09T22:39:32.177 DEBUG:boto:Params: {} 2022-09-09T22:39:32.177 DEBUG:boto:Token: None 2022-09-09T22:39:32.178 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.179 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:EL0yjVuP63I1KJHCs8bsGXpr4Vw= 2022-09-09T22:39:32.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:EL0yjVuP63I1KJHCs8bsGXpr4Vw=', 'Content-Length': '0'} 2022-09-09T22:39:32.183 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004e9b38a826e0c39-00631bc0a4-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.184 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:32.184 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:32.185 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.185 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.186 DEBUG:boto:Method: HEAD 2022-09-09T22:39:32.186 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.187 DEBUG:boto:Data: 2022-09-09T22:39:32.187 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.188 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:32.188 DEBUG:boto:Port: 8001 2022-09-09T22:39:32.189 DEBUG:boto:Params: {} 2022-09-09T22:39:32.189 DEBUG:boto:Token: None 2022-09-09T22:39:32.190 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.190 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7WlOZTEBsSgkHUEKbMuyayhg51g= 2022-09-09T22:39:32.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7WlOZTEBsSgkHUEKbMuyayhg51g=', 'Content-Length': '0'} 2022-09-09T22:39:32.194 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000074c5ad4b68d37a58-00631bc0a4-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.195 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:32.195 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:32.196 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.196 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.197 DEBUG:boto:Method: HEAD 2022-09-09T22:39:32.198 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.198 DEBUG:boto:Data: 2022-09-09T22:39:32.199 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.199 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:32.200 DEBUG:boto:Port: 8000 2022-09-09T22:39:32.200 DEBUG:boto:Params: {} 2022-09-09T22:39:32.201 DEBUG:boto:Token: None 2022-09-09T22:39:32.201 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.202 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7WlOZTEBsSgkHUEKbMuyayhg51g= 2022-09-09T22:39:32.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7WlOZTEBsSgkHUEKbMuyayhg51g=', 'Content-Length': '0'} 2022-09-09T22:39:32.205 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000006bf3828cd033dc6-00631bc0a4-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:39:32.207 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:32.207 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:32.208 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.208 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.209 DEBUG:boto:Method: GET 2022-09-09T22:39:32.209 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.210 DEBUG:boto:Data: 2022-09-09T22:39:32.210 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.211 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:32.211 DEBUG:boto:Port: 8001 2022-09-09T22:39:32.212 DEBUG:boto:Params: {} 2022-09-09T22:39:32.212 DEBUG:boto:Token: None 2022-09-09T22:39:32.213 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.214 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:EL0yjVuP63I1KJHCs8bsGXpr4Vw= 2022-09-09T22:39:32.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:EL0yjVuP63I1KJHCs8bsGXpr4Vw=', 'Content-Length': '0'} 2022-09-09T22:39:32.217 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007271383b04822faf-00631bc0a4-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.258 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:32.259 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:32.259 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.260 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.261 DEBUG:boto:Method: GET 2022-09-09T22:39:32.261 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.262 DEBUG:boto:Data: 2022-09-09T22:39:32.262 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.263 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:32.263 DEBUG:boto:Port: 8000 2022-09-09T22:39:32.264 DEBUG:boto:Params: {} 2022-09-09T22:39:32.264 DEBUG:boto:Token: None 2022-09-09T22:39:32.265 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.266 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:EL0yjVuP63I1KJHCs8bsGXpr4Vw= 2022-09-09T22:39:32.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:EL0yjVuP63I1KJHCs8bsGXpr4Vw=', 'Content-Length': '0'} 2022-09-09T22:39:32.269 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e16122435a9fe9ae-00631bc0a4-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.270 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:32.271 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:32.271 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:32.272 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:32.273 DEBUG:boto:Method: GET 2022-09-09T22:39:32.273 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:32.274 DEBUG:boto:Data: 2022-09-09T22:39:32.274 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.275 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:32.275 DEBUG:boto:Port: 8001 2022-09-09T22:39:32.276 DEBUG:boto:Params: {} 2022-09-09T22:39:32.276 DEBUG:boto:Token: None 2022-09-09T22:39:32.277 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/ 2022-09-09T22:39:32.277 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o= 2022-09-09T22:39:32.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o=', 'Content-Length': '0'} 2022-09-09T22:39:32.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011c1449607f6e938-00631bc0a4-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.285 DEBUG:boto:b'jkcgbj-17obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:32.286 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:32.286 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:32.287 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:32.287 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:32.288 DEBUG:boto:Method: GET 2022-09-09T22:39:32.289 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:32.289 DEBUG:boto:Data: 2022-09-09T22:39:32.290 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.290 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:32.291 DEBUG:boto:Port: 8000 2022-09-09T22:39:32.291 DEBUG:boto:Params: {} 2022-09-09T22:39:32.292 DEBUG:boto:Token: None 2022-09-09T22:39:32.292 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/ 2022-09-09T22:39:32.293 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o= 2022-09-09T22:39:32.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o=', 'Content-Length': '0'} 2022-09-09T22:39:32.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000598bbcfc641bfde-00631bc0a4-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.301 DEBUG:boto:b'jkcgbj-17obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:32.302 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:32.302 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:32.303 DEBUG:boto:Method: HEAD 2022-09-09T22:39:32.303 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1 2022-09-09T22:39:32.304 DEBUG:boto:Data: 2022-09-09T22:39:32.304 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.305 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:32.305 DEBUG:boto:Port: 8001 2022-09-09T22:39:32.306 DEBUG:boto:Params: {} 2022-09-09T22:39:32.306 DEBUG:boto:Token: None 2022-09-09T22:39:32.307 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/obj-test-zone1 2022-09-09T22:39:32.307 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YS+3CTJ7E/PNtYYKywIeYH6xW/I= 2022-09-09T22:39:32.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YS+3CTJ7E/PNtYYKywIeYH6xW/I=', 'Content-Length': '0'} 2022-09-09T22:39:32.312 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d0d3b9b9b54102cf-00631bc0a4-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.313 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:32.313 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:32.314 DEBUG:boto:Method: HEAD 2022-09-09T22:39:32.314 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1 2022-09-09T22:39:32.315 DEBUG:boto:Data: 2022-09-09T22:39:32.315 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.316 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:32.316 DEBUG:boto:Port: 8000 2022-09-09T22:39:32.317 DEBUG:boto:Params: {} 2022-09-09T22:39:32.317 DEBUG:boto:Token: None 2022-09-09T22:39:32.318 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/obj-test-zone1 2022-09-09T22:39:32.319 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YS+3CTJ7E/PNtYYKywIeYH6xW/I= 2022-09-09T22:39:32.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YS+3CTJ7E/PNtYYKywIeYH6xW/I=', 'Content-Length': '0'} 2022-09-09T22:39:32.323 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060cb4a1001ab4427-00631bc0a4-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:39:32.324 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:32.325 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:32.326 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.326 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.327 DEBUG:boto:Method: GET 2022-09-09T22:39:32.327 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.328 DEBUG:boto:Data: 2022-09-09T22:39:32.328 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.329 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:32.329 DEBUG:boto:Port: 8001 2022-09-09T22:39:32.330 DEBUG:boto:Params: {} 2022-09-09T22:39:32.330 DEBUG:boto:Token: None 2022-09-09T22:39:32.331 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.332 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:EL0yjVuP63I1KJHCs8bsGXpr4Vw= 2022-09-09T22:39:32.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:EL0yjVuP63I1KJHCs8bsGXpr4Vw=', 'Content-Length': '0'} 2022-09-09T22:39:32.335 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000631e2f84190bf611-00631bc0a4-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.378 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:32.379 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1 2022-09-09T22:39:32.379 DEBUG:boto:path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.380 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.381 DEBUG:boto:Method: GET 2022-09-09T22:39:32.381 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.382 DEBUG:boto:Data: 2022-09-09T22:39:32.382 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.383 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:32.384 DEBUG:boto:Port: 8000 2022-09-09T22:39:32.384 DEBUG:boto:Params: {} 2022-09-09T22:39:32.385 DEBUG:boto:Token: None 2022-09-09T22:39:32.385 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/obj-test-zone1?versionId=uPhFM-4pEeqszoz0jfwkNTqs81x9txy 2022-09-09T22:39:32.386 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:EL0yjVuP63I1KJHCs8bsGXpr4Vw= 2022-09-09T22:39:32.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:EL0yjVuP63I1KJHCs8bsGXpr4Vw=', 'Content-Length': '0'} 2022-09-09T22:39:32.390 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'uPhFM-4pEeqszoz0jfwkNTqs81x9txy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009121449079b623ad-00631bc0a4-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.391 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:32.391 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:32.392 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:32.393 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:32.393 DEBUG:boto:Method: GET 2022-09-09T22:39:32.394 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:32.394 DEBUG:boto:Data: 2022-09-09T22:39:32.395 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.395 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:32.396 DEBUG:boto:Port: 8001 2022-09-09T22:39:32.396 DEBUG:boto:Params: {} 2022-09-09T22:39:32.397 DEBUG:boto:Token: None 2022-09-09T22:39:32.398 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/ 2022-09-09T22:39:32.398 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o= 2022-09-09T22:39:32.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o=', 'Content-Length': '0'} 2022-09-09T22:39:32.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000674f051ecc77c886-00631bc0a4-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.406 DEBUG:boto:b'jkcgbj-17obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:32.406 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:32.407 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:32.408 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:32.408 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:32.409 DEBUG:boto:Method: GET 2022-09-09T22:39:32.409 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone1 2022-09-09T22:39:32.410 DEBUG:boto:Data: 2022-09-09T22:39:32.410 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.411 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:32.412 DEBUG:boto:Port: 8000 2022-09-09T22:39:32.412 DEBUG:boto:Params: {} 2022-09-09T22:39:32.413 DEBUG:boto:Token: None 2022-09-09T22:39:32.413 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/ 2022-09-09T22:39:32.414 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o= 2022-09-09T22:39:32.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o=', 'Content-Length': '0'} 2022-09-09T22:39:32.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb5c990a2b17bbf9-00631bc0a4-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.421 DEBUG:boto:b'jkcgbj-17obj-test-zone11000falseobj-test-zone12022-09-09T22:36:01.948Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:32.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:39:32.422 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:32.423 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:32.423 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.424 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.424 DEBUG:boto:Method: GET 2022-09-09T22:39:32.425 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.426 DEBUG:boto:Data: 2022-09-09T22:39:32.426 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.427 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:32.427 DEBUG:boto:Port: 8001 2022-09-09T22:39:32.428 DEBUG:boto:Params: {} 2022-09-09T22:39:32.428 DEBUG:boto:Token: None 2022-09-09T22:39:32.429 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.430 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wSgiuEq5ktaLZK+VpialKfM4wOw= 2022-09-09T22:39:32.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wSgiuEq5ktaLZK+VpialKfM4wOw=', 'Content-Length': '0'} 2022-09-09T22:39:32.434 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc47d506cec94967-00631bc0a4-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.435 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:32.435 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:32.436 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.436 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.437 DEBUG:boto:Method: GET 2022-09-09T22:39:32.437 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.438 DEBUG:boto:Data: 2022-09-09T22:39:32.439 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.439 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:32.440 DEBUG:boto:Port: 8000 2022-09-09T22:39:32.440 DEBUG:boto:Params: {} 2022-09-09T22:39:32.441 DEBUG:boto:Token: None 2022-09-09T22:39:32.442 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.442 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wSgiuEq5ktaLZK+VpialKfM4wOw= 2022-09-09T22:39:32.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wSgiuEq5ktaLZK+VpialKfM4wOw=', 'Content-Length': '0'} 2022-09-09T22:39:32.446 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c131f6d069c8d03-00631bc0a4-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.490 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:32.491 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:32.491 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.492 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.492 DEBUG:boto:Method: HEAD 2022-09-09T22:39:32.493 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.493 DEBUG:boto:Data: 2022-09-09T22:39:32.494 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.495 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:32.495 DEBUG:boto:Port: 8001 2022-09-09T22:39:32.496 DEBUG:boto:Params: {} 2022-09-09T22:39:32.496 DEBUG:boto:Token: None 2022-09-09T22:39:32.497 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.498 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Iso1tp9ebg8EG/k4iRfc802+agk= 2022-09-09T22:39:32.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Iso1tp9ebg8EG/k4iRfc802+agk=', 'Content-Length': '0'} 2022-09-09T22:39:32.503 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f74ae19ccd4109e-00631bc0a4-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.503 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:32.504 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:32.505 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.505 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.506 DEBUG:boto:Method: HEAD 2022-09-09T22:39:32.506 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.507 DEBUG:boto:Data: 2022-09-09T22:39:32.507 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.508 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:32.508 DEBUG:boto:Port: 8000 2022-09-09T22:39:32.509 DEBUG:boto:Params: {} 2022-09-09T22:39:32.509 DEBUG:boto:Token: None 2022-09-09T22:39:32.510 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.510 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Iso1tp9ebg8EG/k4iRfc802+agk= 2022-09-09T22:39:32.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Iso1tp9ebg8EG/k4iRfc802+agk=', 'Content-Length': '0'} 2022-09-09T22:39:32.514 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000703e019c2cc10677-00631bc0a4-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:39:32.515 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:32.515 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:32.516 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.517 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.517 DEBUG:boto:Method: GET 2022-09-09T22:39:32.518 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.518 DEBUG:boto:Data: 2022-09-09T22:39:32.519 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.519 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:32.520 DEBUG:boto:Port: 8001 2022-09-09T22:39:32.520 DEBUG:boto:Params: {} 2022-09-09T22:39:32.521 DEBUG:boto:Token: None 2022-09-09T22:39:32.522 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.523 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wSgiuEq5ktaLZK+VpialKfM4wOw= 2022-09-09T22:39:32.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wSgiuEq5ktaLZK+VpialKfM4wOw=', 'Content-Length': '0'} 2022-09-09T22:39:32.528 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091b21f59438a3d5c-00631bc0a4-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.570 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:32.571 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:32.571 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.572 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.572 DEBUG:boto:Method: GET 2022-09-09T22:39:32.572 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.573 DEBUG:boto:Data: 2022-09-09T22:39:32.573 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.574 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:32.575 DEBUG:boto:Port: 8000 2022-09-09T22:39:32.575 DEBUG:boto:Params: {} 2022-09-09T22:39:32.576 DEBUG:boto:Token: None 2022-09-09T22:39:32.577 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.577 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wSgiuEq5ktaLZK+VpialKfM4wOw= 2022-09-09T22:39:32.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wSgiuEq5ktaLZK+VpialKfM4wOw=', 'Content-Length': '0'} 2022-09-09T22:39:32.581 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000445f796f9ebe7575-00631bc0a4-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.582 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:32.582 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:32.583 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:32.583 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:32.584 DEBUG:boto:Method: GET 2022-09-09T22:39:32.584 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:32.585 DEBUG:boto:Data: 2022-09-09T22:39:32.585 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.586 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:32.586 DEBUG:boto:Port: 8001 2022-09-09T22:39:32.587 DEBUG:boto:Params: {} 2022-09-09T22:39:32.588 DEBUG:boto:Token: None 2022-09-09T22:39:32.588 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/ 2022-09-09T22:39:32.589 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o= 2022-09-09T22:39:32.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o=', 'Content-Length': '0'} 2022-09-09T22:39:32.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d8d2ee629cf9008-00631bc0a4-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.596 DEBUG:boto:b'jkcgbj-17obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:32.597 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:32.597 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:32.598 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:32.598 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:32.599 DEBUG:boto:Method: GET 2022-09-09T22:39:32.599 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:32.600 DEBUG:boto:Data: 2022-09-09T22:39:32.600 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.600 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:32.601 DEBUG:boto:Port: 8000 2022-09-09T22:39:32.601 DEBUG:boto:Params: {} 2022-09-09T22:39:32.602 DEBUG:boto:Token: None 2022-09-09T22:39:32.603 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/ 2022-09-09T22:39:32.603 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o= 2022-09-09T22:39:32.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o=', 'Content-Length': '0'} 2022-09-09T22:39:32.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3b0ce46aaad1d81-00631bc0a4-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.612 DEBUG:boto:b'jkcgbj-17obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:32.613 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:32.613 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:32.614 DEBUG:boto:Method: HEAD 2022-09-09T22:39:32.614 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2 2022-09-09T22:39:32.615 DEBUG:boto:Data: 2022-09-09T22:39:32.615 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.616 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:32.616 DEBUG:boto:Port: 8001 2022-09-09T22:39:32.617 DEBUG:boto:Params: {} 2022-09-09T22:39:32.617 DEBUG:boto:Token: None 2022-09-09T22:39:32.618 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/obj-test-zone2 2022-09-09T22:39:32.618 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2m4GaN3x738vkcrqy17Su+lXh8g= 2022-09-09T22:39:32.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2m4GaN3x738vkcrqy17Su+lXh8g=', 'Content-Length': '0'} 2022-09-09T22:39:32.623 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e40fc5e8fed92c6b-00631bc0a4-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.623 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:32.624 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:32.625 DEBUG:boto:Method: HEAD 2022-09-09T22:39:32.625 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2 2022-09-09T22:39:32.626 DEBUG:boto:Data: 2022-09-09T22:39:32.626 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.627 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:32.627 DEBUG:boto:Port: 8000 2022-09-09T22:39:32.628 DEBUG:boto:Params: {} 2022-09-09T22:39:32.628 DEBUG:boto:Token: None 2022-09-09T22:39:32.629 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/obj-test-zone2 2022-09-09T22:39:32.630 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2m4GaN3x738vkcrqy17Su+lXh8g= 2022-09-09T22:39:32.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2m4GaN3x738vkcrqy17Su+lXh8g=', 'Content-Length': '0'} 2022-09-09T22:39:32.633 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e51e0ce31b4b9fc4-00631bc0a4-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:39:32.634 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:32.635 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:32.636 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.637 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.638 DEBUG:boto:Method: GET 2022-09-09T22:39:32.639 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.640 DEBUG:boto:Data: 2022-09-09T22:39:32.641 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.642 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:32.643 DEBUG:boto:Port: 8001 2022-09-09T22:39:32.644 DEBUG:boto:Params: {} 2022-09-09T22:39:32.645 DEBUG:boto:Token: None 2022-09-09T22:39:32.646 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.647 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wSgiuEq5ktaLZK+VpialKfM4wOw= 2022-09-09T22:39:32.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wSgiuEq5ktaLZK+VpialKfM4wOw=', 'Content-Length': '0'} 2022-09-09T22:39:32.652 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000103cb659c2dd1652-00631bc0a4-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.698 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:32.699 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2 2022-09-09T22:39:32.699 DEBUG:boto:path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.700 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.700 DEBUG:boto:Method: GET 2022-09-09T22:39:32.701 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.701 DEBUG:boto:Data: 2022-09-09T22:39:32.702 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.703 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:32.703 DEBUG:boto:Port: 8000 2022-09-09T22:39:32.704 DEBUG:boto:Params: {} 2022-09-09T22:39:32.704 DEBUG:boto:Token: None 2022-09-09T22:39:32.705 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/obj-test-zone2?versionId=17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg 2022-09-09T22:39:32.706 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wSgiuEq5ktaLZK+VpialKfM4wOw= 2022-09-09T22:39:32.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wSgiuEq5ktaLZK+VpialKfM4wOw=', 'Content-Length': '0'} 2022-09-09T22:39:32.709 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '17lUOlN45NLK2DSLxrYnqbOAg6jm2Tg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1e1e0f7816c71b1-00631bc0a4-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.710 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:32.711 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:32.711 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:32.712 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:32.712 DEBUG:boto:Method: GET 2022-09-09T22:39:32.713 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:32.714 DEBUG:boto:Data: 2022-09-09T22:39:32.714 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.715 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:32.715 DEBUG:boto:Port: 8001 2022-09-09T22:39:32.716 DEBUG:boto:Params: {} 2022-09-09T22:39:32.716 DEBUG:boto:Token: None 2022-09-09T22:39:32.717 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/ 2022-09-09T22:39:32.717 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o= 2022-09-09T22:39:32.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o=', 'Content-Length': '0'} 2022-09-09T22:39:32.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c5985a58146be41-00631bc0a4-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.725 DEBUG:boto:b'jkcgbj-17obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:32.726 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:32.726 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:32.727 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:32.727 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:32.728 DEBUG:boto:Method: GET 2022-09-09T22:39:32.728 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone2 2022-09-09T22:39:32.729 DEBUG:boto:Data: 2022-09-09T22:39:32.729 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.730 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:32.730 DEBUG:boto:Port: 8000 2022-09-09T22:39:32.731 DEBUG:boto:Params: {} 2022-09-09T22:39:32.732 DEBUG:boto:Token: None 2022-09-09T22:39:32.732 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/ 2022-09-09T22:39:32.733 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o= 2022-09-09T22:39:32.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o=', 'Content-Length': '0'} 2022-09-09T22:39:32.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002df87b1170ce3a02-00631bc0a4-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.740 DEBUG:boto:b'jkcgbj-17obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:32.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:39:32.742 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:32.742 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:32.743 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.743 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.744 DEBUG:boto:Method: GET 2022-09-09T22:39:32.744 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.745 DEBUG:boto:Data: 2022-09-09T22:39:32.745 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.746 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:32.746 DEBUG:boto:Port: 8001 2022-09-09T22:39:32.747 DEBUG:boto:Params: {} 2022-09-09T22:39:32.747 DEBUG:boto:Token: None 2022-09-09T22:39:32.748 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.749 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:MIv1BkaJUrV8Fs1mFcuYDRdcJxs= 2022-09-09T22:39:32.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:MIv1BkaJUrV8Fs1mFcuYDRdcJxs=', 'Content-Length': '0'} 2022-09-09T22:39:32.752 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000190a745a96904cfe-00631bc0a4-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.753 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:32.754 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:32.755 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.755 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.756 DEBUG:boto:Method: GET 2022-09-09T22:39:32.756 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.757 DEBUG:boto:Data: 2022-09-09T22:39:32.757 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.758 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:32.758 DEBUG:boto:Port: 8000 2022-09-09T22:39:32.759 DEBUG:boto:Params: {} 2022-09-09T22:39:32.759 DEBUG:boto:Token: None 2022-09-09T22:39:32.760 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.761 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:MIv1BkaJUrV8Fs1mFcuYDRdcJxs= 2022-09-09T22:39:32.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:MIv1BkaJUrV8Fs1mFcuYDRdcJxs=', 'Content-Length': '0'} 2022-09-09T22:39:32.764 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000562f45d31a6686fd-00631bc0a4-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.806 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:32.807 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:32.807 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.808 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.808 DEBUG:boto:Method: HEAD 2022-09-09T22:39:32.809 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.809 DEBUG:boto:Data: 2022-09-09T22:39:32.810 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.810 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:32.811 DEBUG:boto:Port: 8001 2022-09-09T22:39:32.811 DEBUG:boto:Params: {} 2022-09-09T22:39:32.812 DEBUG:boto:Token: None 2022-09-09T22:39:32.812 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.813 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ZWkTU8kdkwSkpaqSTDyptyXWvnk= 2022-09-09T22:39:32.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ZWkTU8kdkwSkpaqSTDyptyXWvnk=', 'Content-Length': '0'} 2022-09-09T22:39:32.817 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e59eed2cb3521207-00631bc0a4-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.818 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:32.818 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:32.819 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.819 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.820 DEBUG:boto:Method: HEAD 2022-09-09T22:39:32.820 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.821 DEBUG:boto:Data: 2022-09-09T22:39:32.821 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.822 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:32.822 DEBUG:boto:Port: 8000 2022-09-09T22:39:32.823 DEBUG:boto:Params: {} 2022-09-09T22:39:32.823 DEBUG:boto:Token: None 2022-09-09T22:39:32.824 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.825 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ZWkTU8kdkwSkpaqSTDyptyXWvnk= 2022-09-09T22:39:32.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ZWkTU8kdkwSkpaqSTDyptyXWvnk=', 'Content-Length': '0'} 2022-09-09T22:39:32.828 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e91f517269d574e-00631bc0a4-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:39:32.829 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:32.830 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:32.830 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.831 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.831 DEBUG:boto:Method: GET 2022-09-09T22:39:32.832 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.833 DEBUG:boto:Data: 2022-09-09T22:39:32.833 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.834 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:32.834 DEBUG:boto:Port: 8001 2022-09-09T22:39:32.835 DEBUG:boto:Params: {} 2022-09-09T22:39:32.835 DEBUG:boto:Token: None 2022-09-09T22:39:32.836 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.836 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:MIv1BkaJUrV8Fs1mFcuYDRdcJxs= 2022-09-09T22:39:32.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:MIv1BkaJUrV8Fs1mFcuYDRdcJxs=', 'Content-Length': '0'} 2022-09-09T22:39:32.840 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8946210c916dd07-00631bc0a4-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.841 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:32.842 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:32.842 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.843 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.843 DEBUG:boto:Method: GET 2022-09-09T22:39:32.844 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.844 DEBUG:boto:Data: 2022-09-09T22:39:32.845 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.845 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:32.846 DEBUG:boto:Port: 8000 2022-09-09T22:39:32.846 DEBUG:boto:Params: {} 2022-09-09T22:39:32.847 DEBUG:boto:Token: None 2022-09-09T22:39:32.847 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.848 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:MIv1BkaJUrV8Fs1mFcuYDRdcJxs= 2022-09-09T22:39:32.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:MIv1BkaJUrV8Fs1mFcuYDRdcJxs=', 'Content-Length': '0'} 2022-09-09T22:39:32.852 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bc0bea1cf4415175-00631bc0a4-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.852 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:32.853 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:32.854 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:32.854 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:32.855 DEBUG:boto:Method: GET 2022-09-09T22:39:32.855 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:32.856 DEBUG:boto:Data: 2022-09-09T22:39:32.856 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.857 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:32.857 DEBUG:boto:Port: 8001 2022-09-09T22:39:32.858 DEBUG:boto:Params: {} 2022-09-09T22:39:32.859 DEBUG:boto:Token: None 2022-09-09T22:39:32.859 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/ 2022-09-09T22:39:32.860 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o= 2022-09-09T22:39:32.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o=', 'Content-Length': '0'} 2022-09-09T22:39:32.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001195631b86e60c22-00631bc0a4-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.867 DEBUG:boto:b'jkcgbj-17obj-test-zone31000falseobj-test-zone32022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:32.868 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:32.868 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:32.869 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:32.869 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:32.870 DEBUG:boto:Method: GET 2022-09-09T22:39:32.870 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:32.880 DEBUG:boto:Data: 2022-09-09T22:39:32.880 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.881 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:32.881 DEBUG:boto:Port: 8000 2022-09-09T22:39:32.882 DEBUG:boto:Params: {} 2022-09-09T22:39:32.882 DEBUG:boto:Token: None 2022-09-09T22:39:32.883 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/ 2022-09-09T22:39:32.884 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o= 2022-09-09T22:39:32.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o=', 'Content-Length': '0'} 2022-09-09T22:39:32.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eff499828745ef7d-00631bc0a4-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.890 DEBUG:boto:b'jkcgbj-17obj-test-zone31000falseobj-test-zone32022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:32.891 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:32.892 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:32.893 DEBUG:boto:Method: HEAD 2022-09-09T22:39:32.893 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3 2022-09-09T22:39:32.894 DEBUG:boto:Data: 2022-09-09T22:39:32.894 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.895 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:32.895 DEBUG:boto:Port: 8001 2022-09-09T22:39:32.896 DEBUG:boto:Params: {} 2022-09-09T22:39:32.896 DEBUG:boto:Token: None 2022-09-09T22:39:32.897 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/obj-test-zone3 2022-09-09T22:39:32.898 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ON0ulIb5AcEMGUxd0tH2zeO35e8= 2022-09-09T22:39:32.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ON0ulIb5AcEMGUxd0tH2zeO35e8=', 'Content-Length': '0'} 2022-09-09T22:39:32.902 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000020b0c9416a17149d-00631bc0a4-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.903 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:32.903 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:32.904 DEBUG:boto:Method: HEAD 2022-09-09T22:39:32.905 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3 2022-09-09T22:39:32.905 DEBUG:boto:Data: 2022-09-09T22:39:32.906 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.906 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:32.907 DEBUG:boto:Port: 8000 2022-09-09T22:39:32.907 DEBUG:boto:Params: {} 2022-09-09T22:39:32.908 DEBUG:boto:Token: None 2022-09-09T22:39:32.908 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/obj-test-zone3 2022-09-09T22:39:32.909 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ON0ulIb5AcEMGUxd0tH2zeO35e8= 2022-09-09T22:39:32.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ON0ulIb5AcEMGUxd0tH2zeO35e8=', 'Content-Length': '0'} 2022-09-09T22:39:32.913 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e24022deb865aa06-00631bc0a4-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:39:32.915 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:32.915 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:32.916 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.916 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.917 DEBUG:boto:Method: GET 2022-09-09T22:39:32.917 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.918 DEBUG:boto:Data: 2022-09-09T22:39:32.918 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.919 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:32.919 DEBUG:boto:Port: 8001 2022-09-09T22:39:32.920 DEBUG:boto:Params: {} 2022-09-09T22:39:32.921 DEBUG:boto:Token: None 2022-09-09T22:39:32.921 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.922 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:MIv1BkaJUrV8Fs1mFcuYDRdcJxs= 2022-09-09T22:39:32.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:MIv1BkaJUrV8Fs1mFcuYDRdcJxs=', 'Content-Length': '0'} 2022-09-09T22:39:32.926 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b810ac7fbd1d3ac-00631bc0a4-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.966 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:32.967 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3 2022-09-09T22:39:32.967 DEBUG:boto:path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.968 DEBUG:boto:auth_path=/jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.968 DEBUG:boto:Method: GET 2022-09-09T22:39:32.969 DEBUG:boto:Path: /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.970 DEBUG:boto:Data: 2022-09-09T22:39:32.970 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.971 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:32.971 DEBUG:boto:Port: 8000 2022-09-09T22:39:32.972 DEBUG:boto:Params: {} 2022-09-09T22:39:32.972 DEBUG:boto:Token: None 2022-09-09T22:39:32.973 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/obj-test-zone3?versionId=1RXW239HrAmSBD6KjNRB3qRTyY29OqZ 2022-09-09T22:39:32.973 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:MIv1BkaJUrV8Fs1mFcuYDRdcJxs= 2022-09-09T22:39:32.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:MIv1BkaJUrV8Fs1mFcuYDRdcJxs=', 'Content-Length': '0'} 2022-09-09T22:39:32.977 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', '1RXW239HrAmSBD6KjNRB3qRTyY29OqZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008270cbd56ceb3be8-00631bc0a4-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.978 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:32.979 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:32.979 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:32.980 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:32.980 DEBUG:boto:Method: GET 2022-09-09T22:39:32.981 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:32.981 DEBUG:boto:Data: 2022-09-09T22:39:32.982 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.982 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:32.983 DEBUG:boto:Port: 8001 2022-09-09T22:39:32.983 DEBUG:boto:Params: {} 2022-09-09T22:39:32.984 DEBUG:boto:Token: None 2022-09-09T22:39:32.984 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/ 2022-09-09T22:39:32.985 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o= 2022-09-09T22:39:32.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o=', 'Content-Length': '0'} 2022-09-09T22:39:32.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095aa9f5c92e0cb4c-00631bc0a4-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:32.992 DEBUG:boto:b'jkcgbj-17obj-test-zone31000falseobj-test-zone32022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:32.993 DEBUG:boto:path=/jkcgbj-17/ 2022-09-09T22:39:32.993 DEBUG:boto:auth_path=/jkcgbj-17/ 2022-09-09T22:39:32.994 DEBUG:boto:path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:32.994 DEBUG:boto:auth_path=/jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:32.995 DEBUG:boto:Method: GET 2022-09-09T22:39:32.995 DEBUG:boto:Path: /jkcgbj-17/?prefix=obj-test-zone3 2022-09-09T22:39:32.996 DEBUG:boto:Data: 2022-09-09T22:39:32.997 DEBUG:boto:Headers: {} 2022-09-09T22:39:32.997 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:32.998 DEBUG:boto:Port: 8000 2022-09-09T22:39:32.998 DEBUG:boto:Params: {} 2022-09-09T22:39:32.999 DEBUG:boto:Token: None 2022-09-09T22:39:32.999 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:32 GMT /jkcgbj-17/ 2022-09-09T22:39:33.000 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o= 2022-09-09T22:39:33.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nad7oCKbji2v4lcZxOdyOgQ542o=', 'Content-Length': '0'} 2022-09-09T22:39:33.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af276c794dea4879-00631bc0a4-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:33.006 DEBUG:boto:b'jkcgbj-17obj-test-zone31000falseobj-test-zone32022-09-09T22:36:02.393Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:33.007 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-17 zones={test-zone3, test-zone2} 2022-09-09T22:39:33.008 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jkcgbj-18 2022-09-09T22:39:33.008 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:33.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:33.009 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:39:33.061 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:39:33.061 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:39:33.142 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:39:33.138+0000 7fbf6a6b0dc0 1 bucket sync caught up with source: 2022-09-09T22:39:33.142 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000011.30.5, , 00000000004.19.5, , 00000000008.18.5, , , , , ] 2022-09-09T22:39:33.143 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000011.30.5, , 00000000004.19.5, , 00000000008.18.5, , , , , ] 2022-09-09T22:39:33.143 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:39:33.138+0000 7fbf6a6b0dc0 0 bucket checkpoint complete 2022-09-09T22:39:33.170 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jkcgbj-18 2022-09-09T22:39:33.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:33.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:33.172 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:39:33.227 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:39:33.227 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:39:33.292 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:39:33.283+0000 7faad8fcbdc0 1 bucket sync caught up with source: 2022-09-09T22:39:33.293 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000011.30.5, , 00000000004.19.5, , 00000000008.18.5, , , , , ] 2022-09-09T22:39:33.293 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000011.30.5, , 00000000004.19.5, , 00000000008.18.5, , , , , ] 2022-09-09T22:39:33.293 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:39:33.283+0000 7faad8fcbdc0 0 bucket checkpoint complete 2022-09-09T22:39:33.317 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jkcgbj-18 2022-09-09T22:39:33.317 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:33.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:33.319 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:39:33.373 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:39:33.373 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:39:33.450 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:39:33.446+0000 7fe2162d7dc0 1 bucket sync caught up with source: 2022-09-09T22:39:33.451 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000011.30.5, , 00000000004.19.5, , 00000000008.18.5, , , , , ] 2022-09-09T22:39:33.451 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000011.30.5, , 00000000004.19.5, , 00000000008.18.5, , , , , ] 2022-09-09T22:39:33.451 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:39:33.446+0000 7fe2162d7dc0 0 bucket checkpoint complete 2022-09-09T22:39:33.476 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jkcgbj-18 2022-09-09T22:39:33.477 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:33.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:33.478 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:39:33.534 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:39:33.534 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:39:33.608 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:39:33.599+0000 7f294e72fdc0 1 bucket sync caught up with source: 2022-09-09T22:39:33.608 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000004.40.5, , 00000000011.55.5, , 00000000008.24.5, , , , , ] 2022-09-09T22:39:33.608 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000004.40.5, , 00000000011.55.5, , 00000000008.24.5, , , , , ] 2022-09-09T22:39:33.609 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:39:33.599+0000 7f294e72fdc0 0 bucket checkpoint complete 2022-09-09T22:39:33.634 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jkcgbj-18 2022-09-09T22:39:33.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:33.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:33.636 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:39:33.684 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:39:33.685 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:39:33.751 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:39:33.743+0000 7ff2846f3dc0 1 bucket sync caught up with source: 2022-09-09T22:39:33.751 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000004.40.5, , 00000000011.55.5, , 00000000008.24.5, , , , , ] 2022-09-09T22:39:33.752 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000004.40.5, , 00000000011.55.5, , 00000000008.24.5, , , , , ] 2022-09-09T22:39:33.752 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:39:33.743+0000 7ff2846f3dc0 0 bucket checkpoint complete 2022-09-09T22:39:33.775 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jkcgbj-18 2022-09-09T22:39:33.776 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jkcgbj-18 2022-09-09T22:39:33.776 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:33.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:33.777 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:39:33.831 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:39:33.832 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:39:33.895 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:39:33.887+0000 7fc04b25cdc0 1 bucket sync caught up with source: 2022-09-09T22:39:33.896 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, 00000000004.41.5, , 00000000004.41.5, , 00000000016.39.5, , , , , ] 2022-09-09T22:39:33.896 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, 00000000004.41.5, , 00000000004.41.5, , 00000000016.39.5, , , , , ] 2022-09-09T22:39:33.896 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:39:33.887+0000 7fc04b25cdc0 0 bucket checkpoint complete 2022-09-09T22:39:33.920 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jkcgbj-18 2022-09-09T22:39:33.921 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:33.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:33.922 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:39:33.975 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:39:33.975 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:39:34.051 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:39:34.050+0000 7f362185edc0 1 bucket sync caught up with source: 2022-09-09T22:39:34.052 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, 00000000004.41.5, , 00000000004.41.5, , 00000000016.39.5, , , , , ] 2022-09-09T22:39:34.052 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, 00000000004.41.5, , 00000000004.41.5, , 00000000016.39.5, , , , , ] 2022-09-09T22:39:34.052 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:39:34.050+0000 7f362185edc0 0 bucket checkpoint complete 2022-09-09T22:39:34.079 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jkcgbj-18 2022-09-09T22:39:34.080 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-18 zones={test-zone2, test-zone1} 2022-09-09T22:39:34.080 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:34.081 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:34.081 DEBUG:boto:Method: HEAD 2022-09-09T22:39:34.082 DEBUG:boto:Path: /jkcgbj-18/ 2022-09-09T22:39:34.082 DEBUG:boto:Data: 2022-09-09T22:39:34.083 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.083 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:34.084 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.085 DEBUG:boto:Params: {} 2022-09-09T22:39:34.085 DEBUG:boto:Token: None 2022-09-09T22:39:34.086 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/ 2022-09-09T22:39:34.087 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:hTGaoLpB/KPjkpCmOrhxSotoktk= 2022-09-09T22:39:34.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:hTGaoLpB/KPjkpCmOrhxSotoktk=', 'Content-Length': '0'} 2022-09-09T22:39:34.092 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f5417079617c7d53-00631bc0a6-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.093 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:34.093 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:34.094 DEBUG:boto:Method: HEAD 2022-09-09T22:39:34.094 DEBUG:boto:Path: /jkcgbj-18/ 2022-09-09T22:39:34.095 DEBUG:boto:Data: 2022-09-09T22:39:34.095 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.096 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:34.096 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.097 DEBUG:boto:Params: {} 2022-09-09T22:39:34.097 DEBUG:boto:Token: None 2022-09-09T22:39:34.098 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/ 2022-09-09T22:39:34.099 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:hTGaoLpB/KPjkpCmOrhxSotoktk= 2022-09-09T22:39:34.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:hTGaoLpB/KPjkpCmOrhxSotoktk=', 'Content-Length': '0'} 2022-09-09T22:39:34.104 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000017ee24dc47b3d76f-00631bc0a6-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.104 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:39:34.105 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:34.105 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:34.106 DEBUG:boto:path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:34.106 DEBUG:boto:auth_path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:34.107 DEBUG:boto:Method: GET 2022-09-09T22:39:34.108 DEBUG:boto:Path: /jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:34.108 DEBUG:boto:Data: 2022-09-09T22:39:34.109 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.109 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:34.110 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.110 DEBUG:boto:Params: {} 2022-09-09T22:39:34.111 DEBUG:boto:Token: None 2022-09-09T22:39:34.111 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/?versions 2022-09-09T22:39:34.112 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:X9q+IbvyYUJRCqD5/iMmDqqvha4= 2022-09-09T22:39:34.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:X9q+IbvyYUJRCqD5/iMmDqqvha4=', 'Content-Length': '0'} 2022-09-09T22:39:34.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3eff4a04d8ea43f-00631bc0a6-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.119 DEBUG:boto:b'jkcgbj-18999falsedummyRQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5true2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1a8lrwM74XPlF1mwXf8hNl9Y9ISdV53ltrue2022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2e.gQ6yFECscD-oc4OV0dmk6aAg9jH.otrue2022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1true2022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:34.120 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:39:34.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:39:34.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:39:34.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:39:34.122 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:39:34.122 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:34.123 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:34.123 DEBUG:boto:path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:34.124 DEBUG:boto:auth_path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:34.124 DEBUG:boto:Method: GET 2022-09-09T22:39:34.125 DEBUG:boto:Path: /jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:34.126 DEBUG:boto:Data: 2022-09-09T22:39:34.126 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.127 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:34.127 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.128 DEBUG:boto:Params: {} 2022-09-09T22:39:34.128 DEBUG:boto:Token: None 2022-09-09T22:39:34.129 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/?versions 2022-09-09T22:39:34.129 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:X9q+IbvyYUJRCqD5/iMmDqqvha4= 2022-09-09T22:39:34.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:X9q+IbvyYUJRCqD5/iMmDqqvha4=', 'Content-Length': '0'} 2022-09-09T22:39:34.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055605108f75065c5-00631bc0a6-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.136 DEBUG:boto:b'jkcgbj-18999falsedummyRQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5true2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1a8lrwM74XPlF1mwXf8hNl9Y9ISdV53ltrue2022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2e.gQ6yFECscD-oc4OV0dmk6aAg9jH.otrue2022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1true2022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:34.137 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:39:34.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:39:34.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:39:34.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:39:34.140 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:34.140 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:34.141 DEBUG:boto:path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:34.141 DEBUG:boto:auth_path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:34.142 DEBUG:boto:Method: GET 2022-09-09T22:39:34.143 DEBUG:boto:Path: /jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:34.143 DEBUG:boto:Data: 2022-09-09T22:39:34.144 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.144 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:34.145 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.145 DEBUG:boto:Params: {} 2022-09-09T22:39:34.146 DEBUG:boto:Token: None 2022-09-09T22:39:34.146 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/?versions 2022-09-09T22:39:34.147 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:X9q+IbvyYUJRCqD5/iMmDqqvha4= 2022-09-09T22:39:34.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:X9q+IbvyYUJRCqD5/iMmDqqvha4=', 'Content-Length': '0'} 2022-09-09T22:39:34.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005eeb84b0f4fc561f-00631bc0a6-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.154 DEBUG:boto:b'jkcgbj-18999falsedummyRQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5true2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1a8lrwM74XPlF1mwXf8hNl9Y9ISdV53ltrue2022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2e.gQ6yFECscD-oc4OV0dmk6aAg9jH.otrue2022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1true2022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:34.155 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:34.156 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:34.156 DEBUG:boto:path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:34.157 DEBUG:boto:auth_path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:34.157 DEBUG:boto:Method: GET 2022-09-09T22:39:34.158 DEBUG:boto:Path: /jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:34.159 DEBUG:boto:Data: 2022-09-09T22:39:34.159 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.160 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:34.160 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.161 DEBUG:boto:Params: {} 2022-09-09T22:39:34.162 DEBUG:boto:Token: None 2022-09-09T22:39:34.162 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/?versions 2022-09-09T22:39:34.163 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:X9q+IbvyYUJRCqD5/iMmDqqvha4= 2022-09-09T22:39:34.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:X9q+IbvyYUJRCqD5/iMmDqqvha4=', 'Content-Length': '0'} 2022-09-09T22:39:34.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c0ddd50b442cad7-00631bc0a6-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.170 DEBUG:boto:b'jkcgbj-18999falsedummyRQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5true2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1a8lrwM74XPlF1mwXf8hNl9Y9ISdV53ltrue2022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2e.gQ6yFECscD-oc4OV0dmk6aAg9jH.otrue2022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1true2022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:34.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:39:34.171 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:39:34.172 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:39:34.172 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.173 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.174 DEBUG:boto:Method: GET 2022-09-09T22:39:34.174 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.175 DEBUG:boto:Data: 2022-09-09T22:39:34.175 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.176 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:34.176 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.177 DEBUG:boto:Params: {} 2022-09-09T22:39:34.177 DEBUG:boto:Token: None 2022-09-09T22:39:34.178 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.179 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:k0dGW6dKS6+UtReT4LuTtjmICqI= 2022-09-09T22:39:34.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:k0dGW6dKS6+UtReT4LuTtjmICqI=', 'Content-Length': '0'} 2022-09-09T22:39:34.182 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f65fa48028d71d7f-00631bc0a6-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.183 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:39:34.183 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:39:34.184 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.184 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.185 DEBUG:boto:Method: GET 2022-09-09T22:39:34.186 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.186 DEBUG:boto:Data: 2022-09-09T22:39:34.187 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.187 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:34.188 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.189 DEBUG:boto:Params: {} 2022-09-09T22:39:34.189 DEBUG:boto:Token: None 2022-09-09T22:39:34.190 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.190 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:k0dGW6dKS6+UtReT4LuTtjmICqI= 2022-09-09T22:39:34.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:k0dGW6dKS6+UtReT4LuTtjmICqI=', 'Content-Length': '0'} 2022-09-09T22:39:34.194 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c0c126240e5cc89c-00631bc0a6-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.195 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:39:34.195 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:39:34.196 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.197 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.197 DEBUG:boto:Method: HEAD 2022-09-09T22:39:34.198 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.198 DEBUG:boto:Data: 2022-09-09T22:39:34.199 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.199 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:34.200 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.200 DEBUG:boto:Params: {} 2022-09-09T22:39:34.201 DEBUG:boto:Token: None 2022-09-09T22:39:34.202 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.203 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vgmMJpdYQ61b1Kx2CBd49UEzMPI= 2022-09-09T22:39:34.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vgmMJpdYQ61b1Kx2CBd49UEzMPI=', 'Content-Length': '0'} 2022-09-09T22:39:34.207 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000047d4cce8a85f84fb-00631bc0a6-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.208 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:39:34.209 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:39:34.209 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.210 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.210 DEBUG:boto:Method: HEAD 2022-09-09T22:39:34.211 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.211 DEBUG:boto:Data: 2022-09-09T22:39:34.212 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.212 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:34.213 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.213 DEBUG:boto:Params: {} 2022-09-09T22:39:34.214 DEBUG:boto:Token: None 2022-09-09T22:39:34.215 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.215 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vgmMJpdYQ61b1Kx2CBd49UEzMPI= 2022-09-09T22:39:34.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vgmMJpdYQ61b1Kx2CBd49UEzMPI=', 'Content-Length': '0'} 2022-09-09T22:39:34.219 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d2ff1ff50b1d04bc-00631bc0a6-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:39:34.220 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:39:34.220 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:39:34.221 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.221 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.222 DEBUG:boto:Method: GET 2022-09-09T22:39:34.223 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.223 DEBUG:boto:Data: 2022-09-09T22:39:34.224 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.224 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:34.225 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.225 DEBUG:boto:Params: {} 2022-09-09T22:39:34.226 DEBUG:boto:Token: None 2022-09-09T22:39:34.226 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.227 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:k0dGW6dKS6+UtReT4LuTtjmICqI= 2022-09-09T22:39:34.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:k0dGW6dKS6+UtReT4LuTtjmICqI=', 'Content-Length': '0'} 2022-09-09T22:39:34.230 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000092c75c9272021bf2-00631bc0a6-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.231 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:39:34.232 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:39:34.232 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.233 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.233 DEBUG:boto:Method: GET 2022-09-09T22:39:34.234 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.234 DEBUG:boto:Data: 2022-09-09T22:39:34.235 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.236 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:34.236 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.237 DEBUG:boto:Params: {} 2022-09-09T22:39:34.237 DEBUG:boto:Token: None 2022-09-09T22:39:34.238 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.239 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:k0dGW6dKS6+UtReT4LuTtjmICqI= 2022-09-09T22:39:34.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:k0dGW6dKS6+UtReT4LuTtjmICqI=', 'Content-Length': '0'} 2022-09-09T22:39:34.242 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dd959f521dcab920-00631bc0a6-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.243 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:34.243 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:34.244 DEBUG:boto:path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:39:34.244 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:39:34.245 DEBUG:boto:Method: GET 2022-09-09T22:39:34.245 DEBUG:boto:Path: /jkcgbj-18/?prefix=dummy 2022-09-09T22:39:34.246 DEBUG:boto:Data: 2022-09-09T22:39:34.246 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.247 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:34.247 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.248 DEBUG:boto:Params: {} 2022-09-09T22:39:34.248 DEBUG:boto:Token: None 2022-09-09T22:39:34.249 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/ 2022-09-09T22:39:34.250 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:iwIWse8TI1qtX/P4n0lZerBCIyY= 2022-09-09T22:39:34.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:iwIWse8TI1qtX/P4n0lZerBCIyY=', 'Content-Length': '0'} 2022-09-09T22:39:34.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000705b62a13de4a107-00631bc0a6-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.298 DEBUG:boto:b'jkcgbj-18dummy1000falsedummy2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:34.299 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:34.300 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:34.300 DEBUG:boto:path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:39:34.301 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:39:34.301 DEBUG:boto:Method: GET 2022-09-09T22:39:34.302 DEBUG:boto:Path: /jkcgbj-18/?prefix=dummy 2022-09-09T22:39:34.303 DEBUG:boto:Data: 2022-09-09T22:39:34.303 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.304 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:34.304 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.305 DEBUG:boto:Params: {} 2022-09-09T22:39:34.305 DEBUG:boto:Token: None 2022-09-09T22:39:34.306 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/ 2022-09-09T22:39:34.307 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:iwIWse8TI1qtX/P4n0lZerBCIyY= 2022-09-09T22:39:34.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:iwIWse8TI1qtX/P4n0lZerBCIyY=', 'Content-Length': '0'} 2022-09-09T22:39:34.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097f5e49a41118b93-00631bc0a6-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.313 DEBUG:boto:b'jkcgbj-18dummy1000falsedummy2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:34.314 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:39:34.315 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:39:34.315 DEBUG:boto:Method: HEAD 2022-09-09T22:39:34.316 DEBUG:boto:Path: /jkcgbj-18/dummy 2022-09-09T22:39:34.316 DEBUG:boto:Data: 2022-09-09T22:39:34.317 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.317 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:34.318 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.318 DEBUG:boto:Params: {} 2022-09-09T22:39:34.319 DEBUG:boto:Token: None 2022-09-09T22:39:34.319 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/dummy 2022-09-09T22:39:34.320 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:aUNeHhu10BLw40P8WdCqgW+dUYs= 2022-09-09T22:39:34.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:aUNeHhu10BLw40P8WdCqgW+dUYs=', 'Content-Length': '0'} 2022-09-09T22:39:34.324 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004428d7991a22e9bc-00631bc0a6-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.325 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:39:34.325 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:39:34.326 DEBUG:boto:Method: HEAD 2022-09-09T22:39:34.326 DEBUG:boto:Path: /jkcgbj-18/dummy 2022-09-09T22:39:34.327 DEBUG:boto:Data: 2022-09-09T22:39:34.327 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.328 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:34.329 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.329 DEBUG:boto:Params: {} 2022-09-09T22:39:34.330 DEBUG:boto:Token: None 2022-09-09T22:39:34.330 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/dummy 2022-09-09T22:39:34.331 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:aUNeHhu10BLw40P8WdCqgW+dUYs= 2022-09-09T22:39:34.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:aUNeHhu10BLw40P8WdCqgW+dUYs=', 'Content-Length': '0'} 2022-09-09T22:39:34.335 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000048a27756a209139a-00631bc0a6-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:39:34.336 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:39:34.337 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:39:34.338 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.338 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.339 DEBUG:boto:Method: GET 2022-09-09T22:39:34.339 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.340 DEBUG:boto:Data: 2022-09-09T22:39:34.340 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.341 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:34.341 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.342 DEBUG:boto:Params: {} 2022-09-09T22:39:34.343 DEBUG:boto:Token: None 2022-09-09T22:39:34.343 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.344 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:k0dGW6dKS6+UtReT4LuTtjmICqI= 2022-09-09T22:39:34.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:k0dGW6dKS6+UtReT4LuTtjmICqI=', 'Content-Length': '0'} 2022-09-09T22:39:34.347 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002b75decae114f78e-00631bc0a6-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.348 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:39:34.348 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:39:34.349 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.349 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.350 DEBUG:boto:Method: GET 2022-09-09T22:39:34.350 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.351 DEBUG:boto:Data: 2022-09-09T22:39:34.352 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.352 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:34.353 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.353 DEBUG:boto:Params: {} 2022-09-09T22:39:34.354 DEBUG:boto:Token: None 2022-09-09T22:39:34.355 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:34.355 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:k0dGW6dKS6+UtReT4LuTtjmICqI= 2022-09-09T22:39:34.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:k0dGW6dKS6+UtReT4LuTtjmICqI=', 'Content-Length': '0'} 2022-09-09T22:39:34.359 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000012a73eb779705c53-00631bc0a6-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.359 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:34.360 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:34.360 DEBUG:boto:path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:39:34.361 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:39:34.362 DEBUG:boto:Method: GET 2022-09-09T22:39:34.362 DEBUG:boto:Path: /jkcgbj-18/?prefix=dummy 2022-09-09T22:39:34.363 DEBUG:boto:Data: 2022-09-09T22:39:34.363 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.364 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:34.364 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.365 DEBUG:boto:Params: {} 2022-09-09T22:39:34.366 DEBUG:boto:Token: None 2022-09-09T22:39:34.366 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/ 2022-09-09T22:39:34.367 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:iwIWse8TI1qtX/P4n0lZerBCIyY= 2022-09-09T22:39:34.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:iwIWse8TI1qtX/P4n0lZerBCIyY=', 'Content-Length': '0'} 2022-09-09T22:39:34.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdaf47fc06839299-00631bc0a6-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.414 DEBUG:boto:b'jkcgbj-18dummy1000falsedummy2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:34.416 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:34.417 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:34.418 DEBUG:boto:path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:39:34.419 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:39:34.420 DEBUG:boto:Method: GET 2022-09-09T22:39:34.421 DEBUG:boto:Path: /jkcgbj-18/?prefix=dummy 2022-09-09T22:39:34.422 DEBUG:boto:Data: 2022-09-09T22:39:34.423 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.424 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:34.425 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.426 DEBUG:boto:Params: {} 2022-09-09T22:39:34.427 DEBUG:boto:Token: None 2022-09-09T22:39:34.428 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/ 2022-09-09T22:39:34.429 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:iwIWse8TI1qtX/P4n0lZerBCIyY= 2022-09-09T22:39:34.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:iwIWse8TI1qtX/P4n0lZerBCIyY=', 'Content-Length': '0'} 2022-09-09T22:39:34.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dea551807481c582-00631bc0a6-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.436 DEBUG:boto:b'jkcgbj-18dummy1000falsedummy2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:34.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:39:34.438 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:34.439 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:34.439 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.440 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.440 DEBUG:boto:Method: GET 2022-09-09T22:39:34.441 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.442 DEBUG:boto:Data: 2022-09-09T22:39:34.443 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.443 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:34.444 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.444 DEBUG:boto:Params: {} 2022-09-09T22:39:34.445 DEBUG:boto:Token: None 2022-09-09T22:39:34.445 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.446 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:n9ySMpxuCDs8dx53+E0fKL0obMs= 2022-09-09T22:39:34.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:n9ySMpxuCDs8dx53+E0fKL0obMs=', 'Content-Length': '0'} 2022-09-09T22:39:34.450 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008351d4d2d5918e96-00631bc0a6-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.450 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:34.451 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:34.451 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.452 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.453 DEBUG:boto:Method: GET 2022-09-09T22:39:34.453 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.454 DEBUG:boto:Data: 2022-09-09T22:39:34.454 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.455 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:34.455 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.456 DEBUG:boto:Params: {} 2022-09-09T22:39:34.456 DEBUG:boto:Token: None 2022-09-09T22:39:34.457 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.457 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:n9ySMpxuCDs8dx53+E0fKL0obMs= 2022-09-09T22:39:34.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:n9ySMpxuCDs8dx53+E0fKL0obMs=', 'Content-Length': '0'} 2022-09-09T22:39:34.461 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000dff9abbd8113e5a-00631bc0a6-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.502 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:34.503 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:34.503 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.504 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.504 DEBUG:boto:Method: HEAD 2022-09-09T22:39:34.505 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.505 DEBUG:boto:Data: 2022-09-09T22:39:34.506 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.507 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:34.507 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.508 DEBUG:boto:Params: {} 2022-09-09T22:39:34.509 DEBUG:boto:Token: None 2022-09-09T22:39:34.510 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.511 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:gfzQ+RuD58ix/uh53HHxNOi8PBc= 2022-09-09T22:39:34.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:gfzQ+RuD58ix/uh53HHxNOi8PBc=', 'Content-Length': '0'} 2022-09-09T22:39:34.516 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000074e2874034b2da48-00631bc0a6-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.517 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:34.518 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:34.519 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.519 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.520 DEBUG:boto:Method: HEAD 2022-09-09T22:39:34.520 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.521 DEBUG:boto:Data: 2022-09-09T22:39:34.521 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.522 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:34.523 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.523 DEBUG:boto:Params: {} 2022-09-09T22:39:34.524 DEBUG:boto:Token: None 2022-09-09T22:39:34.524 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.525 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:gfzQ+RuD58ix/uh53HHxNOi8PBc= 2022-09-09T22:39:34.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:gfzQ+RuD58ix/uh53HHxNOi8PBc=', 'Content-Length': '0'} 2022-09-09T22:39:34.528 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072baeea2d228220b-00631bc0a6-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:39:34.530 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:34.531 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:34.531 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.532 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.532 DEBUG:boto:Method: GET 2022-09-09T22:39:34.533 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.533 DEBUG:boto:Data: 2022-09-09T22:39:34.534 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.535 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:34.535 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.536 DEBUG:boto:Params: {} 2022-09-09T22:39:34.536 DEBUG:boto:Token: None 2022-09-09T22:39:34.537 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.538 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:n9ySMpxuCDs8dx53+E0fKL0obMs= 2022-09-09T22:39:34.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:n9ySMpxuCDs8dx53+E0fKL0obMs=', 'Content-Length': '0'} 2022-09-09T22:39:34.541 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b47a6812cea694ce-00631bc0a6-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.542 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:34.543 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:34.543 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.544 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.544 DEBUG:boto:Method: GET 2022-09-09T22:39:34.545 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.545 DEBUG:boto:Data: 2022-09-09T22:39:34.546 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.547 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:34.547 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.548 DEBUG:boto:Params: {} 2022-09-09T22:39:34.548 DEBUG:boto:Token: None 2022-09-09T22:39:34.549 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.550 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:n9ySMpxuCDs8dx53+E0fKL0obMs= 2022-09-09T22:39:34.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:n9ySMpxuCDs8dx53+E0fKL0obMs=', 'Content-Length': '0'} 2022-09-09T22:39:34.554 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058e09fbe22ae0a54-00631bc0a6-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.555 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:34.555 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:34.556 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:34.556 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:34.557 DEBUG:boto:Method: GET 2022-09-09T22:39:34.557 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:34.558 DEBUG:boto:Data: 2022-09-09T22:39:34.559 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.559 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:34.560 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.560 DEBUG:boto:Params: {} 2022-09-09T22:39:34.561 DEBUG:boto:Token: None 2022-09-09T22:39:34.561 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/ 2022-09-09T22:39:34.562 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:iwIWse8TI1qtX/P4n0lZerBCIyY= 2022-09-09T22:39:34.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:iwIWse8TI1qtX/P4n0lZerBCIyY=', 'Content-Length': '0'} 2022-09-09T22:39:34.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035e94bea7749abe1-00631bc0a6-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.569 DEBUG:boto:b'jkcgbj-18obj-test-zone11000falseobj-test-zone12022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:34.570 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:34.571 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:34.571 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:34.572 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:34.572 DEBUG:boto:Method: GET 2022-09-09T22:39:34.573 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:34.574 DEBUG:boto:Data: 2022-09-09T22:39:34.574 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.575 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:34.575 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.576 DEBUG:boto:Params: {} 2022-09-09T22:39:34.576 DEBUG:boto:Token: None 2022-09-09T22:39:34.577 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/ 2022-09-09T22:39:34.578 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:iwIWse8TI1qtX/P4n0lZerBCIyY= 2022-09-09T22:39:34.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:iwIWse8TI1qtX/P4n0lZerBCIyY=', 'Content-Length': '0'} 2022-09-09T22:39:34.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2ef28025a9e4113-00631bc0a6-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.585 DEBUG:boto:b'jkcgbj-18obj-test-zone11000falseobj-test-zone12022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:34.585 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:34.586 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:34.587 DEBUG:boto:Method: HEAD 2022-09-09T22:39:34.587 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1 2022-09-09T22:39:34.588 DEBUG:boto:Data: 2022-09-09T22:39:34.588 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.589 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:34.589 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.590 DEBUG:boto:Params: {} 2022-09-09T22:39:34.590 DEBUG:boto:Token: None 2022-09-09T22:39:34.591 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/obj-test-zone1 2022-09-09T22:39:34.591 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Cz5XnDh8uqGF53Y0Ge3BCfbYTVs= 2022-09-09T22:39:34.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Cz5XnDh8uqGF53Y0Ge3BCfbYTVs=', 'Content-Length': '0'} 2022-09-09T22:39:34.595 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000116a3bbee54855b9-00631bc0a6-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.595 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:34.596 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:34.596 DEBUG:boto:Method: HEAD 2022-09-09T22:39:34.597 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1 2022-09-09T22:39:34.597 DEBUG:boto:Data: 2022-09-09T22:39:34.598 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.599 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:34.599 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.600 DEBUG:boto:Params: {} 2022-09-09T22:39:34.600 DEBUG:boto:Token: None 2022-09-09T22:39:34.601 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/obj-test-zone1 2022-09-09T22:39:34.602 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Cz5XnDh8uqGF53Y0Ge3BCfbYTVs= 2022-09-09T22:39:34.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Cz5XnDh8uqGF53Y0Ge3BCfbYTVs=', 'Content-Length': '0'} 2022-09-09T22:39:34.607 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002806e03f16bdbb90-00631bc0a6-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:39:34.608 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:34.609 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:34.609 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.610 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.611 DEBUG:boto:Method: GET 2022-09-09T22:39:34.612 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.613 DEBUG:boto:Data: 2022-09-09T22:39:34.614 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.615 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:34.615 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.616 DEBUG:boto:Params: {} 2022-09-09T22:39:34.617 DEBUG:boto:Token: None 2022-09-09T22:39:34.618 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.618 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:n9ySMpxuCDs8dx53+E0fKL0obMs= 2022-09-09T22:39:34.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:n9ySMpxuCDs8dx53+E0fKL0obMs=', 'Content-Length': '0'} 2022-09-09T22:39:34.622 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c645eaaa174e2f01-00631bc0a6-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.666 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:34.667 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:34.667 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.668 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.668 DEBUG:boto:Method: GET 2022-09-09T22:39:34.669 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.669 DEBUG:boto:Data: 2022-09-09T22:39:34.670 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.670 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:34.671 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.671 DEBUG:boto:Params: {} 2022-09-09T22:39:34.672 DEBUG:boto:Token: None 2022-09-09T22:39:34.673 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:34.673 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:n9ySMpxuCDs8dx53+E0fKL0obMs= 2022-09-09T22:39:34.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:n9ySMpxuCDs8dx53+E0fKL0obMs=', 'Content-Length': '0'} 2022-09-09T22:39:34.677 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cdc9fc4dd8823c1f-00631bc0a6-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.678 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:34.679 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:34.679 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:34.680 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:34.680 DEBUG:boto:Method: GET 2022-09-09T22:39:34.681 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:34.681 DEBUG:boto:Data: 2022-09-09T22:39:34.682 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.682 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:34.683 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.684 DEBUG:boto:Params: {} 2022-09-09T22:39:34.684 DEBUG:boto:Token: None 2022-09-09T22:39:34.685 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/ 2022-09-09T22:39:34.685 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:iwIWse8TI1qtX/P4n0lZerBCIyY= 2022-09-09T22:39:34.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:iwIWse8TI1qtX/P4n0lZerBCIyY=', 'Content-Length': '0'} 2022-09-09T22:39:34.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087f76cb56a28a3f9-00631bc0a6-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.693 DEBUG:boto:b'jkcgbj-18obj-test-zone11000falseobj-test-zone12022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:34.693 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:34.694 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:34.695 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:34.695 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:34.696 DEBUG:boto:Method: GET 2022-09-09T22:39:34.696 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:34.697 DEBUG:boto:Data: 2022-09-09T22:39:34.697 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.698 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:34.698 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.699 DEBUG:boto:Params: {} 2022-09-09T22:39:34.700 DEBUG:boto:Token: None 2022-09-09T22:39:34.700 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/ 2022-09-09T22:39:34.701 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:iwIWse8TI1qtX/P4n0lZerBCIyY= 2022-09-09T22:39:34.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:iwIWse8TI1qtX/P4n0lZerBCIyY=', 'Content-Length': '0'} 2022-09-09T22:39:34.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082b52ca3fe392505-00631bc0a6-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.708 DEBUG:boto:b'jkcgbj-18obj-test-zone11000falseobj-test-zone12022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:34.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:39:34.709 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:34.710 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:34.710 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.711 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.711 DEBUG:boto:Method: GET 2022-09-09T22:39:34.712 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.713 DEBUG:boto:Data: 2022-09-09T22:39:34.713 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.714 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:34.714 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.715 DEBUG:boto:Params: {} 2022-09-09T22:39:34.716 DEBUG:boto:Token: None 2022-09-09T22:39:34.716 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.717 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LTeqoIoelP5ihq9+LqvwzlLVGT8= 2022-09-09T22:39:34.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LTeqoIoelP5ihq9+LqvwzlLVGT8=', 'Content-Length': '0'} 2022-09-09T22:39:34.721 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000014cf468595010a15-00631bc0a6-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.721 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:34.722 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:34.723 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.723 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.724 DEBUG:boto:Method: GET 2022-09-09T22:39:34.724 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.725 DEBUG:boto:Data: 2022-09-09T22:39:34.725 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.726 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:34.727 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.727 DEBUG:boto:Params: {} 2022-09-09T22:39:34.728 DEBUG:boto:Token: None 2022-09-09T22:39:34.728 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.729 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LTeqoIoelP5ihq9+LqvwzlLVGT8= 2022-09-09T22:39:34.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LTeqoIoelP5ihq9+LqvwzlLVGT8=', 'Content-Length': '0'} 2022-09-09T22:39:34.735 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7247564189ee3f3-00631bc0a6-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.736 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:34.737 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:34.737 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.738 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.738 DEBUG:boto:Method: HEAD 2022-09-09T22:39:34.739 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.740 DEBUG:boto:Data: 2022-09-09T22:39:34.740 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.741 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:34.741 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.742 DEBUG:boto:Params: {} 2022-09-09T22:39:34.742 DEBUG:boto:Token: None 2022-09-09T22:39:34.743 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.743 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Jd6AfJ976VAXzpdwSye6uKFie9U= 2022-09-09T22:39:34.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Jd6AfJ976VAXzpdwSye6uKFie9U=', 'Content-Length': '0'} 2022-09-09T22:39:34.747 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000047321a2937dfda79-00631bc0a6-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.748 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:34.749 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:34.749 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.750 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.750 DEBUG:boto:Method: HEAD 2022-09-09T22:39:34.751 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.752 DEBUG:boto:Data: 2022-09-09T22:39:34.752 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.753 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:34.753 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.754 DEBUG:boto:Params: {} 2022-09-09T22:39:34.754 DEBUG:boto:Token: None 2022-09-09T22:39:34.755 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.755 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Jd6AfJ976VAXzpdwSye6uKFie9U= 2022-09-09T22:39:34.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Jd6AfJ976VAXzpdwSye6uKFie9U=', 'Content-Length': '0'} 2022-09-09T22:39:34.759 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000979c961674134d0f-00631bc0a6-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:39:34.760 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:34.760 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:34.761 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.762 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.762 DEBUG:boto:Method: GET 2022-09-09T22:39:34.763 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.763 DEBUG:boto:Data: 2022-09-09T22:39:34.764 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.764 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:34.765 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.765 DEBUG:boto:Params: {} 2022-09-09T22:39:34.766 DEBUG:boto:Token: None 2022-09-09T22:39:34.766 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.767 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LTeqoIoelP5ihq9+LqvwzlLVGT8= 2022-09-09T22:39:34.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LTeqoIoelP5ihq9+LqvwzlLVGT8=', 'Content-Length': '0'} 2022-09-09T22:39:34.770 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000665ff425b1ad7497-00631bc0a6-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.814 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:34.815 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:34.815 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.816 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.816 DEBUG:boto:Method: GET 2022-09-09T22:39:34.817 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.817 DEBUG:boto:Data: 2022-09-09T22:39:34.818 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.819 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:34.819 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.820 DEBUG:boto:Params: {} 2022-09-09T22:39:34.820 DEBUG:boto:Token: None 2022-09-09T22:39:34.821 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.821 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LTeqoIoelP5ihq9+LqvwzlLVGT8= 2022-09-09T22:39:34.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LTeqoIoelP5ihq9+LqvwzlLVGT8=', 'Content-Length': '0'} 2022-09-09T22:39:34.825 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b46b24747aa2ed6d-00631bc0a6-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.826 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:34.827 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:34.827 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:34.828 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:34.828 DEBUG:boto:Method: GET 2022-09-09T22:39:34.829 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:34.830 DEBUG:boto:Data: 2022-09-09T22:39:34.830 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.831 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:34.831 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.832 DEBUG:boto:Params: {} 2022-09-09T22:39:34.832 DEBUG:boto:Token: None 2022-09-09T22:39:34.833 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/ 2022-09-09T22:39:34.834 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:iwIWse8TI1qtX/P4n0lZerBCIyY= 2022-09-09T22:39:34.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:iwIWse8TI1qtX/P4n0lZerBCIyY=', 'Content-Length': '0'} 2022-09-09T22:39:34.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fa9e84498f8d513-00631bc0a6-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.840 DEBUG:boto:b'jkcgbj-18obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:34.841 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:34.841 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:34.842 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:34.843 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:34.843 DEBUG:boto:Method: GET 2022-09-09T22:39:34.844 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:34.844 DEBUG:boto:Data: 2022-09-09T22:39:34.845 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.846 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:34.846 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.847 DEBUG:boto:Params: {} 2022-09-09T22:39:34.847 DEBUG:boto:Token: None 2022-09-09T22:39:34.848 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/ 2022-09-09T22:39:34.849 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:iwIWse8TI1qtX/P4n0lZerBCIyY= 2022-09-09T22:39:34.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:iwIWse8TI1qtX/P4n0lZerBCIyY=', 'Content-Length': '0'} 2022-09-09T22:39:34.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd14794b62002595-00631bc0a6-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.856 DEBUG:boto:b'jkcgbj-18obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:34.857 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:34.858 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:34.858 DEBUG:boto:Method: HEAD 2022-09-09T22:39:34.859 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2 2022-09-09T22:39:34.859 DEBUG:boto:Data: 2022-09-09T22:39:34.860 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.860 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:34.861 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.862 DEBUG:boto:Params: {} 2022-09-09T22:39:34.862 DEBUG:boto:Token: None 2022-09-09T22:39:34.863 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/obj-test-zone2 2022-09-09T22:39:34.864 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:oGWE+GhHpap2vmNXZ5imuU7FjOk= 2022-09-09T22:39:34.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:oGWE+GhHpap2vmNXZ5imuU7FjOk=', 'Content-Length': '0'} 2022-09-09T22:39:34.868 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b683962a17e4b52-00631bc0a6-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.869 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:34.870 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:34.870 DEBUG:boto:Method: HEAD 2022-09-09T22:39:34.871 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2 2022-09-09T22:39:34.871 DEBUG:boto:Data: 2022-09-09T22:39:34.872 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.872 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:34.873 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.874 DEBUG:boto:Params: {} 2022-09-09T22:39:34.874 DEBUG:boto:Token: None 2022-09-09T22:39:34.875 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/obj-test-zone2 2022-09-09T22:39:34.875 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:oGWE+GhHpap2vmNXZ5imuU7FjOk= 2022-09-09T22:39:34.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:oGWE+GhHpap2vmNXZ5imuU7FjOk=', 'Content-Length': '0'} 2022-09-09T22:39:34.880 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000afbcdca4f182f825-00631bc0a6-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:39:34.881 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:34.882 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:34.882 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.883 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.883 DEBUG:boto:Method: GET 2022-09-09T22:39:34.884 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.885 DEBUG:boto:Data: 2022-09-09T22:39:34.885 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.886 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:34.886 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.887 DEBUG:boto:Params: {} 2022-09-09T22:39:34.888 DEBUG:boto:Token: None 2022-09-09T22:39:34.888 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.889 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LTeqoIoelP5ihq9+LqvwzlLVGT8= 2022-09-09T22:39:34.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LTeqoIoelP5ihq9+LqvwzlLVGT8=', 'Content-Length': '0'} 2022-09-09T22:39:34.893 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c2dad3fe4dc22a4-00631bc0a6-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.934 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:34.935 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:34.935 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.936 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.936 DEBUG:boto:Method: GET 2022-09-09T22:39:34.937 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.937 DEBUG:boto:Data: 2022-09-09T22:39:34.938 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.939 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:34.939 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.940 DEBUG:boto:Params: {} 2022-09-09T22:39:34.940 DEBUG:boto:Token: None 2022-09-09T22:39:34.941 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:34.941 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LTeqoIoelP5ihq9+LqvwzlLVGT8= 2022-09-09T22:39:34.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LTeqoIoelP5ihq9+LqvwzlLVGT8=', 'Content-Length': '0'} 2022-09-09T22:39:34.945 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000087c866a4cc762fde-00631bc0a6-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.946 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:34.947 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:34.947 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:34.948 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:34.948 DEBUG:boto:Method: GET 2022-09-09T22:39:34.949 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:34.949 DEBUG:boto:Data: 2022-09-09T22:39:34.950 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.950 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:34.951 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.951 DEBUG:boto:Params: {} 2022-09-09T22:39:34.952 DEBUG:boto:Token: None 2022-09-09T22:39:34.953 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/ 2022-09-09T22:39:34.953 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:iwIWse8TI1qtX/P4n0lZerBCIyY= 2022-09-09T22:39:34.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:iwIWse8TI1qtX/P4n0lZerBCIyY=', 'Content-Length': '0'} 2022-09-09T22:39:34.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d909a8b8ae4bda2-00631bc0a6-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.960 DEBUG:boto:b'jkcgbj-18obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:34.961 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:34.961 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:34.962 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:34.962 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:34.963 DEBUG:boto:Method: GET 2022-09-09T22:39:34.963 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:34.964 DEBUG:boto:Data: 2022-09-09T22:39:34.964 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.965 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:34.965 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.966 DEBUG:boto:Params: {} 2022-09-09T22:39:34.967 DEBUG:boto:Token: None 2022-09-09T22:39:34.967 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/ 2022-09-09T22:39:34.968 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:iwIWse8TI1qtX/P4n0lZerBCIyY= 2022-09-09T22:39:34.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:iwIWse8TI1qtX/P4n0lZerBCIyY=', 'Content-Length': '0'} 2022-09-09T22:39:34.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007af478e6a48aee23-00631bc0a6-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.975 DEBUG:boto:b'jkcgbj-18obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:34.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:39:34.976 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:34.976 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:34.977 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:34.977 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:34.978 DEBUG:boto:Method: GET 2022-09-09T22:39:34.979 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:34.979 DEBUG:boto:Data: 2022-09-09T22:39:34.980 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.980 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:34.981 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.981 DEBUG:boto:Params: {} 2022-09-09T22:39:34.982 DEBUG:boto:Token: None 2022-09-09T22:39:34.983 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:34.983 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:U8CnHuHj5PqfED2XHtJ+4Pyq/eU= 2022-09-09T22:39:34.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:U8CnHuHj5PqfED2XHtJ+4Pyq/eU=', 'Content-Length': '0'} 2022-09-09T22:39:34.987 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e41930955328062c-00631bc0a6-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:34.988 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:34.989 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:34.989 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:34.989 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:34.990 DEBUG:boto:Method: GET 2022-09-09T22:39:34.990 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:34.991 DEBUG:boto:Data: 2022-09-09T22:39:34.992 DEBUG:boto:Headers: {} 2022-09-09T22:39:34.992 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:34.993 DEBUG:boto:Port: 8000 2022-09-09T22:39:34.993 DEBUG:boto:Params: {} 2022-09-09T22:39:34.994 DEBUG:boto:Token: None 2022-09-09T22:39:34.994 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:34 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:34.995 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:U8CnHuHj5PqfED2XHtJ+4Pyq/eU= 2022-09-09T22:39:34.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:34 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:U8CnHuHj5PqfED2XHtJ+4Pyq/eU=', 'Content-Length': '0'} 2022-09-09T22:39:34.999 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008750151497f1ebc3-00631bc0a6-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.000 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:35.000 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:35.001 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:35.001 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:35.002 DEBUG:boto:Method: HEAD 2022-09-09T22:39:35.003 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:35.003 DEBUG:boto:Data: 2022-09-09T22:39:35.004 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.004 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:35.005 DEBUG:boto:Port: 8000 2022-09-09T22:39:35.005 DEBUG:boto:Params: {} 2022-09-09T22:39:35.006 DEBUG:boto:Token: None 2022-09-09T22:39:35.006 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:35.007 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:SJI1Ohs29yQgoJUO4yv77Ia6C4Q= 2022-09-09T22:39:35.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:SJI1Ohs29yQgoJUO4yv77Ia6C4Q=', 'Content-Length': '0'} 2022-09-09T22:39:35.011 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000062adb9296bf92e4f-00631bc0a7-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.012 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:35.012 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:35.013 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:35.013 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:35.014 DEBUG:boto:Method: HEAD 2022-09-09T22:39:35.014 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:35.015 DEBUG:boto:Data: 2022-09-09T22:39:35.015 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.016 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:35.016 DEBUG:boto:Port: 8000 2022-09-09T22:39:35.017 DEBUG:boto:Params: {} 2022-09-09T22:39:35.017 DEBUG:boto:Token: None 2022-09-09T22:39:35.018 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:35.018 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:SJI1Ohs29yQgoJUO4yv77Ia6C4Q= 2022-09-09T22:39:35.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:SJI1Ohs29yQgoJUO4yv77Ia6C4Q=', 'Content-Length': '0'} 2022-09-09T22:39:35.022 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d48d449b3af4d1f6-00631bc0a7-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:39:35.024 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:35.024 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:35.025 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:35.025 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:35.026 DEBUG:boto:Method: GET 2022-09-09T22:39:35.026 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:35.027 DEBUG:boto:Data: 2022-09-09T22:39:35.027 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.028 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:35.028 DEBUG:boto:Port: 8000 2022-09-09T22:39:35.029 DEBUG:boto:Params: {} 2022-09-09T22:39:35.029 DEBUG:boto:Token: None 2022-09-09T22:39:35.030 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:35.031 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:QvRVsVM/ZrNCeFa0y1Og2gJ5HdI= 2022-09-09T22:39:35.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:QvRVsVM/ZrNCeFa0y1Og2gJ5HdI=', 'Content-Length': '0'} 2022-09-09T22:39:35.034 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006cc8e0d23d1e2a87-00631bc0a7-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.078 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:35.079 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:35.079 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:35.080 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:35.080 DEBUG:boto:Method: GET 2022-09-09T22:39:35.081 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:35.081 DEBUG:boto:Data: 2022-09-09T22:39:35.082 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.082 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:35.083 DEBUG:boto:Port: 8000 2022-09-09T22:39:35.083 DEBUG:boto:Params: {} 2022-09-09T22:39:35.084 DEBUG:boto:Token: None 2022-09-09T22:39:35.085 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:35.085 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:QvRVsVM/ZrNCeFa0y1Og2gJ5HdI= 2022-09-09T22:39:35.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:QvRVsVM/ZrNCeFa0y1Og2gJ5HdI=', 'Content-Length': '0'} 2022-09-09T22:39:35.090 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b8e45c544d6a6b1-00631bc0a7-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.091 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:35.091 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:35.092 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:35.092 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:35.093 DEBUG:boto:Method: GET 2022-09-09T22:39:35.093 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:35.094 DEBUG:boto:Data: 2022-09-09T22:39:35.094 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.095 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:35.095 DEBUG:boto:Port: 8000 2022-09-09T22:39:35.096 DEBUG:boto:Params: {} 2022-09-09T22:39:35.096 DEBUG:boto:Token: None 2022-09-09T22:39:35.097 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/ 2022-09-09T22:39:35.098 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4g1ZDrPEpy6T2w/03nrjjWrlAN4= 2022-09-09T22:39:35.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4g1ZDrPEpy6T2w/03nrjjWrlAN4=', 'Content-Length': '0'} 2022-09-09T22:39:35.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2918811f6b53f25-00631bc0a7-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.104 DEBUG:boto:b'jkcgbj-18obj-test-zone31000falseobj-test-zone32022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:35.105 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:35.106 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:35.106 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:35.107 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:35.108 DEBUG:boto:Method: GET 2022-09-09T22:39:35.108 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:35.109 DEBUG:boto:Data: 2022-09-09T22:39:35.109 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.110 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:35.110 DEBUG:boto:Port: 8000 2022-09-09T22:39:35.111 DEBUG:boto:Params: {} 2022-09-09T22:39:35.111 DEBUG:boto:Token: None 2022-09-09T22:39:35.112 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/ 2022-09-09T22:39:35.113 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4g1ZDrPEpy6T2w/03nrjjWrlAN4= 2022-09-09T22:39:35.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4g1ZDrPEpy6T2w/03nrjjWrlAN4=', 'Content-Length': '0'} 2022-09-09T22:39:35.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b789f54aa66eb66-00631bc0a7-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.162 DEBUG:boto:b'jkcgbj-18obj-test-zone31000falseobj-test-zone32022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:35.163 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:35.164 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:35.164 DEBUG:boto:Method: HEAD 2022-09-09T22:39:35.165 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3 2022-09-09T22:39:35.166 DEBUG:boto:Data: 2022-09-09T22:39:35.166 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.167 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:35.167 DEBUG:boto:Port: 8000 2022-09-09T22:39:35.168 DEBUG:boto:Params: {} 2022-09-09T22:39:35.169 DEBUG:boto:Token: None 2022-09-09T22:39:35.169 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/obj-test-zone3 2022-09-09T22:39:35.170 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9zBXUJLbFONgYOtjY29mjK34c3A= 2022-09-09T22:39:35.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9zBXUJLbFONgYOtjY29mjK34c3A=', 'Content-Length': '0'} 2022-09-09T22:39:35.175 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008aee56411f0a8bd6-00631bc0a7-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.175 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:35.176 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:35.177 DEBUG:boto:Method: HEAD 2022-09-09T22:39:35.177 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3 2022-09-09T22:39:35.178 DEBUG:boto:Data: 2022-09-09T22:39:35.178 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.179 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:35.179 DEBUG:boto:Port: 8000 2022-09-09T22:39:35.180 DEBUG:boto:Params: {} 2022-09-09T22:39:35.180 DEBUG:boto:Token: None 2022-09-09T22:39:35.181 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/obj-test-zone3 2022-09-09T22:39:35.181 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9zBXUJLbFONgYOtjY29mjK34c3A= 2022-09-09T22:39:35.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9zBXUJLbFONgYOtjY29mjK34c3A=', 'Content-Length': '0'} 2022-09-09T22:39:35.186 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029eab8a2ba5534c3-00631bc0a7-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:39:35.187 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:35.188 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:35.188 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:35.189 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:35.189 DEBUG:boto:Method: GET 2022-09-09T22:39:35.190 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:35.191 DEBUG:boto:Data: 2022-09-09T22:39:35.191 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.192 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:35.192 DEBUG:boto:Port: 8000 2022-09-09T22:39:35.193 DEBUG:boto:Params: {} 2022-09-09T22:39:35.193 DEBUG:boto:Token: None 2022-09-09T22:39:35.194 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:35.195 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:QvRVsVM/ZrNCeFa0y1Og2gJ5HdI= 2022-09-09T22:39:35.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:QvRVsVM/ZrNCeFa0y1Og2gJ5HdI=', 'Content-Length': '0'} 2022-09-09T22:39:35.198 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ef6884cdb0497bb-00631bc0a7-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.199 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:35.200 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:35.200 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:35.201 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:35.201 DEBUG:boto:Method: GET 2022-09-09T22:39:35.202 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:35.203 DEBUG:boto:Data: 2022-09-09T22:39:35.203 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.204 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:35.204 DEBUG:boto:Port: 8000 2022-09-09T22:39:35.205 DEBUG:boto:Params: {} 2022-09-09T22:39:35.205 DEBUG:boto:Token: None 2022-09-09T22:39:35.206 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:35.207 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:QvRVsVM/ZrNCeFa0y1Og2gJ5HdI= 2022-09-09T22:39:35.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:QvRVsVM/ZrNCeFa0y1Og2gJ5HdI=', 'Content-Length': '0'} 2022-09-09T22:39:35.210 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f68fc8c1594edb7a-00631bc0a7-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.211 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:35.212 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:35.212 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:35.213 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:35.213 DEBUG:boto:Method: GET 2022-09-09T22:39:35.214 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:35.215 DEBUG:boto:Data: 2022-09-09T22:39:35.215 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.216 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:35.216 DEBUG:boto:Port: 8000 2022-09-09T22:39:35.217 DEBUG:boto:Params: {} 2022-09-09T22:39:35.217 DEBUG:boto:Token: None 2022-09-09T22:39:35.218 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/ 2022-09-09T22:39:35.219 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4g1ZDrPEpy6T2w/03nrjjWrlAN4= 2022-09-09T22:39:35.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4g1ZDrPEpy6T2w/03nrjjWrlAN4=', 'Content-Length': '0'} 2022-09-09T22:39:35.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056b61ba1c0c9425e-00631bc0a7-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.266 DEBUG:boto:b'jkcgbj-18obj-test-zone31000falseobj-test-zone32022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:35.267 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:35.268 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:35.268 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:35.269 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:35.269 DEBUG:boto:Method: GET 2022-09-09T22:39:35.270 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:35.270 DEBUG:boto:Data: 2022-09-09T22:39:35.271 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.272 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:35.272 DEBUG:boto:Port: 8000 2022-09-09T22:39:35.273 DEBUG:boto:Params: {} 2022-09-09T22:39:35.273 DEBUG:boto:Token: None 2022-09-09T22:39:35.274 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/ 2022-09-09T22:39:35.275 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4g1ZDrPEpy6T2w/03nrjjWrlAN4= 2022-09-09T22:39:35.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4g1ZDrPEpy6T2w/03nrjjWrlAN4=', 'Content-Length': '0'} 2022-09-09T22:39:35.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3afc18c60dca435-00631bc0a7-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.281 DEBUG:boto:b'jkcgbj-18obj-test-zone31000falseobj-test-zone32022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:35.282 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-18 zones={test-zone2, test-zone1} 2022-09-09T22:39:35.283 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-18 zones={test-zone3, test-zone1} 2022-09-09T22:39:35.283 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:35.284 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:35.284 DEBUG:boto:Method: HEAD 2022-09-09T22:39:35.285 DEBUG:boto:Path: /jkcgbj-18/ 2022-09-09T22:39:35.285 DEBUG:boto:Data: 2022-09-09T22:39:35.286 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.286 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:35.287 DEBUG:boto:Port: 8001 2022-09-09T22:39:35.287 DEBUG:boto:Params: {} 2022-09-09T22:39:35.288 DEBUG:boto:Token: None 2022-09-09T22:39:35.289 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/ 2022-09-09T22:39:35.289 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bTF6sxVfnXk6sWWVJg/dstcE3vI= 2022-09-09T22:39:35.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bTF6sxVfnXk6sWWVJg/dstcE3vI=', 'Content-Length': '0'} 2022-09-09T22:39:35.294 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006250ff0dddef4f7d-00631bc0a7-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.295 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:35.296 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:35.296 DEBUG:boto:Method: HEAD 2022-09-09T22:39:35.297 DEBUG:boto:Path: /jkcgbj-18/ 2022-09-09T22:39:35.297 DEBUG:boto:Data: 2022-09-09T22:39:35.298 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.298 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:35.299 DEBUG:boto:Port: 8000 2022-09-09T22:39:35.299 DEBUG:boto:Params: {} 2022-09-09T22:39:35.300 DEBUG:boto:Token: None 2022-09-09T22:39:35.300 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/ 2022-09-09T22:39:35.301 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:bTF6sxVfnXk6sWWVJg/dstcE3vI= 2022-09-09T22:39:35.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:bTF6sxVfnXk6sWWVJg/dstcE3vI=', 'Content-Length': '0'} 2022-09-09T22:39:35.306 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e555d98a39263d43-00631bc0a7-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.307 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:39:35.307 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:35.308 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:35.309 DEBUG:boto:path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:35.309 DEBUG:boto:auth_path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:35.310 DEBUG:boto:Method: GET 2022-09-09T22:39:35.310 DEBUG:boto:Path: /jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:35.311 DEBUG:boto:Data: 2022-09-09T22:39:35.311 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.312 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:35.312 DEBUG:boto:Port: 8001 2022-09-09T22:39:35.313 DEBUG:boto:Params: {} 2022-09-09T22:39:35.313 DEBUG:boto:Token: None 2022-09-09T22:39:35.314 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/?versions 2022-09-09T22:39:35.315 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:/uvD4Gg3T4azbXGvDMPLJWBOzJs= 2022-09-09T22:39:35.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:/uvD4Gg3T4azbXGvDMPLJWBOzJs=', 'Content-Length': '0'} 2022-09-09T22:39:35.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003636c70933b02027-00631bc0a7-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.322 DEBUG:boto:b'jkcgbj-18999falsedummyRQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5true2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1a8lrwM74XPlF1mwXf8hNl9Y9ISdV53ltrue2022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2e.gQ6yFECscD-oc4OV0dmk6aAg9jH.otrue2022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1true2022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:35.323 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:39:35.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:39:35.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:39:35.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:39:35.325 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:39:35.325 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:35.326 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:35.326 DEBUG:boto:path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:35.327 DEBUG:boto:auth_path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:35.327 DEBUG:boto:Method: GET 2022-09-09T22:39:35.328 DEBUG:boto:Path: /jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:35.328 DEBUG:boto:Data: 2022-09-09T22:39:35.329 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.329 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:35.330 DEBUG:boto:Port: 8000 2022-09-09T22:39:35.330 DEBUG:boto:Params: {} 2022-09-09T22:39:35.331 DEBUG:boto:Token: None 2022-09-09T22:39:35.331 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/?versions 2022-09-09T22:39:35.332 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:/uvD4Gg3T4azbXGvDMPLJWBOzJs= 2022-09-09T22:39:35.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:/uvD4Gg3T4azbXGvDMPLJWBOzJs=', 'Content-Length': '0'} 2022-09-09T22:39:35.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000541c99d0f2fbc649-00631bc0a7-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.339 DEBUG:boto:b'jkcgbj-18999falsedummyRQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5true2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1a8lrwM74XPlF1mwXf8hNl9Y9ISdV53ltrue2022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2e.gQ6yFECscD-oc4OV0dmk6aAg9jH.otrue2022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1true2022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:35.340 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:39:35.340 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:39:35.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:39:35.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:39:35.342 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:35.343 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:35.343 DEBUG:boto:path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:35.344 DEBUG:boto:auth_path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:35.344 DEBUG:boto:Method: GET 2022-09-09T22:39:35.345 DEBUG:boto:Path: /jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:35.345 DEBUG:boto:Data: 2022-09-09T22:39:35.346 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.346 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:35.347 DEBUG:boto:Port: 8001 2022-09-09T22:39:35.347 DEBUG:boto:Params: {} 2022-09-09T22:39:35.348 DEBUG:boto:Token: None 2022-09-09T22:39:35.348 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/?versions 2022-09-09T22:39:35.349 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:/uvD4Gg3T4azbXGvDMPLJWBOzJs= 2022-09-09T22:39:35.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:/uvD4Gg3T4azbXGvDMPLJWBOzJs=', 'Content-Length': '0'} 2022-09-09T22:39:35.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4fe9d239c055d9a-00631bc0a7-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.356 DEBUG:boto:b'jkcgbj-18999falsedummyRQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5true2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1a8lrwM74XPlF1mwXf8hNl9Y9ISdV53ltrue2022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2e.gQ6yFECscD-oc4OV0dmk6aAg9jH.otrue2022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1true2022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:35.357 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:35.358 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:35.358 DEBUG:boto:path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:35.359 DEBUG:boto:auth_path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:35.359 DEBUG:boto:Method: GET 2022-09-09T22:39:35.360 DEBUG:boto:Path: /jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:35.360 DEBUG:boto:Data: 2022-09-09T22:39:35.361 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.361 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:35.362 DEBUG:boto:Port: 8000 2022-09-09T22:39:35.363 DEBUG:boto:Params: {} 2022-09-09T22:39:35.363 DEBUG:boto:Token: None 2022-09-09T22:39:35.364 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/?versions 2022-09-09T22:39:35.364 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:/uvD4Gg3T4azbXGvDMPLJWBOzJs= 2022-09-09T22:39:35.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:/uvD4Gg3T4azbXGvDMPLJWBOzJs=', 'Content-Length': '0'} 2022-09-09T22:39:35.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abe27b16147693db-00631bc0a7-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.371 DEBUG:boto:b'jkcgbj-18999falsedummyRQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5true2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1a8lrwM74XPlF1mwXf8hNl9Y9ISdV53ltrue2022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2e.gQ6yFECscD-oc4OV0dmk6aAg9jH.otrue2022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1true2022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:35.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:39:35.373 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:39:35.373 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:39:35.374 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.374 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.375 DEBUG:boto:Method: GET 2022-09-09T22:39:35.375 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.376 DEBUG:boto:Data: 2022-09-09T22:39:35.376 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.377 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:35.377 DEBUG:boto:Port: 8001 2022-09-09T22:39:35.378 DEBUG:boto:Params: {} 2022-09-09T22:39:35.379 DEBUG:boto:Token: None 2022-09-09T22:39:35.379 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.380 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JUFn6Xv6jj9Rxb+Q6ZyP48MTVU8= 2022-09-09T22:39:35.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JUFn6Xv6jj9Rxb+Q6ZyP48MTVU8=', 'Content-Length': '0'} 2022-09-09T22:39:35.383 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000697de8d2ec8afa71-00631bc0a7-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.384 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:39:35.385 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:39:35.385 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.386 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.386 DEBUG:boto:Method: GET 2022-09-09T22:39:35.387 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.387 DEBUG:boto:Data: 2022-09-09T22:39:35.388 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.388 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:35.389 DEBUG:boto:Port: 8000 2022-09-09T22:39:35.389 DEBUG:boto:Params: {} 2022-09-09T22:39:35.390 DEBUG:boto:Token: None 2022-09-09T22:39:35.391 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.391 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JUFn6Xv6jj9Rxb+Q6ZyP48MTVU8= 2022-09-09T22:39:35.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JUFn6Xv6jj9Rxb+Q6ZyP48MTVU8=', 'Content-Length': '0'} 2022-09-09T22:39:35.395 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ab387eebe74a3109-00631bc0a7-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.396 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:39:35.396 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:39:35.397 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.397 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.398 DEBUG:boto:Method: HEAD 2022-09-09T22:39:35.398 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.399 DEBUG:boto:Data: 2022-09-09T22:39:35.399 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.400 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:35.400 DEBUG:boto:Port: 8001 2022-09-09T22:39:35.401 DEBUG:boto:Params: {} 2022-09-09T22:39:35.401 DEBUG:boto:Token: None 2022-09-09T22:39:35.402 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.403 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Rvb9hr5SYYnxng/SV6GaOP/ZXEk= 2022-09-09T22:39:35.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Rvb9hr5SYYnxng/SV6GaOP/ZXEk=', 'Content-Length': '0'} 2022-09-09T22:39:35.406 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000166e4e79ee0b75ec-00631bc0a7-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.407 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:39:35.408 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:39:35.408 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.409 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.409 DEBUG:boto:Method: HEAD 2022-09-09T22:39:35.410 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.410 DEBUG:boto:Data: 2022-09-09T22:39:35.411 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.411 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:35.412 DEBUG:boto:Port: 8000 2022-09-09T22:39:35.412 DEBUG:boto:Params: {} 2022-09-09T22:39:35.413 DEBUG:boto:Token: None 2022-09-09T22:39:35.413 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.414 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Rvb9hr5SYYnxng/SV6GaOP/ZXEk= 2022-09-09T22:39:35.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Rvb9hr5SYYnxng/SV6GaOP/ZXEk=', 'Content-Length': '0'} 2022-09-09T22:39:35.418 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008e4875973520a022-00631bc0a7-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:39:35.419 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:39:35.420 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:39:35.420 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.421 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.421 DEBUG:boto:Method: GET 2022-09-09T22:39:35.422 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.422 DEBUG:boto:Data: 2022-09-09T22:39:35.423 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.423 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:35.424 DEBUG:boto:Port: 8001 2022-09-09T22:39:35.424 DEBUG:boto:Params: {} 2022-09-09T22:39:35.425 DEBUG:boto:Token: None 2022-09-09T22:39:35.426 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.426 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JUFn6Xv6jj9Rxb+Q6ZyP48MTVU8= 2022-09-09T22:39:35.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JUFn6Xv6jj9Rxb+Q6ZyP48MTVU8=', 'Content-Length': '0'} 2022-09-09T22:39:35.430 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000766cedb7f923a0e7-00631bc0a7-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.431 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:39:35.431 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:39:35.432 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.432 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.433 DEBUG:boto:Method: GET 2022-09-09T22:39:35.433 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.434 DEBUG:boto:Data: 2022-09-09T22:39:35.434 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.435 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:35.435 DEBUG:boto:Port: 8000 2022-09-09T22:39:35.436 DEBUG:boto:Params: {} 2022-09-09T22:39:35.437 DEBUG:boto:Token: None 2022-09-09T22:39:35.437 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.438 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JUFn6Xv6jj9Rxb+Q6ZyP48MTVU8= 2022-09-09T22:39:35.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JUFn6Xv6jj9Rxb+Q6ZyP48MTVU8=', 'Content-Length': '0'} 2022-09-09T22:39:35.442 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000123645ff09824eb-00631bc0a7-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.443 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:35.443 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:35.444 DEBUG:boto:path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:39:35.444 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:39:35.445 DEBUG:boto:Method: GET 2022-09-09T22:39:35.445 DEBUG:boto:Path: /jkcgbj-18/?prefix=dummy 2022-09-09T22:39:35.446 DEBUG:boto:Data: 2022-09-09T22:39:35.446 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.447 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:35.447 DEBUG:boto:Port: 8001 2022-09-09T22:39:35.448 DEBUG:boto:Params: {} 2022-09-09T22:39:35.448 DEBUG:boto:Token: None 2022-09-09T22:39:35.449 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/ 2022-09-09T22:39:35.450 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4g1ZDrPEpy6T2w/03nrjjWrlAN4= 2022-09-09T22:39:35.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4g1ZDrPEpy6T2w/03nrjjWrlAN4=', 'Content-Length': '0'} 2022-09-09T22:39:35.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ad4526a078cbfde-00631bc0a7-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.510 DEBUG:boto:b'jkcgbj-18dummy1000falsedummy2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:35.511 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:35.512 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:35.512 DEBUG:boto:path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:39:35.513 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:39:35.513 DEBUG:boto:Method: GET 2022-09-09T22:39:35.514 DEBUG:boto:Path: /jkcgbj-18/?prefix=dummy 2022-09-09T22:39:35.515 DEBUG:boto:Data: 2022-09-09T22:39:35.515 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.515 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:35.516 DEBUG:boto:Port: 8000 2022-09-09T22:39:35.516 DEBUG:boto:Params: {} 2022-09-09T22:39:35.517 DEBUG:boto:Token: None 2022-09-09T22:39:35.518 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/ 2022-09-09T22:39:35.519 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4g1ZDrPEpy6T2w/03nrjjWrlAN4= 2022-09-09T22:39:35.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4g1ZDrPEpy6T2w/03nrjjWrlAN4=', 'Content-Length': '0'} 2022-09-09T22:39:35.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3035d3457ae4805-00631bc0a7-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.525 DEBUG:boto:b'jkcgbj-18dummy1000falsedummy2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:35.526 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:39:35.527 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:39:35.527 DEBUG:boto:Method: HEAD 2022-09-09T22:39:35.528 DEBUG:boto:Path: /jkcgbj-18/dummy 2022-09-09T22:39:35.528 DEBUG:boto:Data: 2022-09-09T22:39:35.529 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.529 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:35.530 DEBUG:boto:Port: 8001 2022-09-09T22:39:35.530 DEBUG:boto:Params: {} 2022-09-09T22:39:35.531 DEBUG:boto:Token: None 2022-09-09T22:39:35.531 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/dummy 2022-09-09T22:39:35.532 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:WvgxLsb3qCskERwHkCUaKgyFlsc= 2022-09-09T22:39:35.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:WvgxLsb3qCskERwHkCUaKgyFlsc=', 'Content-Length': '0'} 2022-09-09T22:39:35.536 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005ebfaa357781fbbf-00631bc0a7-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.537 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:39:35.537 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:39:35.538 DEBUG:boto:Method: HEAD 2022-09-09T22:39:35.538 DEBUG:boto:Path: /jkcgbj-18/dummy 2022-09-09T22:39:35.539 DEBUG:boto:Data: 2022-09-09T22:39:35.540 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.540 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:35.541 DEBUG:boto:Port: 8000 2022-09-09T22:39:35.541 DEBUG:boto:Params: {} 2022-09-09T22:39:35.542 DEBUG:boto:Token: None 2022-09-09T22:39:35.542 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/dummy 2022-09-09T22:39:35.543 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:WvgxLsb3qCskERwHkCUaKgyFlsc= 2022-09-09T22:39:35.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:WvgxLsb3qCskERwHkCUaKgyFlsc=', 'Content-Length': '0'} 2022-09-09T22:39:35.547 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cf0367dd9525c35b-00631bc0a7-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:39:35.549 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:39:35.549 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:39:35.550 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.550 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.551 DEBUG:boto:Method: GET 2022-09-09T22:39:35.551 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.552 DEBUG:boto:Data: 2022-09-09T22:39:35.552 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.553 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:35.553 DEBUG:boto:Port: 8001 2022-09-09T22:39:35.554 DEBUG:boto:Params: {} 2022-09-09T22:39:35.554 DEBUG:boto:Token: None 2022-09-09T22:39:35.555 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.555 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JUFn6Xv6jj9Rxb+Q6ZyP48MTVU8= 2022-09-09T22:39:35.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JUFn6Xv6jj9Rxb+Q6ZyP48MTVU8=', 'Content-Length': '0'} 2022-09-09T22:39:35.559 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c59e1f9b09e4ceff-00631bc0a7-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.560 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:39:35.560 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:39:35.561 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.561 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.562 DEBUG:boto:Method: GET 2022-09-09T22:39:35.562 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.563 DEBUG:boto:Data: 2022-09-09T22:39:35.563 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.564 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:35.564 DEBUG:boto:Port: 8000 2022-09-09T22:39:35.565 DEBUG:boto:Params: {} 2022-09-09T22:39:35.565 DEBUG:boto:Token: None 2022-09-09T22:39:35.566 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:35.567 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JUFn6Xv6jj9Rxb+Q6ZyP48MTVU8= 2022-09-09T22:39:35.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JUFn6Xv6jj9Rxb+Q6ZyP48MTVU8=', 'Content-Length': '0'} 2022-09-09T22:39:35.570 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003907aff6eb450031-00631bc0a7-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.571 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:35.572 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:35.572 DEBUG:boto:path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:39:35.573 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:39:35.573 DEBUG:boto:Method: GET 2022-09-09T22:39:35.574 DEBUG:boto:Path: /jkcgbj-18/?prefix=dummy 2022-09-09T22:39:35.574 DEBUG:boto:Data: 2022-09-09T22:39:35.575 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.575 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:35.576 DEBUG:boto:Port: 8001 2022-09-09T22:39:35.576 DEBUG:boto:Params: {} 2022-09-09T22:39:35.577 DEBUG:boto:Token: None 2022-09-09T22:39:35.577 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/ 2022-09-09T22:39:35.578 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4g1ZDrPEpy6T2w/03nrjjWrlAN4= 2022-09-09T22:39:35.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4g1ZDrPEpy6T2w/03nrjjWrlAN4=', 'Content-Length': '0'} 2022-09-09T22:39:35.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052519a740da15c90-00631bc0a7-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.626 DEBUG:boto:b'jkcgbj-18dummy1000falsedummy2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:35.627 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:35.628 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:35.628 DEBUG:boto:path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:39:35.629 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:39:35.629 DEBUG:boto:Method: GET 2022-09-09T22:39:35.630 DEBUG:boto:Path: /jkcgbj-18/?prefix=dummy 2022-09-09T22:39:35.630 DEBUG:boto:Data: 2022-09-09T22:39:35.631 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.631 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:35.632 DEBUG:boto:Port: 8000 2022-09-09T22:39:35.632 DEBUG:boto:Params: {} 2022-09-09T22:39:35.633 DEBUG:boto:Token: None 2022-09-09T22:39:35.633 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/ 2022-09-09T22:39:35.634 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4g1ZDrPEpy6T2w/03nrjjWrlAN4= 2022-09-09T22:39:35.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4g1ZDrPEpy6T2w/03nrjjWrlAN4=', 'Content-Length': '0'} 2022-09-09T22:39:35.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcee2a17fc664883-00631bc0a7-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.641 DEBUG:boto:b'jkcgbj-18dummy1000falsedummy2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:35.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:39:35.642 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:35.643 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:35.643 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.644 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.644 DEBUG:boto:Method: GET 2022-09-09T22:39:35.645 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.645 DEBUG:boto:Data: 2022-09-09T22:39:35.646 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.646 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:35.647 DEBUG:boto:Port: 8001 2022-09-09T22:39:35.647 DEBUG:boto:Params: {} 2022-09-09T22:39:35.648 DEBUG:boto:Token: None 2022-09-09T22:39:35.648 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.649 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JaMpG7NV9U3vWWwR468B4DzFQwQ= 2022-09-09T22:39:35.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JaMpG7NV9U3vWWwR468B4DzFQwQ=', 'Content-Length': '0'} 2022-09-09T22:39:35.652 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ac6ed9c29183244-00631bc0a7-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.653 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:35.654 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:35.655 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.656 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.656 DEBUG:boto:Method: GET 2022-09-09T22:39:35.657 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.658 DEBUG:boto:Data: 2022-09-09T22:39:35.659 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.659 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:35.660 DEBUG:boto:Port: 8000 2022-09-09T22:39:35.661 DEBUG:boto:Params: {} 2022-09-09T22:39:35.662 DEBUG:boto:Token: None 2022-09-09T22:39:35.663 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.664 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JaMpG7NV9U3vWWwR468B4DzFQwQ= 2022-09-09T22:39:35.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JaMpG7NV9U3vWWwR468B4DzFQwQ=', 'Content-Length': '0'} 2022-09-09T22:39:35.668 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f903439da68f373b-00631bc0a7-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.710 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:35.711 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:35.711 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.712 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.712 DEBUG:boto:Method: HEAD 2022-09-09T22:39:35.713 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.713 DEBUG:boto:Data: 2022-09-09T22:39:35.714 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.714 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:35.715 DEBUG:boto:Port: 8001 2022-09-09T22:39:35.715 DEBUG:boto:Params: {} 2022-09-09T22:39:35.716 DEBUG:boto:Token: None 2022-09-09T22:39:35.717 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.717 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UkGR54DYwpTiROIBZ0Ag1l/hN1A= 2022-09-09T22:39:35.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UkGR54DYwpTiROIBZ0Ag1l/hN1A=', 'Content-Length': '0'} 2022-09-09T22:39:35.721 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a7192fd2b0c342ef-00631bc0a7-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.722 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:35.722 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:35.723 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.723 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.724 DEBUG:boto:Method: HEAD 2022-09-09T22:39:35.724 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.725 DEBUG:boto:Data: 2022-09-09T22:39:35.725 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.726 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:35.726 DEBUG:boto:Port: 8000 2022-09-09T22:39:35.727 DEBUG:boto:Params: {} 2022-09-09T22:39:35.727 DEBUG:boto:Token: None 2022-09-09T22:39:35.728 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.728 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UkGR54DYwpTiROIBZ0Ag1l/hN1A= 2022-09-09T22:39:35.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UkGR54DYwpTiROIBZ0Ag1l/hN1A=', 'Content-Length': '0'} 2022-09-09T22:39:35.732 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a851935ae5fee325-00631bc0a7-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:39:35.733 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:35.734 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:35.734 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.735 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.735 DEBUG:boto:Method: GET 2022-09-09T22:39:35.736 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.736 DEBUG:boto:Data: 2022-09-09T22:39:35.737 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.737 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:35.738 DEBUG:boto:Port: 8001 2022-09-09T22:39:35.738 DEBUG:boto:Params: {} 2022-09-09T22:39:35.739 DEBUG:boto:Token: None 2022-09-09T22:39:35.739 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.740 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JaMpG7NV9U3vWWwR468B4DzFQwQ= 2022-09-09T22:39:35.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JaMpG7NV9U3vWWwR468B4DzFQwQ=', 'Content-Length': '0'} 2022-09-09T22:39:35.743 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e1c40b6e9491ce7-00631bc0a7-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.744 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:35.744 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:35.745 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.745 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.746 DEBUG:boto:Method: GET 2022-09-09T22:39:35.746 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.747 DEBUG:boto:Data: 2022-09-09T22:39:35.747 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.748 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:35.748 DEBUG:boto:Port: 8000 2022-09-09T22:39:35.749 DEBUG:boto:Params: {} 2022-09-09T22:39:35.749 DEBUG:boto:Token: None 2022-09-09T22:39:35.750 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.751 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JaMpG7NV9U3vWWwR468B4DzFQwQ= 2022-09-09T22:39:35.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JaMpG7NV9U3vWWwR468B4DzFQwQ=', 'Content-Length': '0'} 2022-09-09T22:39:35.754 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb6efadf7878b725-00631bc0a7-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.755 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:35.755 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:35.756 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:35.756 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:35.757 DEBUG:boto:Method: GET 2022-09-09T22:39:35.757 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:35.758 DEBUG:boto:Data: 2022-09-09T22:39:35.758 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.759 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:35.759 DEBUG:boto:Port: 8001 2022-09-09T22:39:35.760 DEBUG:boto:Params: {} 2022-09-09T22:39:35.760 DEBUG:boto:Token: None 2022-09-09T22:39:35.761 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/ 2022-09-09T22:39:35.762 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4g1ZDrPEpy6T2w/03nrjjWrlAN4= 2022-09-09T22:39:35.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4g1ZDrPEpy6T2w/03nrjjWrlAN4=', 'Content-Length': '0'} 2022-09-09T22:39:35.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8daa54db1cd2b4a-00631bc0a7-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.768 DEBUG:boto:b'jkcgbj-18obj-test-zone11000falseobj-test-zone12022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:35.769 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:35.770 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:35.770 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:35.771 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:35.771 DEBUG:boto:Method: GET 2022-09-09T22:39:35.772 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:35.772 DEBUG:boto:Data: 2022-09-09T22:39:35.773 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.773 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:35.774 DEBUG:boto:Port: 8000 2022-09-09T22:39:35.774 DEBUG:boto:Params: {} 2022-09-09T22:39:35.775 DEBUG:boto:Token: None 2022-09-09T22:39:35.776 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/ 2022-09-09T22:39:35.776 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4g1ZDrPEpy6T2w/03nrjjWrlAN4= 2022-09-09T22:39:35.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4g1ZDrPEpy6T2w/03nrjjWrlAN4=', 'Content-Length': '0'} 2022-09-09T22:39:35.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087a3b6bd7bd821f3-00631bc0a7-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.826 DEBUG:boto:b'jkcgbj-18obj-test-zone11000falseobj-test-zone12022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:35.827 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:35.827 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:35.828 DEBUG:boto:Method: HEAD 2022-09-09T22:39:35.828 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1 2022-09-09T22:39:35.829 DEBUG:boto:Data: 2022-09-09T22:39:35.829 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.830 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:35.830 DEBUG:boto:Port: 8001 2022-09-09T22:39:35.831 DEBUG:boto:Params: {} 2022-09-09T22:39:35.831 DEBUG:boto:Token: None 2022-09-09T22:39:35.832 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/obj-test-zone1 2022-09-09T22:39:35.832 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Y/ehHYRu59creo+YOhWn5gukgmE= 2022-09-09T22:39:35.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Y/ehHYRu59creo+YOhWn5gukgmE=', 'Content-Length': '0'} 2022-09-09T22:39:35.837 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f1afe530df241d35-00631bc0a7-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.838 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:35.838 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:35.839 DEBUG:boto:Method: HEAD 2022-09-09T22:39:35.839 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1 2022-09-09T22:39:35.840 DEBUG:boto:Data: 2022-09-09T22:39:35.840 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.841 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:35.841 DEBUG:boto:Port: 8000 2022-09-09T22:39:35.842 DEBUG:boto:Params: {} 2022-09-09T22:39:35.843 DEBUG:boto:Token: None 2022-09-09T22:39:35.843 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/obj-test-zone1 2022-09-09T22:39:35.844 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Y/ehHYRu59creo+YOhWn5gukgmE= 2022-09-09T22:39:35.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Y/ehHYRu59creo+YOhWn5gukgmE=', 'Content-Length': '0'} 2022-09-09T22:39:35.848 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dabb388b0810ea07-00631bc0a7-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:39:35.850 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:35.850 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:35.851 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.851 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.852 DEBUG:boto:Method: GET 2022-09-09T22:39:35.853 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.853 DEBUG:boto:Data: 2022-09-09T22:39:35.854 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.854 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:35.855 DEBUG:boto:Port: 8001 2022-09-09T22:39:35.855 DEBUG:boto:Params: {} 2022-09-09T22:39:35.856 DEBUG:boto:Token: None 2022-09-09T22:39:35.856 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.857 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JaMpG7NV9U3vWWwR468B4DzFQwQ= 2022-09-09T22:39:35.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JaMpG7NV9U3vWWwR468B4DzFQwQ=', 'Content-Length': '0'} 2022-09-09T22:39:35.861 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000674e6e49ff1d7199-00631bc0a7-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.861 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:35.862 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:35.863 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.863 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.864 DEBUG:boto:Method: GET 2022-09-09T22:39:35.864 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.865 DEBUG:boto:Data: 2022-09-09T22:39:35.865 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.866 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:35.866 DEBUG:boto:Port: 8000 2022-09-09T22:39:35.867 DEBUG:boto:Params: {} 2022-09-09T22:39:35.868 DEBUG:boto:Token: None 2022-09-09T22:39:35.868 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:35.869 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JaMpG7NV9U3vWWwR468B4DzFQwQ= 2022-09-09T22:39:35.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JaMpG7NV9U3vWWwR468B4DzFQwQ=', 'Content-Length': '0'} 2022-09-09T22:39:35.873 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a7c1fa6d413ae3c5-00631bc0a7-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.873 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:35.874 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:35.875 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:35.875 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:35.876 DEBUG:boto:Method: GET 2022-09-09T22:39:35.876 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:35.877 DEBUG:boto:Data: 2022-09-09T22:39:35.877 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.878 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:35.878 DEBUG:boto:Port: 8001 2022-09-09T22:39:35.879 DEBUG:boto:Params: {} 2022-09-09T22:39:35.880 DEBUG:boto:Token: None 2022-09-09T22:39:35.880 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/ 2022-09-09T22:39:35.881 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4g1ZDrPEpy6T2w/03nrjjWrlAN4= 2022-09-09T22:39:35.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4g1ZDrPEpy6T2w/03nrjjWrlAN4=', 'Content-Length': '0'} 2022-09-09T22:39:35.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c91ae1d56358e6fc-00631bc0a7-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.887 DEBUG:boto:b'jkcgbj-18obj-test-zone11000falseobj-test-zone12022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:35.888 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:35.889 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:35.889 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:35.890 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:35.890 DEBUG:boto:Method: GET 2022-09-09T22:39:35.891 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:35.892 DEBUG:boto:Data: 2022-09-09T22:39:35.892 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.893 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:35.893 DEBUG:boto:Port: 8000 2022-09-09T22:39:35.894 DEBUG:boto:Params: {} 2022-09-09T22:39:35.895 DEBUG:boto:Token: None 2022-09-09T22:39:35.895 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/ 2022-09-09T22:39:35.896 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4g1ZDrPEpy6T2w/03nrjjWrlAN4= 2022-09-09T22:39:35.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4g1ZDrPEpy6T2w/03nrjjWrlAN4=', 'Content-Length': '0'} 2022-09-09T22:39:35.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008dd93bdcaa6f8047-00631bc0a7-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.902 DEBUG:boto:b'jkcgbj-18obj-test-zone11000falseobj-test-zone12022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:35.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:39:35.904 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:35.904 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:35.905 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:35.905 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:35.906 DEBUG:boto:Method: GET 2022-09-09T22:39:35.906 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:35.907 DEBUG:boto:Data: 2022-09-09T22:39:35.907 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.908 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:35.909 DEBUG:boto:Port: 8001 2022-09-09T22:39:35.909 DEBUG:boto:Params: {} 2022-09-09T22:39:35.910 DEBUG:boto:Token: None 2022-09-09T22:39:35.910 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:35.911 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rS867m76c42FhQn5mWQCUzaHv9I= 2022-09-09T22:39:35.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rS867m76c42FhQn5mWQCUzaHv9I=', 'Content-Length': '0'} 2022-09-09T22:39:35.915 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc9a8df8db56e7bf-00631bc0a7-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.962 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:35.963 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:35.963 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:35.964 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:35.965 DEBUG:boto:Method: GET 2022-09-09T22:39:35.965 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:35.966 DEBUG:boto:Data: 2022-09-09T22:39:35.966 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.967 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:35.967 DEBUG:boto:Port: 8000 2022-09-09T22:39:35.968 DEBUG:boto:Params: {} 2022-09-09T22:39:35.968 DEBUG:boto:Token: None 2022-09-09T22:39:35.969 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:35.970 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rS867m76c42FhQn5mWQCUzaHv9I= 2022-09-09T22:39:35.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rS867m76c42FhQn5mWQCUzaHv9I=', 'Content-Length': '0'} 2022-09-09T22:39:35.974 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9de6164a621f41a-00631bc0a7-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.974 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:35.975 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:35.975 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:35.976 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:35.976 DEBUG:boto:Method: HEAD 2022-09-09T22:39:35.977 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:35.978 DEBUG:boto:Data: 2022-09-09T22:39:35.978 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.979 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:35.979 DEBUG:boto:Port: 8001 2022-09-09T22:39:35.979 DEBUG:boto:Params: {} 2022-09-09T22:39:35.980 DEBUG:boto:Token: None 2022-09-09T22:39:35.981 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:35.981 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7jmn12YDKcWoifLw93tbRA9wE9A= 2022-09-09T22:39:35.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7jmn12YDKcWoifLw93tbRA9wE9A=', 'Content-Length': '0'} 2022-09-09T22:39:35.985 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a61e2aa374b2eaf-00631bc0a7-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.986 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:35.986 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:35.987 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:35.987 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:35.988 DEBUG:boto:Method: HEAD 2022-09-09T22:39:35.988 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:35.989 DEBUG:boto:Data: 2022-09-09T22:39:35.989 DEBUG:boto:Headers: {} 2022-09-09T22:39:35.990 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:35.990 DEBUG:boto:Port: 8000 2022-09-09T22:39:35.991 DEBUG:boto:Params: {} 2022-09-09T22:39:35.991 DEBUG:boto:Token: None 2022-09-09T22:39:35.992 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:35 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:35.992 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7jmn12YDKcWoifLw93tbRA9wE9A= 2022-09-09T22:39:35.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7jmn12YDKcWoifLw93tbRA9wE9A=', 'Content-Length': '0'} 2022-09-09T22:39:35.996 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3ae24644cf95c2b-00631bc0a7-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:35.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:39:35.997 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:35.997 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:35.998 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:35.999 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:35.999 DEBUG:boto:Method: GET 2022-09-09T22:39:36.000 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:36.000 DEBUG:boto:Data: 2022-09-09T22:39:36.001 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.001 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:36.002 DEBUG:boto:Port: 8001 2022-09-09T22:39:36.002 DEBUG:boto:Params: {} 2022-09-09T22:39:36.003 DEBUG:boto:Token: None 2022-09-09T22:39:36.003 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:36.004 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4EoQKWLSDB54AbICmeheYtwTUXU= 2022-09-09T22:39:36.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4EoQKWLSDB54AbICmeheYtwTUXU=', 'Content-Length': '0'} 2022-09-09T22:39:36.008 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f87eedc3c85fd7af-00631bc0a7-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.009 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:36.009 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:36.010 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:36.010 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:36.011 DEBUG:boto:Method: GET 2022-09-09T22:39:36.012 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:36.012 DEBUG:boto:Data: 2022-09-09T22:39:36.013 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.013 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:36.014 DEBUG:boto:Port: 8000 2022-09-09T22:39:36.014 DEBUG:boto:Params: {} 2022-09-09T22:39:36.015 DEBUG:boto:Token: None 2022-09-09T22:39:36.016 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:36.017 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4EoQKWLSDB54AbICmeheYtwTUXU= 2022-09-09T22:39:36.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4EoQKWLSDB54AbICmeheYtwTUXU=', 'Content-Length': '0'} 2022-09-09T22:39:36.021 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002949a8e0a0b6600a-00631bc0a8-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.062 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:36.063 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:36.063 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:36.064 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:36.064 DEBUG:boto:Method: GET 2022-09-09T22:39:36.065 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:36.066 DEBUG:boto:Data: 2022-09-09T22:39:36.066 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.067 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:36.067 DEBUG:boto:Port: 8001 2022-09-09T22:39:36.068 DEBUG:boto:Params: {} 2022-09-09T22:39:36.069 DEBUG:boto:Token: None 2022-09-09T22:39:36.069 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/ 2022-09-09T22:39:36.070 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pKHhQr+LLYS+2Qua0mRCkmakpJQ= 2022-09-09T22:39:36.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pKHhQr+LLYS+2Qua0mRCkmakpJQ=', 'Content-Length': '0'} 2022-09-09T22:39:36.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008090e7175025e110-00631bc0a8-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.077 DEBUG:boto:b'jkcgbj-18obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:36.078 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:36.078 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:36.079 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:36.079 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:36.080 DEBUG:boto:Method: GET 2022-09-09T22:39:36.080 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:36.081 DEBUG:boto:Data: 2022-09-09T22:39:36.081 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.081 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:36.082 DEBUG:boto:Port: 8000 2022-09-09T22:39:36.083 DEBUG:boto:Params: {} 2022-09-09T22:39:36.083 DEBUG:boto:Token: None 2022-09-09T22:39:36.084 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/ 2022-09-09T22:39:36.084 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pKHhQr+LLYS+2Qua0mRCkmakpJQ= 2022-09-09T22:39:36.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pKHhQr+LLYS+2Qua0mRCkmakpJQ=', 'Content-Length': '0'} 2022-09-09T22:39:36.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004860dca6fcea8e1c-00631bc0a8-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.091 DEBUG:boto:b'jkcgbj-18obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:36.092 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:36.092 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:36.093 DEBUG:boto:Method: HEAD 2022-09-09T22:39:36.093 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2 2022-09-09T22:39:36.094 DEBUG:boto:Data: 2022-09-09T22:39:36.094 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.095 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:36.095 DEBUG:boto:Port: 8001 2022-09-09T22:39:36.096 DEBUG:boto:Params: {} 2022-09-09T22:39:36.096 DEBUG:boto:Token: None 2022-09-09T22:39:36.097 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/obj-test-zone2 2022-09-09T22:39:36.097 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jf43Rt+01vlBQSayEyHAvpcoANs= 2022-09-09T22:39:36.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jf43Rt+01vlBQSayEyHAvpcoANs=', 'Content-Length': '0'} 2022-09-09T22:39:36.102 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007b079509f176f954-00631bc0a8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.103 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:36.103 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:36.104 DEBUG:boto:Method: HEAD 2022-09-09T22:39:36.104 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2 2022-09-09T22:39:36.105 DEBUG:boto:Data: 2022-09-09T22:39:36.105 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.106 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:36.106 DEBUG:boto:Port: 8000 2022-09-09T22:39:36.107 DEBUG:boto:Params: {} 2022-09-09T22:39:36.107 DEBUG:boto:Token: None 2022-09-09T22:39:36.108 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/obj-test-zone2 2022-09-09T22:39:36.108 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jf43Rt+01vlBQSayEyHAvpcoANs= 2022-09-09T22:39:36.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jf43Rt+01vlBQSayEyHAvpcoANs=', 'Content-Length': '0'} 2022-09-09T22:39:36.113 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c8ec8dd5210ce10-00631bc0a8-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:39:36.114 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:36.115 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:36.115 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:36.116 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:36.116 DEBUG:boto:Method: GET 2022-09-09T22:39:36.117 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:36.117 DEBUG:boto:Data: 2022-09-09T22:39:36.118 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.118 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:36.119 DEBUG:boto:Port: 8001 2022-09-09T22:39:36.120 DEBUG:boto:Params: {} 2022-09-09T22:39:36.120 DEBUG:boto:Token: None 2022-09-09T22:39:36.121 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:36.121 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4EoQKWLSDB54AbICmeheYtwTUXU= 2022-09-09T22:39:36.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4EoQKWLSDB54AbICmeheYtwTUXU=', 'Content-Length': '0'} 2022-09-09T22:39:36.125 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b778ce21da6fecc-00631bc0a8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.170 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:36.171 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:36.171 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:36.172 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:36.173 DEBUG:boto:Method: GET 2022-09-09T22:39:36.173 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:36.174 DEBUG:boto:Data: 2022-09-09T22:39:36.174 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.175 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:36.175 DEBUG:boto:Port: 8000 2022-09-09T22:39:36.176 DEBUG:boto:Params: {} 2022-09-09T22:39:36.177 DEBUG:boto:Token: None 2022-09-09T22:39:36.177 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:36.178 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4EoQKWLSDB54AbICmeheYtwTUXU= 2022-09-09T22:39:36.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4EoQKWLSDB54AbICmeheYtwTUXU=', 'Content-Length': '0'} 2022-09-09T22:39:36.182 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d35704cc9ec4a0e-00631bc0a8-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.183 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:36.183 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:36.184 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:36.184 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:36.185 DEBUG:boto:Method: GET 2022-09-09T22:39:36.185 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:36.186 DEBUG:boto:Data: 2022-09-09T22:39:36.187 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.187 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:36.188 DEBUG:boto:Port: 8001 2022-09-09T22:39:36.188 DEBUG:boto:Params: {} 2022-09-09T22:39:36.189 DEBUG:boto:Token: None 2022-09-09T22:39:36.189 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/ 2022-09-09T22:39:36.190 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pKHhQr+LLYS+2Qua0mRCkmakpJQ= 2022-09-09T22:39:36.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pKHhQr+LLYS+2Qua0mRCkmakpJQ=', 'Content-Length': '0'} 2022-09-09T22:39:36.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ccb76d810a29a46-00631bc0a8-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.197 DEBUG:boto:b'jkcgbj-18obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:36.198 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:36.199 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:36.199 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:36.200 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:36.200 DEBUG:boto:Method: GET 2022-09-09T22:39:36.201 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:36.201 DEBUG:boto:Data: 2022-09-09T22:39:36.202 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.203 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:36.203 DEBUG:boto:Port: 8000 2022-09-09T22:39:36.204 DEBUG:boto:Params: {} 2022-09-09T22:39:36.204 DEBUG:boto:Token: None 2022-09-09T22:39:36.205 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/ 2022-09-09T22:39:36.205 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pKHhQr+LLYS+2Qua0mRCkmakpJQ= 2022-09-09T22:39:36.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pKHhQr+LLYS+2Qua0mRCkmakpJQ=', 'Content-Length': '0'} 2022-09-09T22:39:36.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039dd89cf978fc321-00631bc0a8-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.212 DEBUG:boto:b'jkcgbj-18obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:36.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:39:36.213 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:36.214 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:36.215 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.215 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.216 DEBUG:boto:Method: GET 2022-09-09T22:39:36.216 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.217 DEBUG:boto:Data: 2022-09-09T22:39:36.217 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.218 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:36.219 DEBUG:boto:Port: 8001 2022-09-09T22:39:36.219 DEBUG:boto:Params: {} 2022-09-09T22:39:36.220 DEBUG:boto:Token: None 2022-09-09T22:39:36.220 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.221 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2lNiMOHmlz1UkCF08R6TGDALPpc= 2022-09-09T22:39:36.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2lNiMOHmlz1UkCF08R6TGDALPpc=', 'Content-Length': '0'} 2022-09-09T22:39:36.225 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000065029152031ca78e-00631bc0a8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.226 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:36.226 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:36.227 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.227 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.228 DEBUG:boto:Method: GET 2022-09-09T22:39:36.229 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.229 DEBUG:boto:Data: 2022-09-09T22:39:36.230 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.230 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:36.231 DEBUG:boto:Port: 8000 2022-09-09T22:39:36.231 DEBUG:boto:Params: {} 2022-09-09T22:39:36.232 DEBUG:boto:Token: None 2022-09-09T22:39:36.233 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.233 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2lNiMOHmlz1UkCF08R6TGDALPpc= 2022-09-09T22:39:36.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2lNiMOHmlz1UkCF08R6TGDALPpc=', 'Content-Length': '0'} 2022-09-09T22:39:36.237 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053904673fbc315d7-00631bc0a8-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.278 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:36.279 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:36.279 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.280 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.280 DEBUG:boto:Method: HEAD 2022-09-09T22:39:36.281 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.281 DEBUG:boto:Data: 2022-09-09T22:39:36.282 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.282 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:36.283 DEBUG:boto:Port: 8001 2022-09-09T22:39:36.283 DEBUG:boto:Params: {} 2022-09-09T22:39:36.284 DEBUG:boto:Token: None 2022-09-09T22:39:36.285 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.285 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:xxC5iNtp0LlkE+LVlCLoxp/qTQ8= 2022-09-09T22:39:36.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:xxC5iNtp0LlkE+LVlCLoxp/qTQ8=', 'Content-Length': '0'} 2022-09-09T22:39:36.289 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a396a118ddfccf91-00631bc0a8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.290 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:36.291 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:36.291 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.292 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.292 DEBUG:boto:Method: HEAD 2022-09-09T22:39:36.293 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.293 DEBUG:boto:Data: 2022-09-09T22:39:36.294 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.294 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:36.295 DEBUG:boto:Port: 8000 2022-09-09T22:39:36.295 DEBUG:boto:Params: {} 2022-09-09T22:39:36.296 DEBUG:boto:Token: None 2022-09-09T22:39:36.296 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.297 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:xxC5iNtp0LlkE+LVlCLoxp/qTQ8= 2022-09-09T22:39:36.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:xxC5iNtp0LlkE+LVlCLoxp/qTQ8=', 'Content-Length': '0'} 2022-09-09T22:39:36.301 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0c6758f47897e8f-00631bc0a8-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:39:36.302 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:36.302 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:36.303 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.303 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.304 DEBUG:boto:Method: GET 2022-09-09T22:39:36.304 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.305 DEBUG:boto:Data: 2022-09-09T22:39:36.306 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.306 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:36.307 DEBUG:boto:Port: 8001 2022-09-09T22:39:36.307 DEBUG:boto:Params: {} 2022-09-09T22:39:36.308 DEBUG:boto:Token: None 2022-09-09T22:39:36.308 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.309 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2lNiMOHmlz1UkCF08R6TGDALPpc= 2022-09-09T22:39:36.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2lNiMOHmlz1UkCF08R6TGDALPpc=', 'Content-Length': '0'} 2022-09-09T22:39:36.313 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e31a0b5bf970e3c-00631bc0a8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.314 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:36.314 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:36.315 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.315 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.316 DEBUG:boto:Method: GET 2022-09-09T22:39:36.317 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.317 DEBUG:boto:Data: 2022-09-09T22:39:36.318 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.318 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:36.319 DEBUG:boto:Port: 8000 2022-09-09T22:39:36.319 DEBUG:boto:Params: {} 2022-09-09T22:39:36.320 DEBUG:boto:Token: None 2022-09-09T22:39:36.320 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.321 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2lNiMOHmlz1UkCF08R6TGDALPpc= 2022-09-09T22:39:36.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2lNiMOHmlz1UkCF08R6TGDALPpc=', 'Content-Length': '0'} 2022-09-09T22:39:36.325 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005311a8ca8c7bc5d6-00631bc0a8-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.326 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:36.326 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:36.327 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:36.327 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:36.328 DEBUG:boto:Method: GET 2022-09-09T22:39:36.328 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:36.329 DEBUG:boto:Data: 2022-09-09T22:39:36.329 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.330 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:36.331 DEBUG:boto:Port: 8001 2022-09-09T22:39:36.331 DEBUG:boto:Params: {} 2022-09-09T22:39:36.332 DEBUG:boto:Token: None 2022-09-09T22:39:36.332 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/ 2022-09-09T22:39:36.333 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pKHhQr+LLYS+2Qua0mRCkmakpJQ= 2022-09-09T22:39:36.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pKHhQr+LLYS+2Qua0mRCkmakpJQ=', 'Content-Length': '0'} 2022-09-09T22:39:36.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007503a030ae07216f-00631bc0a8-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.339 DEBUG:boto:b'jkcgbj-18obj-test-zone31000falseobj-test-zone32022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:36.340 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:36.341 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:36.341 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:36.342 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:36.342 DEBUG:boto:Method: GET 2022-09-09T22:39:36.343 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:36.343 DEBUG:boto:Data: 2022-09-09T22:39:36.344 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.344 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:36.345 DEBUG:boto:Port: 8000 2022-09-09T22:39:36.346 DEBUG:boto:Params: {} 2022-09-09T22:39:36.346 DEBUG:boto:Token: None 2022-09-09T22:39:36.347 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/ 2022-09-09T22:39:36.347 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pKHhQr+LLYS+2Qua0mRCkmakpJQ= 2022-09-09T22:39:36.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pKHhQr+LLYS+2Qua0mRCkmakpJQ=', 'Content-Length': '0'} 2022-09-09T22:39:36.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c522ee645c196f9a-00631bc0a8-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.354 DEBUG:boto:b'jkcgbj-18obj-test-zone31000falseobj-test-zone32022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:36.355 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:36.355 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:36.356 DEBUG:boto:Method: HEAD 2022-09-09T22:39:36.356 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3 2022-09-09T22:39:36.357 DEBUG:boto:Data: 2022-09-09T22:39:36.357 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.358 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:36.358 DEBUG:boto:Port: 8001 2022-09-09T22:39:36.359 DEBUG:boto:Params: {} 2022-09-09T22:39:36.359 DEBUG:boto:Token: None 2022-09-09T22:39:36.360 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/obj-test-zone3 2022-09-09T22:39:36.361 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ZMgSv/lBf1bFZr591zU72XsGJjU= 2022-09-09T22:39:36.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ZMgSv/lBf1bFZr591zU72XsGJjU=', 'Content-Length': '0'} 2022-09-09T22:39:36.365 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f7091e9913708ab-00631bc0a8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.366 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:36.367 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:36.367 DEBUG:boto:Method: HEAD 2022-09-09T22:39:36.368 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3 2022-09-09T22:39:36.368 DEBUG:boto:Data: 2022-09-09T22:39:36.369 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.369 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:36.370 DEBUG:boto:Port: 8000 2022-09-09T22:39:36.370 DEBUG:boto:Params: {} 2022-09-09T22:39:36.371 DEBUG:boto:Token: None 2022-09-09T22:39:36.371 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/obj-test-zone3 2022-09-09T22:39:36.372 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ZMgSv/lBf1bFZr591zU72XsGJjU= 2022-09-09T22:39:36.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ZMgSv/lBf1bFZr591zU72XsGJjU=', 'Content-Length': '0'} 2022-09-09T22:39:36.376 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000024acb38379c8d76d-00631bc0a8-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:39:36.377 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:36.378 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:36.379 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.379 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.380 DEBUG:boto:Method: GET 2022-09-09T22:39:36.380 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.381 DEBUG:boto:Data: 2022-09-09T22:39:36.381 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.382 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:36.382 DEBUG:boto:Port: 8001 2022-09-09T22:39:36.383 DEBUG:boto:Params: {} 2022-09-09T22:39:36.384 DEBUG:boto:Token: None 2022-09-09T22:39:36.384 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.385 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2lNiMOHmlz1UkCF08R6TGDALPpc= 2022-09-09T22:39:36.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2lNiMOHmlz1UkCF08R6TGDALPpc=', 'Content-Length': '0'} 2022-09-09T22:39:36.388 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090a857acf5893eee-00631bc0a8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.430 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:36.431 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:36.432 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.432 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.433 DEBUG:boto:Method: GET 2022-09-09T22:39:36.434 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.434 DEBUG:boto:Data: 2022-09-09T22:39:36.435 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.435 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:36.436 DEBUG:boto:Port: 8000 2022-09-09T22:39:36.436 DEBUG:boto:Params: {} 2022-09-09T22:39:36.437 DEBUG:boto:Token: None 2022-09-09T22:39:36.437 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:36.438 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2lNiMOHmlz1UkCF08R6TGDALPpc= 2022-09-09T22:39:36.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2lNiMOHmlz1UkCF08R6TGDALPpc=', 'Content-Length': '0'} 2022-09-09T22:39:36.442 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009499ba4a1ec61b5b-00631bc0a8-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.443 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:36.444 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:36.444 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:36.445 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:36.446 DEBUG:boto:Method: GET 2022-09-09T22:39:36.446 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:36.447 DEBUG:boto:Data: 2022-09-09T22:39:36.447 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.448 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:36.448 DEBUG:boto:Port: 8001 2022-09-09T22:39:36.449 DEBUG:boto:Params: {} 2022-09-09T22:39:36.449 DEBUG:boto:Token: None 2022-09-09T22:39:36.450 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/ 2022-09-09T22:39:36.450 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pKHhQr+LLYS+2Qua0mRCkmakpJQ= 2022-09-09T22:39:36.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pKHhQr+LLYS+2Qua0mRCkmakpJQ=', 'Content-Length': '0'} 2022-09-09T22:39:36.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b113d76cd6fbcf55-00631bc0a8-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.457 DEBUG:boto:b'jkcgbj-18obj-test-zone31000falseobj-test-zone32022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:36.458 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:36.459 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:36.459 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:36.460 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:36.460 DEBUG:boto:Method: GET 2022-09-09T22:39:36.461 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:36.461 DEBUG:boto:Data: 2022-09-09T22:39:36.462 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.462 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:36.463 DEBUG:boto:Port: 8000 2022-09-09T22:39:36.463 DEBUG:boto:Params: {} 2022-09-09T22:39:36.464 DEBUG:boto:Token: None 2022-09-09T22:39:36.464 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/ 2022-09-09T22:39:36.465 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pKHhQr+LLYS+2Qua0mRCkmakpJQ= 2022-09-09T22:39:36.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pKHhQr+LLYS+2Qua0mRCkmakpJQ=', 'Content-Length': '0'} 2022-09-09T22:39:36.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aee7448966eeecd7-00631bc0a8-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.518 DEBUG:boto:b'jkcgbj-18obj-test-zone31000falseobj-test-zone32022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:36.519 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-18 zones={test-zone3, test-zone1} 2022-09-09T22:39:36.519 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-18 zones={test-zone3, test-zone2} 2022-09-09T22:39:36.520 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:36.520 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:36.521 DEBUG:boto:Method: HEAD 2022-09-09T22:39:36.521 DEBUG:boto:Path: /jkcgbj-18/ 2022-09-09T22:39:36.522 DEBUG:boto:Data: 2022-09-09T22:39:36.522 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.523 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:36.523 DEBUG:boto:Port: 8001 2022-09-09T22:39:36.524 DEBUG:boto:Params: {} 2022-09-09T22:39:36.524 DEBUG:boto:Token: None 2022-09-09T22:39:36.525 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/ 2022-09-09T22:39:36.526 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8Q4/48n/LgHEjKr4LcF7ruiyy5s= 2022-09-09T22:39:36.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8Q4/48n/LgHEjKr4LcF7ruiyy5s=', 'Content-Length': '0'} 2022-09-09T22:39:36.531 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e9509bbfb115b8b-00631bc0a8-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.532 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:36.532 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:36.533 DEBUG:boto:Method: HEAD 2022-09-09T22:39:36.533 DEBUG:boto:Path: /jkcgbj-18/ 2022-09-09T22:39:36.534 DEBUG:boto:Data: 2022-09-09T22:39:36.534 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.535 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:36.535 DEBUG:boto:Port: 8000 2022-09-09T22:39:36.536 DEBUG:boto:Params: {} 2022-09-09T22:39:36.536 DEBUG:boto:Token: None 2022-09-09T22:39:36.537 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/ 2022-09-09T22:39:36.537 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8Q4/48n/LgHEjKr4LcF7ruiyy5s= 2022-09-09T22:39:36.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8Q4/48n/LgHEjKr4LcF7ruiyy5s=', 'Content-Length': '0'} 2022-09-09T22:39:36.542 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b0ff7465e27e178-00631bc0a8-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.543 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:39:36.543 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:36.544 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:36.544 DEBUG:boto:path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:36.545 DEBUG:boto:auth_path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:36.545 DEBUG:boto:Method: GET 2022-09-09T22:39:36.546 DEBUG:boto:Path: /jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:36.547 DEBUG:boto:Data: 2022-09-09T22:39:36.547 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.548 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:36.548 DEBUG:boto:Port: 8001 2022-09-09T22:39:36.549 DEBUG:boto:Params: {} 2022-09-09T22:39:36.549 DEBUG:boto:Token: None 2022-09-09T22:39:36.550 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/?versions 2022-09-09T22:39:36.550 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:yuOn0sRC2K7X/fUUd6C69dNmzTU= 2022-09-09T22:39:36.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:yuOn0sRC2K7X/fUUd6C69dNmzTU=', 'Content-Length': '0'} 2022-09-09T22:39:36.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000219ad315f0c5aa7a-00631bc0a8-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.558 DEBUG:boto:b'jkcgbj-18999falsedummyRQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5true2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1a8lrwM74XPlF1mwXf8hNl9Y9ISdV53ltrue2022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2e.gQ6yFECscD-oc4OV0dmk6aAg9jH.otrue2022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1true2022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:36.559 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:39:36.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:39:36.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:39:36.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:39:36.561 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:39:36.561 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:36.562 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:36.562 DEBUG:boto:path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:36.563 DEBUG:boto:auth_path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:36.563 DEBUG:boto:Method: GET 2022-09-09T22:39:36.564 DEBUG:boto:Path: /jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:36.564 DEBUG:boto:Data: 2022-09-09T22:39:36.565 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.565 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:36.566 DEBUG:boto:Port: 8000 2022-09-09T22:39:36.566 DEBUG:boto:Params: {} 2022-09-09T22:39:36.567 DEBUG:boto:Token: None 2022-09-09T22:39:36.567 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/?versions 2022-09-09T22:39:36.568 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:yuOn0sRC2K7X/fUUd6C69dNmzTU= 2022-09-09T22:39:36.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:yuOn0sRC2K7X/fUUd6C69dNmzTU=', 'Content-Length': '0'} 2022-09-09T22:39:36.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccc8cb5c721e80c8-00631bc0a8-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.574 DEBUG:boto:b'jkcgbj-18999falsedummyRQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5true2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1a8lrwM74XPlF1mwXf8hNl9Y9ISdV53ltrue2022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2e.gQ6yFECscD-oc4OV0dmk6aAg9jH.otrue2022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1true2022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:36.575 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:39:36.576 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-09T22:39:36.576 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-09T22:39:36.576 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-09T22:39:36.577 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:36.577 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:36.578 DEBUG:boto:path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:36.578 DEBUG:boto:auth_path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:36.579 DEBUG:boto:Method: GET 2022-09-09T22:39:36.579 DEBUG:boto:Path: /jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:36.580 DEBUG:boto:Data: 2022-09-09T22:39:36.580 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.581 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:36.581 DEBUG:boto:Port: 8001 2022-09-09T22:39:36.582 DEBUG:boto:Params: {} 2022-09-09T22:39:36.582 DEBUG:boto:Token: None 2022-09-09T22:39:36.583 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/?versions 2022-09-09T22:39:36.584 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:yuOn0sRC2K7X/fUUd6C69dNmzTU= 2022-09-09T22:39:36.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:yuOn0sRC2K7X/fUUd6C69dNmzTU=', 'Content-Length': '0'} 2022-09-09T22:39:36.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ade4f5d66e11ceb-00631bc0a8-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.590 DEBUG:boto:b'jkcgbj-18999falsedummyRQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5true2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1a8lrwM74XPlF1mwXf8hNl9Y9ISdV53ltrue2022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2e.gQ6yFECscD-oc4OV0dmk6aAg9jH.otrue2022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1true2022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:36.591 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:36.592 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:36.592 DEBUG:boto:path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:36.593 DEBUG:boto:auth_path=/jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:36.593 DEBUG:boto:Method: GET 2022-09-09T22:39:36.594 DEBUG:boto:Path: /jkcgbj-18/?versions&max-keys=999 2022-09-09T22:39:36.594 DEBUG:boto:Data: 2022-09-09T22:39:36.595 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.595 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:36.596 DEBUG:boto:Port: 8000 2022-09-09T22:39:36.596 DEBUG:boto:Params: {} 2022-09-09T22:39:36.597 DEBUG:boto:Token: None 2022-09-09T22:39:36.597 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/?versions 2022-09-09T22:39:36.598 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:yuOn0sRC2K7X/fUUd6C69dNmzTU= 2022-09-09T22:39:36.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:yuOn0sRC2K7X/fUUd6C69dNmzTU=', 'Content-Length': '0'} 2022-09-09T22:39:36.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004593d20841361c46-00631bc0a8-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.604 DEBUG:boto:b'jkcgbj-18999falsedummyRQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5true2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1a8lrwM74XPlF1mwXf8hNl9Y9ISdV53ltrue2022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2e.gQ6yFECscD-oc4OV0dmk6aAg9jH.otrue2022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1true2022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:36.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:39:36.605 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:39:36.606 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:39:36.606 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.607 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.607 DEBUG:boto:Method: GET 2022-09-09T22:39:36.608 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.608 DEBUG:boto:Data: 2022-09-09T22:39:36.609 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.609 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:36.610 DEBUG:boto:Port: 8001 2022-09-09T22:39:36.610 DEBUG:boto:Params: {} 2022-09-09T22:39:36.611 DEBUG:boto:Token: None 2022-09-09T22:39:36.612 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.612 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:XZaKi95nhE/88H7ObcYPJfeIIAg= 2022-09-09T22:39:36.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:XZaKi95nhE/88H7ObcYPJfeIIAg=', 'Content-Length': '0'} 2022-09-09T22:39:36.616 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008d37879dbc75286b-00631bc0a8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.617 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:39:36.617 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:39:36.618 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.618 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.619 DEBUG:boto:Method: GET 2022-09-09T22:39:36.619 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.619 DEBUG:boto:Data: 2022-09-09T22:39:36.620 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.620 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:36.621 DEBUG:boto:Port: 8000 2022-09-09T22:39:36.621 DEBUG:boto:Params: {} 2022-09-09T22:39:36.622 DEBUG:boto:Token: None 2022-09-09T22:39:36.623 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.623 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:XZaKi95nhE/88H7ObcYPJfeIIAg= 2022-09-09T22:39:36.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:XZaKi95nhE/88H7ObcYPJfeIIAg=', 'Content-Length': '0'} 2022-09-09T22:39:36.627 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b2f6c0c2fffa4620-00631bc0a8-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.627 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:39:36.628 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:39:36.628 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.629 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.629 DEBUG:boto:Method: HEAD 2022-09-09T22:39:36.630 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.630 DEBUG:boto:Data: 2022-09-09T22:39:36.631 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.631 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:36.632 DEBUG:boto:Port: 8001 2022-09-09T22:39:36.632 DEBUG:boto:Params: {} 2022-09-09T22:39:36.633 DEBUG:boto:Token: None 2022-09-09T22:39:36.633 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.634 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:lm9QWnjfBh5NxBMVhUeInMAn2hU= 2022-09-09T22:39:36.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:lm9QWnjfBh5NxBMVhUeInMAn2hU=', 'Content-Length': '0'} 2022-09-09T22:39:36.638 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004398919b9c9df561-00631bc0a8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.638 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:39:36.639 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:39:36.639 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.640 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.640 DEBUG:boto:Method: HEAD 2022-09-09T22:39:36.641 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.641 DEBUG:boto:Data: 2022-09-09T22:39:36.642 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.642 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:36.643 DEBUG:boto:Port: 8000 2022-09-09T22:39:36.643 DEBUG:boto:Params: {} 2022-09-09T22:39:36.644 DEBUG:boto:Token: None 2022-09-09T22:39:36.644 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.645 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:lm9QWnjfBh5NxBMVhUeInMAn2hU= 2022-09-09T22:39:36.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:lm9QWnjfBh5NxBMVhUeInMAn2hU=', 'Content-Length': '0'} 2022-09-09T22:39:36.649 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b500db27426756c5-00631bc0a8-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:39:36.650 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:39:36.650 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:39:36.651 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.651 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.652 DEBUG:boto:Method: GET 2022-09-09T22:39:36.652 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.653 DEBUG:boto:Data: 2022-09-09T22:39:36.653 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.654 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:36.655 DEBUG:boto:Port: 8001 2022-09-09T22:39:36.655 DEBUG:boto:Params: {} 2022-09-09T22:39:36.656 DEBUG:boto:Token: None 2022-09-09T22:39:36.656 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.657 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:XZaKi95nhE/88H7ObcYPJfeIIAg= 2022-09-09T22:39:36.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:XZaKi95nhE/88H7ObcYPJfeIIAg=', 'Content-Length': '0'} 2022-09-09T22:39:36.660 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f3eb0ffb4b241d88-00631bc0a8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.661 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:39:36.661 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:39:36.662 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.662 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.663 DEBUG:boto:Method: GET 2022-09-09T22:39:36.663 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.664 DEBUG:boto:Data: 2022-09-09T22:39:36.664 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.665 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:36.665 DEBUG:boto:Port: 8000 2022-09-09T22:39:36.666 DEBUG:boto:Params: {} 2022-09-09T22:39:36.667 DEBUG:boto:Token: None 2022-09-09T22:39:36.667 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.668 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:XZaKi95nhE/88H7ObcYPJfeIIAg= 2022-09-09T22:39:36.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:XZaKi95nhE/88H7ObcYPJfeIIAg=', 'Content-Length': '0'} 2022-09-09T22:39:36.671 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000203017c7ea129a7a-00631bc0a8-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.672 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:36.673 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:36.673 DEBUG:boto:path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:39:36.674 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:39:36.674 DEBUG:boto:Method: GET 2022-09-09T22:39:36.675 DEBUG:boto:Path: /jkcgbj-18/?prefix=dummy 2022-09-09T22:39:36.675 DEBUG:boto:Data: 2022-09-09T22:39:36.676 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.676 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:36.677 DEBUG:boto:Port: 8001 2022-09-09T22:39:36.677 DEBUG:boto:Params: {} 2022-09-09T22:39:36.678 DEBUG:boto:Token: None 2022-09-09T22:39:36.678 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/ 2022-09-09T22:39:36.679 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pKHhQr+LLYS+2Qua0mRCkmakpJQ= 2022-09-09T22:39:36.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pKHhQr+LLYS+2Qua0mRCkmakpJQ=', 'Content-Length': '0'} 2022-09-09T22:39:36.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047e4a76749c3691c-00631bc0a8-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.726 DEBUG:boto:b'jkcgbj-18dummy1000falsedummy2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:36.727 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:36.727 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:36.728 DEBUG:boto:path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:39:36.728 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:39:36.729 DEBUG:boto:Method: GET 2022-09-09T22:39:36.730 DEBUG:boto:Path: /jkcgbj-18/?prefix=dummy 2022-09-09T22:39:36.730 DEBUG:boto:Data: 2022-09-09T22:39:36.731 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.731 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:36.732 DEBUG:boto:Port: 8000 2022-09-09T22:39:36.732 DEBUG:boto:Params: {} 2022-09-09T22:39:36.733 DEBUG:boto:Token: None 2022-09-09T22:39:36.733 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/ 2022-09-09T22:39:36.734 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pKHhQr+LLYS+2Qua0mRCkmakpJQ= 2022-09-09T22:39:36.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pKHhQr+LLYS+2Qua0mRCkmakpJQ=', 'Content-Length': '0'} 2022-09-09T22:39:36.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b47033cd92f3e82c-00631bc0a8-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.740 DEBUG:boto:b'jkcgbj-18dummy1000falsedummy2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:36.741 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:39:36.742 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:39:36.742 DEBUG:boto:Method: HEAD 2022-09-09T22:39:36.743 DEBUG:boto:Path: /jkcgbj-18/dummy 2022-09-09T22:39:36.743 DEBUG:boto:Data: 2022-09-09T22:39:36.744 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.744 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:36.745 DEBUG:boto:Port: 8001 2022-09-09T22:39:36.745 DEBUG:boto:Params: {} 2022-09-09T22:39:36.746 DEBUG:boto:Token: None 2022-09-09T22:39:36.746 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/dummy 2022-09-09T22:39:36.747 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ev+T9yGQfAt+VvTbVtZ4Wr/n7to= 2022-09-09T22:39:36.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ev+T9yGQfAt+VvTbVtZ4Wr/n7to=', 'Content-Length': '0'} 2022-09-09T22:39:36.751 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000056cc09627fc97d99-00631bc0a8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.752 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:39:36.752 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:39:36.753 DEBUG:boto:Method: HEAD 2022-09-09T22:39:36.753 DEBUG:boto:Path: /jkcgbj-18/dummy 2022-09-09T22:39:36.754 DEBUG:boto:Data: 2022-09-09T22:39:36.754 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.755 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:36.755 DEBUG:boto:Port: 8000 2022-09-09T22:39:36.756 DEBUG:boto:Params: {} 2022-09-09T22:39:36.756 DEBUG:boto:Token: None 2022-09-09T22:39:36.757 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/dummy 2022-09-09T22:39:36.757 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ev+T9yGQfAt+VvTbVtZ4Wr/n7to= 2022-09-09T22:39:36.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ev+T9yGQfAt+VvTbVtZ4Wr/n7to=', 'Content-Length': '0'} 2022-09-09T22:39:36.761 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c606b427ac23a587-00631bc0a8-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:39:36.763 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:39:36.763 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:39:36.764 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.764 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.765 DEBUG:boto:Method: GET 2022-09-09T22:39:36.765 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.766 DEBUG:boto:Data: 2022-09-09T22:39:36.766 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.767 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:36.768 DEBUG:boto:Port: 8001 2022-09-09T22:39:36.768 DEBUG:boto:Params: {} 2022-09-09T22:39:36.769 DEBUG:boto:Token: None 2022-09-09T22:39:36.769 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.770 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:XZaKi95nhE/88H7ObcYPJfeIIAg= 2022-09-09T22:39:36.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:XZaKi95nhE/88H7ObcYPJfeIIAg=', 'Content-Length': '0'} 2022-09-09T22:39:36.773 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d675adce35f855a9-00631bc0a8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.774 DEBUG:boto:path=/jkcgbj-18/dummy 2022-09-09T22:39:36.775 DEBUG:boto:auth_path=/jkcgbj-18/dummy 2022-09-09T22:39:36.775 DEBUG:boto:path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.776 DEBUG:boto:auth_path=/jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.776 DEBUG:boto:Method: GET 2022-09-09T22:39:36.777 DEBUG:boto:Path: /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.777 DEBUG:boto:Data: 2022-09-09T22:39:36.778 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.778 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:36.779 DEBUG:boto:Port: 8000 2022-09-09T22:39:36.779 DEBUG:boto:Params: {} 2022-09-09T22:39:36.780 DEBUG:boto:Token: None 2022-09-09T22:39:36.781 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/dummy?versionId=RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5 2022-09-09T22:39:36.781 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:XZaKi95nhE/88H7ObcYPJfeIIAg= 2022-09-09T22:39:36.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:XZaKi95nhE/88H7ObcYPJfeIIAg=', 'Content-Length': '0'} 2022-09-09T22:39:36.785 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:35:20 GMT'), ('x-amz-version-id', 'RQpY1vVV0X8GE2GK3f2Yt7JTttfZGB5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c4d9e70c683b47df-00631bc0a8-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.786 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:36.786 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:36.787 DEBUG:boto:path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:39:36.787 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:39:36.788 DEBUG:boto:Method: GET 2022-09-09T22:39:36.788 DEBUG:boto:Path: /jkcgbj-18/?prefix=dummy 2022-09-09T22:39:36.789 DEBUG:boto:Data: 2022-09-09T22:39:36.790 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.790 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:36.791 DEBUG:boto:Port: 8001 2022-09-09T22:39:36.791 DEBUG:boto:Params: {} 2022-09-09T22:39:36.792 DEBUG:boto:Token: None 2022-09-09T22:39:36.792 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/ 2022-09-09T22:39:36.793 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pKHhQr+LLYS+2Qua0mRCkmakpJQ= 2022-09-09T22:39:36.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pKHhQr+LLYS+2Qua0mRCkmakpJQ=', 'Content-Length': '0'} 2022-09-09T22:39:36.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000569b7052936a0ec7-00631bc0a8-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.800 DEBUG:boto:b'jkcgbj-18dummy1000falsedummy2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:36.800 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:36.801 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:36.801 DEBUG:boto:path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:39:36.802 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=dummy 2022-09-09T22:39:36.803 DEBUG:boto:Method: GET 2022-09-09T22:39:36.803 DEBUG:boto:Path: /jkcgbj-18/?prefix=dummy 2022-09-09T22:39:36.804 DEBUG:boto:Data: 2022-09-09T22:39:36.804 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.805 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:36.805 DEBUG:boto:Port: 8000 2022-09-09T22:39:36.806 DEBUG:boto:Params: {} 2022-09-09T22:39:36.806 DEBUG:boto:Token: None 2022-09-09T22:39:36.807 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/ 2022-09-09T22:39:36.807 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pKHhQr+LLYS+2Qua0mRCkmakpJQ= 2022-09-09T22:39:36.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pKHhQr+LLYS+2Qua0mRCkmakpJQ=', 'Content-Length': '0'} 2022-09-09T22:39:36.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a6d08cfcf080d6a-00631bc0a8-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.814 DEBUG:boto:b'jkcgbj-18dummy1000falsedummy2022-09-09T22:35:20.934Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:36.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:39:36.815 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:36.816 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:36.816 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:36.817 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:36.817 DEBUG:boto:Method: GET 2022-09-09T22:39:36.818 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:36.819 DEBUG:boto:Data: 2022-09-09T22:39:36.819 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.820 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:36.820 DEBUG:boto:Port: 8001 2022-09-09T22:39:36.821 DEBUG:boto:Params: {} 2022-09-09T22:39:36.821 DEBUG:boto:Token: None 2022-09-09T22:39:36.822 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:36.823 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:iMCDpF9vDGa6pjfFtHo0yI19xDs= 2022-09-09T22:39:36.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:iMCDpF9vDGa6pjfFtHo0yI19xDs=', 'Content-Length': '0'} 2022-09-09T22:39:36.826 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000035a3cc3a52de019-00631bc0a8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.870 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:36.871 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:36.871 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:36.872 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:36.872 DEBUG:boto:Method: GET 2022-09-09T22:39:36.873 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:36.873 DEBUG:boto:Data: 2022-09-09T22:39:36.874 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.874 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:36.875 DEBUG:boto:Port: 8000 2022-09-09T22:39:36.876 DEBUG:boto:Params: {} 2022-09-09T22:39:36.876 DEBUG:boto:Token: None 2022-09-09T22:39:36.877 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:36.877 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:iMCDpF9vDGa6pjfFtHo0yI19xDs= 2022-09-09T22:39:36.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:iMCDpF9vDGa6pjfFtHo0yI19xDs=', 'Content-Length': '0'} 2022-09-09T22:39:36.881 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf4e1ece465b649c-00631bc0a8-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.882 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:36.883 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:36.884 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:36.885 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:36.886 DEBUG:boto:Method: HEAD 2022-09-09T22:39:36.887 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:36.888 DEBUG:boto:Data: 2022-09-09T22:39:36.889 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.889 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:36.890 DEBUG:boto:Port: 8001 2022-09-09T22:39:36.891 DEBUG:boto:Params: {} 2022-09-09T22:39:36.892 DEBUG:boto:Token: None 2022-09-09T22:39:36.893 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:36.894 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ULNEVSVD3hyNBC0jj4rn5N1C5aE= 2022-09-09T22:39:36.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ULNEVSVD3hyNBC0jj4rn5N1C5aE=', 'Content-Length': '0'} 2022-09-09T22:39:36.898 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c64ddec52ba97d3-00631bc0a8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.898 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:36.899 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:36.899 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:36.900 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:36.900 DEBUG:boto:Method: HEAD 2022-09-09T22:39:36.901 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:36.901 DEBUG:boto:Data: 2022-09-09T22:39:36.902 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.902 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:36.903 DEBUG:boto:Port: 8000 2022-09-09T22:39:36.904 DEBUG:boto:Params: {} 2022-09-09T22:39:36.905 DEBUG:boto:Token: None 2022-09-09T22:39:36.906 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:36.907 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ULNEVSVD3hyNBC0jj4rn5N1C5aE= 2022-09-09T22:39:36.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ULNEVSVD3hyNBC0jj4rn5N1C5aE=', 'Content-Length': '0'} 2022-09-09T22:39:36.911 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b699e52988cab2c-00631bc0a8-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:39:36.913 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:36.914 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:36.915 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:36.916 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:36.917 DEBUG:boto:Method: GET 2022-09-09T22:39:36.918 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:36.919 DEBUG:boto:Data: 2022-09-09T22:39:36.920 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.921 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:36.922 DEBUG:boto:Port: 8001 2022-09-09T22:39:36.923 DEBUG:boto:Params: {} 2022-09-09T22:39:36.924 DEBUG:boto:Token: None 2022-09-09T22:39:36.925 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:36.926 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:iMCDpF9vDGa6pjfFtHo0yI19xDs= 2022-09-09T22:39:36.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:iMCDpF9vDGa6pjfFtHo0yI19xDs=', 'Content-Length': '0'} 2022-09-09T22:39:36.931 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f31c7c09d9bf2d9-00631bc0a8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.932 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:36.933 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:36.934 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:36.934 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:36.935 DEBUG:boto:Method: GET 2022-09-09T22:39:36.935 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:36.936 DEBUG:boto:Data: 2022-09-09T22:39:36.936 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.937 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:36.937 DEBUG:boto:Port: 8000 2022-09-09T22:39:36.938 DEBUG:boto:Params: {} 2022-09-09T22:39:36.939 DEBUG:boto:Token: None 2022-09-09T22:39:36.939 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:36.940 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:iMCDpF9vDGa6pjfFtHo0yI19xDs= 2022-09-09T22:39:36.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:iMCDpF9vDGa6pjfFtHo0yI19xDs=', 'Content-Length': '0'} 2022-09-09T22:39:36.943 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ada7b29502f9cc1-00631bc0a8-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.944 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:36.945 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:36.946 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:36.946 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:36.947 DEBUG:boto:Method: GET 2022-09-09T22:39:36.947 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:36.948 DEBUG:boto:Data: 2022-09-09T22:39:36.948 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.949 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:36.949 DEBUG:boto:Port: 8001 2022-09-09T22:39:36.950 DEBUG:boto:Params: {} 2022-09-09T22:39:36.950 DEBUG:boto:Token: None 2022-09-09T22:39:36.951 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/ 2022-09-09T22:39:36.952 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pKHhQr+LLYS+2Qua0mRCkmakpJQ= 2022-09-09T22:39:36.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pKHhQr+LLYS+2Qua0mRCkmakpJQ=', 'Content-Length': '0'} 2022-09-09T22:39:36.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020a020815eec0d0c-00631bc0a8-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:36.959 DEBUG:boto:b'jkcgbj-18obj-test-zone11000falseobj-test-zone12022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:36.959 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:36.960 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:36.960 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:36.961 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:36.961 DEBUG:boto:Method: GET 2022-09-09T22:39:36.962 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:36.963 DEBUG:boto:Data: 2022-09-09T22:39:36.963 DEBUG:boto:Headers: {} 2022-09-09T22:39:36.964 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:36.964 DEBUG:boto:Port: 8000 2022-09-09T22:39:36.965 DEBUG:boto:Params: {} 2022-09-09T22:39:36.965 DEBUG:boto:Token: None 2022-09-09T22:39:36.966 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:36 GMT /jkcgbj-18/ 2022-09-09T22:39:36.966 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pKHhQr+LLYS+2Qua0mRCkmakpJQ= 2022-09-09T22:39:36.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:36 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pKHhQr+LLYS+2Qua0mRCkmakpJQ=', 'Content-Length': '0'} 2022-09-09T22:39:36.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060715afa6253e56e-00631bc0a8-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.014 DEBUG:boto:b'jkcgbj-18obj-test-zone11000falseobj-test-zone12022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:37.015 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:37.016 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:37.016 DEBUG:boto:Method: HEAD 2022-09-09T22:39:37.017 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1 2022-09-09T22:39:37.017 DEBUG:boto:Data: 2022-09-09T22:39:37.018 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.018 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:37.019 DEBUG:boto:Port: 8001 2022-09-09T22:39:37.019 DEBUG:boto:Params: {} 2022-09-09T22:39:37.020 DEBUG:boto:Token: None 2022-09-09T22:39:37.021 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/obj-test-zone1 2022-09-09T22:39:37.021 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:upTvRuLVIGtdjtw2lWhjdqHrS5g= 2022-09-09T22:39:37.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:upTvRuLVIGtdjtw2lWhjdqHrS5g=', 'Content-Length': '0'} 2022-09-09T22:39:37.026 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000cf0febe90056d7d-00631bc0a9-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.026 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:37.027 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:37.028 DEBUG:boto:Method: HEAD 2022-09-09T22:39:37.028 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1 2022-09-09T22:39:37.029 DEBUG:boto:Data: 2022-09-09T22:39:37.029 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.030 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:37.031 DEBUG:boto:Port: 8000 2022-09-09T22:39:37.031 DEBUG:boto:Params: {} 2022-09-09T22:39:37.032 DEBUG:boto:Token: None 2022-09-09T22:39:37.032 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/obj-test-zone1 2022-09-09T22:39:37.033 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:upTvRuLVIGtdjtw2lWhjdqHrS5g= 2022-09-09T22:39:37.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:upTvRuLVIGtdjtw2lWhjdqHrS5g=', 'Content-Length': '0'} 2022-09-09T22:39:37.037 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009139bc3c06057dd9-00631bc0a9-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-09T22:39:37.039 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:37.039 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:37.040 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:37.040 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:37.041 DEBUG:boto:Method: GET 2022-09-09T22:39:37.041 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:37.042 DEBUG:boto:Data: 2022-09-09T22:39:37.042 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.043 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:37.043 DEBUG:boto:Port: 8001 2022-09-09T22:39:37.044 DEBUG:boto:Params: {} 2022-09-09T22:39:37.045 DEBUG:boto:Token: None 2022-09-09T22:39:37.045 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:37.046 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:RWS+IkFFDsDpDhU6r8rl+02bUmE= 2022-09-09T22:39:37.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:RWS+IkFFDsDpDhU6r8rl+02bUmE=', 'Content-Length': '0'} 2022-09-09T22:39:37.050 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f6c53ee8445ff28-00631bc0a9-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.051 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:37.051 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1 2022-09-09T22:39:37.052 DEBUG:boto:path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:37.052 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:37.053 DEBUG:boto:Method: GET 2022-09-09T22:39:37.053 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:37.054 DEBUG:boto:Data: 2022-09-09T22:39:37.055 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.055 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:37.056 DEBUG:boto:Port: 8000 2022-09-09T22:39:37.056 DEBUG:boto:Params: {} 2022-09-09T22:39:37.057 DEBUG:boto:Token: None 2022-09-09T22:39:37.058 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/obj-test-zone1?versionId=a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l 2022-09-09T22:39:37.058 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:RWS+IkFFDsDpDhU6r8rl+02bUmE= 2022-09-09T22:39:37.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:RWS+IkFFDsDpDhU6r8rl+02bUmE=', 'Content-Length': '0'} 2022-09-09T22:39:37.062 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'a8lrwM74XPlF1mwXf8hNl9Y9ISdV53l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041a41da8749ed47c-00631bc0a9-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.063 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:37.063 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:37.064 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:37.065 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:37.065 DEBUG:boto:Method: GET 2022-09-09T22:39:37.066 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:37.066 DEBUG:boto:Data: 2022-09-09T22:39:37.067 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.067 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:37.068 DEBUG:boto:Port: 8001 2022-09-09T22:39:37.068 DEBUG:boto:Params: {} 2022-09-09T22:39:37.069 DEBUG:boto:Token: None 2022-09-09T22:39:37.070 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/ 2022-09-09T22:39:37.071 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sWd7mdST61RlwNhY8z36z8oGtP8= 2022-09-09T22:39:37.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sWd7mdST61RlwNhY8z36z8oGtP8=', 'Content-Length': '0'} 2022-09-09T22:39:37.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebc42add08a73eda-00631bc0a9-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.077 DEBUG:boto:b'jkcgbj-18obj-test-zone11000falseobj-test-zone12022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:37.078 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:37.079 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:37.079 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:37.080 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:37.080 DEBUG:boto:Method: GET 2022-09-09T22:39:37.081 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone1 2022-09-09T22:39:37.081 DEBUG:boto:Data: 2022-09-09T22:39:37.082 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.082 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:37.083 DEBUG:boto:Port: 8000 2022-09-09T22:39:37.083 DEBUG:boto:Params: {} 2022-09-09T22:39:37.084 DEBUG:boto:Token: None 2022-09-09T22:39:37.085 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/ 2022-09-09T22:39:37.085 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sWd7mdST61RlwNhY8z36z8oGtP8= 2022-09-09T22:39:37.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sWd7mdST61RlwNhY8z36z8oGtP8=', 'Content-Length': '0'} 2022-09-09T22:39:37.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000544cbad2a1c91569-00631bc0a9-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.092 DEBUG:boto:b'jkcgbj-18obj-test-zone11000falseobj-test-zone12022-09-09T22:36:02.601Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:37.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:39:37.093 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:37.093 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:37.094 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.095 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.095 DEBUG:boto:Method: GET 2022-09-09T22:39:37.096 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.096 DEBUG:boto:Data: 2022-09-09T22:39:37.097 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.097 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:37.098 DEBUG:boto:Port: 8001 2022-09-09T22:39:37.098 DEBUG:boto:Params: {} 2022-09-09T22:39:37.099 DEBUG:boto:Token: None 2022-09-09T22:39:37.100 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.100 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:PlkhgESvanMsxhxZywxbN4SIo8w= 2022-09-09T22:39:37.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:PlkhgESvanMsxhxZywxbN4SIo8w=', 'Content-Length': '0'} 2022-09-09T22:39:37.104 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dfce4dc1436ff47d-00631bc0a9-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.146 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:37.147 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:37.147 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.148 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.148 DEBUG:boto:Method: GET 2022-09-09T22:39:37.149 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.149 DEBUG:boto:Data: 2022-09-09T22:39:37.150 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.150 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:37.151 DEBUG:boto:Port: 8000 2022-09-09T22:39:37.151 DEBUG:boto:Params: {} 2022-09-09T22:39:37.152 DEBUG:boto:Token: None 2022-09-09T22:39:37.153 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.153 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:PlkhgESvanMsxhxZywxbN4SIo8w= 2022-09-09T22:39:37.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:PlkhgESvanMsxhxZywxbN4SIo8w=', 'Content-Length': '0'} 2022-09-09T22:39:37.157 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d37ac736f5960b1-00631bc0a9-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.158 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:37.158 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:37.159 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.159 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.160 DEBUG:boto:Method: HEAD 2022-09-09T22:39:37.160 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.161 DEBUG:boto:Data: 2022-09-09T22:39:37.161 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.162 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:37.163 DEBUG:boto:Port: 8001 2022-09-09T22:39:37.163 DEBUG:boto:Params: {} 2022-09-09T22:39:37.164 DEBUG:boto:Token: None 2022-09-09T22:39:37.164 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.165 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:h5ZqBOnhAyn3UQgCBIYDVJ0kAEU= 2022-09-09T22:39:37.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:h5ZqBOnhAyn3UQgCBIYDVJ0kAEU=', 'Content-Length': '0'} 2022-09-09T22:39:37.168 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e0c0c8659e499683-00631bc0a9-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.169 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:37.170 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:37.170 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.171 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.172 DEBUG:boto:Method: HEAD 2022-09-09T22:39:37.172 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.173 DEBUG:boto:Data: 2022-09-09T22:39:37.173 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.174 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:37.174 DEBUG:boto:Port: 8000 2022-09-09T22:39:37.175 DEBUG:boto:Params: {} 2022-09-09T22:39:37.175 DEBUG:boto:Token: None 2022-09-09T22:39:37.176 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.176 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:h5ZqBOnhAyn3UQgCBIYDVJ0kAEU= 2022-09-09T22:39:37.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:h5ZqBOnhAyn3UQgCBIYDVJ0kAEU=', 'Content-Length': '0'} 2022-09-09T22:39:37.180 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000039528b4ccf25695a-00631bc0a9-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:39:37.181 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:37.181 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:37.182 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.183 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.183 DEBUG:boto:Method: GET 2022-09-09T22:39:37.184 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.184 DEBUG:boto:Data: 2022-09-09T22:39:37.185 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.185 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:37.186 DEBUG:boto:Port: 8001 2022-09-09T22:39:37.186 DEBUG:boto:Params: {} 2022-09-09T22:39:37.187 DEBUG:boto:Token: None 2022-09-09T22:39:37.188 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.189 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:PlkhgESvanMsxhxZywxbN4SIo8w= 2022-09-09T22:39:37.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:PlkhgESvanMsxhxZywxbN4SIo8w=', 'Content-Length': '0'} 2022-09-09T22:39:37.194 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f15fc258c6258273-00631bc0a9-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.195 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:37.196 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:37.196 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.197 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.197 DEBUG:boto:Method: GET 2022-09-09T22:39:37.198 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.199 DEBUG:boto:Data: 2022-09-09T22:39:37.199 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.200 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:37.200 DEBUG:boto:Port: 8000 2022-09-09T22:39:37.200 DEBUG:boto:Params: {} 2022-09-09T22:39:37.201 DEBUG:boto:Token: None 2022-09-09T22:39:37.202 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.203 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:PlkhgESvanMsxhxZywxbN4SIo8w= 2022-09-09T22:39:37.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:PlkhgESvanMsxhxZywxbN4SIo8w=', 'Content-Length': '0'} 2022-09-09T22:39:37.208 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000abab874ee5d5471d-00631bc0a9-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.250 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:37.251 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:37.252 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:37.252 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:37.253 DEBUG:boto:Method: GET 2022-09-09T22:39:37.253 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:37.254 DEBUG:boto:Data: 2022-09-09T22:39:37.254 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.255 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:37.255 DEBUG:boto:Port: 8001 2022-09-09T22:39:37.256 DEBUG:boto:Params: {} 2022-09-09T22:39:37.257 DEBUG:boto:Token: None 2022-09-09T22:39:37.257 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/ 2022-09-09T22:39:37.258 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sWd7mdST61RlwNhY8z36z8oGtP8= 2022-09-09T22:39:37.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sWd7mdST61RlwNhY8z36z8oGtP8=', 'Content-Length': '0'} 2022-09-09T22:39:37.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000255b4a99039084d7-00631bc0a9-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.265 DEBUG:boto:b'jkcgbj-18obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:37.266 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:37.266 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:37.267 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:37.267 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:37.268 DEBUG:boto:Method: GET 2022-09-09T22:39:37.268 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:37.269 DEBUG:boto:Data: 2022-09-09T22:39:37.269 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.270 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:37.271 DEBUG:boto:Port: 8000 2022-09-09T22:39:37.271 DEBUG:boto:Params: {} 2022-09-09T22:39:37.272 DEBUG:boto:Token: None 2022-09-09T22:39:37.273 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/ 2022-09-09T22:39:37.273 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sWd7mdST61RlwNhY8z36z8oGtP8= 2022-09-09T22:39:37.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sWd7mdST61RlwNhY8z36z8oGtP8=', 'Content-Length': '0'} 2022-09-09T22:39:37.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b67998217801e68b-00631bc0a9-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.280 DEBUG:boto:b'jkcgbj-18obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:37.281 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:37.281 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:37.282 DEBUG:boto:Method: HEAD 2022-09-09T22:39:37.283 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2 2022-09-09T22:39:37.283 DEBUG:boto:Data: 2022-09-09T22:39:37.284 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.284 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:37.285 DEBUG:boto:Port: 8001 2022-09-09T22:39:37.285 DEBUG:boto:Params: {} 2022-09-09T22:39:37.286 DEBUG:boto:Token: None 2022-09-09T22:39:37.287 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/obj-test-zone2 2022-09-09T22:39:37.287 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:tKsoaoR/38EmLwcKGul4pxf+bbU= 2022-09-09T22:39:37.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:tKsoaoR/38EmLwcKGul4pxf+bbU=', 'Content-Length': '0'} 2022-09-09T22:39:37.292 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000932fccf081b2b189-00631bc0a9-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.293 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:37.294 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:37.294 DEBUG:boto:Method: HEAD 2022-09-09T22:39:37.295 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2 2022-09-09T22:39:37.295 DEBUG:boto:Data: 2022-09-09T22:39:37.296 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.296 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:37.297 DEBUG:boto:Port: 8000 2022-09-09T22:39:37.297 DEBUG:boto:Params: {} 2022-09-09T22:39:37.298 DEBUG:boto:Token: None 2022-09-09T22:39:37.298 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/obj-test-zone2 2022-09-09T22:39:37.299 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:tKsoaoR/38EmLwcKGul4pxf+bbU= 2022-09-09T22:39:37.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:tKsoaoR/38EmLwcKGul4pxf+bbU=', 'Content-Length': '0'} 2022-09-09T22:39:37.303 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007756a78d56cb9a0e-00631bc0a9-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-09T22:39:37.305 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:37.305 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:37.306 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.306 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.307 DEBUG:boto:Method: GET 2022-09-09T22:39:37.307 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.308 DEBUG:boto:Data: 2022-09-09T22:39:37.308 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.309 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:37.309 DEBUG:boto:Port: 8001 2022-09-09T22:39:37.310 DEBUG:boto:Params: {} 2022-09-09T22:39:37.311 DEBUG:boto:Token: None 2022-09-09T22:39:37.311 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.312 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:PlkhgESvanMsxhxZywxbN4SIo8w= 2022-09-09T22:39:37.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:PlkhgESvanMsxhxZywxbN4SIo8w=', 'Content-Length': '0'} 2022-09-09T22:39:37.315 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f31a5494bf0c22b-00631bc0a9-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.358 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:37.359 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2 2022-09-09T22:39:37.360 DEBUG:boto:path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.360 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.361 DEBUG:boto:Method: GET 2022-09-09T22:39:37.361 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.362 DEBUG:boto:Data: 2022-09-09T22:39:37.362 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.363 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:37.363 DEBUG:boto:Port: 8000 2022-09-09T22:39:37.364 DEBUG:boto:Params: {} 2022-09-09T22:39:37.365 DEBUG:boto:Token: None 2022-09-09T22:39:37.365 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/obj-test-zone2?versionId=e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o 2022-09-09T22:39:37.366 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:PlkhgESvanMsxhxZywxbN4SIo8w= 2022-09-09T22:39:37.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:PlkhgESvanMsxhxZywxbN4SIo8w=', 'Content-Length': '0'} 2022-09-09T22:39:37.370 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'e.gQ6yFECscD-oc4OV0dmk6aAg9jH.o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ef76dafd0a88f5a-00631bc0a9-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.371 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:37.371 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:37.372 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:37.372 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:37.373 DEBUG:boto:Method: GET 2022-09-09T22:39:37.373 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:37.374 DEBUG:boto:Data: 2022-09-09T22:39:37.374 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.375 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:37.376 DEBUG:boto:Port: 8001 2022-09-09T22:39:37.376 DEBUG:boto:Params: {} 2022-09-09T22:39:37.377 DEBUG:boto:Token: None 2022-09-09T22:39:37.377 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/ 2022-09-09T22:39:37.378 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sWd7mdST61RlwNhY8z36z8oGtP8= 2022-09-09T22:39:37.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sWd7mdST61RlwNhY8z36z8oGtP8=', 'Content-Length': '0'} 2022-09-09T22:39:37.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000069af496b3f3ff6d-00631bc0a9-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.385 DEBUG:boto:b'jkcgbj-18obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:37.386 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:37.386 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:37.387 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:37.387 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:37.388 DEBUG:boto:Method: GET 2022-09-09T22:39:37.388 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone2 2022-09-09T22:39:37.389 DEBUG:boto:Data: 2022-09-09T22:39:37.389 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.390 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:37.390 DEBUG:boto:Port: 8000 2022-09-09T22:39:37.391 DEBUG:boto:Params: {} 2022-09-09T22:39:37.391 DEBUG:boto:Token: None 2022-09-09T22:39:37.392 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/ 2022-09-09T22:39:37.393 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sWd7mdST61RlwNhY8z36z8oGtP8= 2022-09-09T22:39:37.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sWd7mdST61RlwNhY8z36z8oGtP8=', 'Content-Length': '0'} 2022-09-09T22:39:37.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074b6f807ddd18a46-00631bc0a9-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.442 DEBUG:boto:b'jkcgbj-18obj-test-zone21000falseobj-test-zone22022-09-09T22:36:02.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:37.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:39:37.443 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:37.444 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:37.444 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.445 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.445 DEBUG:boto:Method: GET 2022-09-09T22:39:37.446 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.446 DEBUG:boto:Data: 2022-09-09T22:39:37.447 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.448 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:37.449 DEBUG:boto:Port: 8001 2022-09-09T22:39:37.449 DEBUG:boto:Params: {} 2022-09-09T22:39:37.450 DEBUG:boto:Token: None 2022-09-09T22:39:37.451 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.451 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4oqw4DRxE7TJD5FJN4xbhUr2F5E= 2022-09-09T22:39:37.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4oqw4DRxE7TJD5FJN4xbhUr2F5E=', 'Content-Length': '0'} 2022-09-09T22:39:37.455 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c50c7e6c5a6c588-00631bc0a9-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.456 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:37.456 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:37.457 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.457 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.458 DEBUG:boto:Method: GET 2022-09-09T22:39:37.458 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.459 DEBUG:boto:Data: 2022-09-09T22:39:37.459 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.460 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:37.460 DEBUG:boto:Port: 8000 2022-09-09T22:39:37.461 DEBUG:boto:Params: {} 2022-09-09T22:39:37.461 DEBUG:boto:Token: None 2022-09-09T22:39:37.462 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.462 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4oqw4DRxE7TJD5FJN4xbhUr2F5E= 2022-09-09T22:39:37.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4oqw4DRxE7TJD5FJN4xbhUr2F5E=', 'Content-Length': '0'} 2022-09-09T22:39:37.466 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f3fc7060b48f19e-00631bc0a9-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.467 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:37.468 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:37.477 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.477 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.478 DEBUG:boto:Method: HEAD 2022-09-09T22:39:37.478 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.479 DEBUG:boto:Data: 2022-09-09T22:39:37.479 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.480 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:37.480 DEBUG:boto:Port: 8001 2022-09-09T22:39:37.481 DEBUG:boto:Params: {} 2022-09-09T22:39:37.481 DEBUG:boto:Token: None 2022-09-09T22:39:37.482 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.482 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:QN1OC4GhmbERHN7AxOr/m3nHLRU= 2022-09-09T22:39:37.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:QN1OC4GhmbERHN7AxOr/m3nHLRU=', 'Content-Length': '0'} 2022-09-09T22:39:37.486 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3c98cbdfb4a9e1d-00631bc0a9-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.486 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:37.487 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:37.487 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.488 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.489 DEBUG:boto:Method: HEAD 2022-09-09T22:39:37.489 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.490 DEBUG:boto:Data: 2022-09-09T22:39:37.490 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.491 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:37.491 DEBUG:boto:Port: 8000 2022-09-09T22:39:37.491 DEBUG:boto:Params: {} 2022-09-09T22:39:37.492 DEBUG:boto:Token: None 2022-09-09T22:39:37.492 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.493 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:QN1OC4GhmbERHN7AxOr/m3nHLRU= 2022-09-09T22:39:37.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:QN1OC4GhmbERHN7AxOr/m3nHLRU=', 'Content-Length': '0'} 2022-09-09T22:39:37.497 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c154664012be35c-00631bc0a9-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:39:37.498 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:37.499 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:37.499 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.500 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.500 DEBUG:boto:Method: GET 2022-09-09T22:39:37.501 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.501 DEBUG:boto:Data: 2022-09-09T22:39:37.502 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.502 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:37.503 DEBUG:boto:Port: 8001 2022-09-09T22:39:37.503 DEBUG:boto:Params: {} 2022-09-09T22:39:37.504 DEBUG:boto:Token: None 2022-09-09T22:39:37.504 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.505 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4oqw4DRxE7TJD5FJN4xbhUr2F5E= 2022-09-09T22:39:37.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4oqw4DRxE7TJD5FJN4xbhUr2F5E=', 'Content-Length': '0'} 2022-09-09T22:39:37.509 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000009b1d430ccfc731c-00631bc0a9-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.510 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:37.510 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:37.511 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.511 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.512 DEBUG:boto:Method: GET 2022-09-09T22:39:37.513 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.513 DEBUG:boto:Data: 2022-09-09T22:39:37.514 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.514 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:37.515 DEBUG:boto:Port: 8000 2022-09-09T22:39:37.515 DEBUG:boto:Params: {} 2022-09-09T22:39:37.516 DEBUG:boto:Token: None 2022-09-09T22:39:37.516 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.517 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4oqw4DRxE7TJD5FJN4xbhUr2F5E= 2022-09-09T22:39:37.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4oqw4DRxE7TJD5FJN4xbhUr2F5E=', 'Content-Length': '0'} 2022-09-09T22:39:37.521 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c96f384a240e96d-00631bc0a9-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.522 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:37.522 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:37.523 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:37.523 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:37.524 DEBUG:boto:Method: GET 2022-09-09T22:39:37.524 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:37.525 DEBUG:boto:Data: 2022-09-09T22:39:37.525 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.526 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:37.526 DEBUG:boto:Port: 8001 2022-09-09T22:39:37.527 DEBUG:boto:Params: {} 2022-09-09T22:39:37.527 DEBUG:boto:Token: None 2022-09-09T22:39:37.528 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/ 2022-09-09T22:39:37.528 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sWd7mdST61RlwNhY8z36z8oGtP8= 2022-09-09T22:39:37.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sWd7mdST61RlwNhY8z36z8oGtP8=', 'Content-Length': '0'} 2022-09-09T22:39:37.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009edf29351e9ab878-00631bc0a9-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.536 DEBUG:boto:b'jkcgbj-18obj-test-zone31000falseobj-test-zone32022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:37.536 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:37.537 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:37.537 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:37.538 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:37.538 DEBUG:boto:Method: GET 2022-09-09T22:39:37.539 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:37.539 DEBUG:boto:Data: 2022-09-09T22:39:37.540 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.540 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:37.541 DEBUG:boto:Port: 8000 2022-09-09T22:39:37.541 DEBUG:boto:Params: {} 2022-09-09T22:39:37.542 DEBUG:boto:Token: None 2022-09-09T22:39:37.542 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/ 2022-09-09T22:39:37.543 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sWd7mdST61RlwNhY8z36z8oGtP8= 2022-09-09T22:39:37.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sWd7mdST61RlwNhY8z36z8oGtP8=', 'Content-Length': '0'} 2022-09-09T22:39:37.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9ee62c9bb78b84c-00631bc0a9-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.549 DEBUG:boto:b'jkcgbj-18obj-test-zone31000falseobj-test-zone32022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:37.550 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:37.551 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:37.551 DEBUG:boto:Method: HEAD 2022-09-09T22:39:37.552 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3 2022-09-09T22:39:37.552 DEBUG:boto:Data: 2022-09-09T22:39:37.552 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.553 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:37.553 DEBUG:boto:Port: 8001 2022-09-09T22:39:37.554 DEBUG:boto:Params: {} 2022-09-09T22:39:37.554 DEBUG:boto:Token: None 2022-09-09T22:39:37.555 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/obj-test-zone3 2022-09-09T22:39:37.555 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:xjppte3r0O9Ybd9kEGe2kSuqx4o= 2022-09-09T22:39:37.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:xjppte3r0O9Ybd9kEGe2kSuqx4o=', 'Content-Length': '0'} 2022-09-09T22:39:37.560 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002918ed39a299ac92-00631bc0a9-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.560 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:37.561 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:37.561 DEBUG:boto:Method: HEAD 2022-09-09T22:39:37.562 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3 2022-09-09T22:39:37.563 DEBUG:boto:Data: 2022-09-09T22:39:37.563 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.564 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:37.564 DEBUG:boto:Port: 8000 2022-09-09T22:39:37.565 DEBUG:boto:Params: {} 2022-09-09T22:39:37.565 DEBUG:boto:Token: None 2022-09-09T22:39:37.566 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/obj-test-zone3 2022-09-09T22:39:37.566 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:xjppte3r0O9Ybd9kEGe2kSuqx4o= 2022-09-09T22:39:37.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:xjppte3r0O9Ybd9kEGe2kSuqx4o=', 'Content-Length': '0'} 2022-09-09T22:39:37.571 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ddc1be723e89c3b-00631bc0a9-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-09T22:39:37.572 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:37.572 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:37.573 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.573 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.574 DEBUG:boto:Method: GET 2022-09-09T22:39:37.574 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.575 DEBUG:boto:Data: 2022-09-09T22:39:37.575 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.576 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:37.576 DEBUG:boto:Port: 8001 2022-09-09T22:39:37.577 DEBUG:boto:Params: {} 2022-09-09T22:39:37.578 DEBUG:boto:Token: None 2022-09-09T22:39:37.578 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.579 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4oqw4DRxE7TJD5FJN4xbhUr2F5E= 2022-09-09T22:39:37.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4oqw4DRxE7TJD5FJN4xbhUr2F5E=', 'Content-Length': '0'} 2022-09-09T22:39:37.582 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b372ab9f8cfd4eb-00631bc0a9-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.630 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:37.631 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3 2022-09-09T22:39:37.631 DEBUG:boto:path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.632 DEBUG:boto:auth_path=/jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.632 DEBUG:boto:Method: GET 2022-09-09T22:39:37.633 DEBUG:boto:Path: /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.633 DEBUG:boto:Data: 2022-09-09T22:39:37.634 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.635 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:37.635 DEBUG:boto:Port: 8000 2022-09-09T22:39:37.636 DEBUG:boto:Params: {} 2022-09-09T22:39:37.636 DEBUG:boto:Token: None 2022-09-09T22:39:37.637 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/obj-test-zone3?versionId=qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1 2022-09-09T22:39:37.637 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4oqw4DRxE7TJD5FJN4xbhUr2F5E= 2022-09-09T22:39:37.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4oqw4DRxE7TJD5FJN4xbhUr2F5E=', 'Content-Length': '0'} 2022-09-09T22:39:37.641 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:38:18 GMT'), ('x-amz-version-id', 'qPEG3cw6sheQ-OfBgbtCrfF64xfWPQ1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022a8dffbef6f180b-00631bc0a9-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.642 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:37.642 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:37.643 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:37.643 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:37.644 DEBUG:boto:Method: GET 2022-09-09T22:39:37.644 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:37.645 DEBUG:boto:Data: 2022-09-09T22:39:37.645 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.646 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:39:37.647 DEBUG:boto:Port: 8001 2022-09-09T22:39:37.647 DEBUG:boto:Params: {} 2022-09-09T22:39:37.648 DEBUG:boto:Token: None 2022-09-09T22:39:37.648 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/ 2022-09-09T22:39:37.649 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sWd7mdST61RlwNhY8z36z8oGtP8= 2022-09-09T22:39:37.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sWd7mdST61RlwNhY8z36z8oGtP8=', 'Content-Length': '0'} 2022-09-09T22:39:37.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067e71f524dd7b685-00631bc0a9-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.656 DEBUG:boto:b'jkcgbj-18obj-test-zone31000falseobj-test-zone32022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:37.657 DEBUG:boto:path=/jkcgbj-18/ 2022-09-09T22:39:37.657 DEBUG:boto:auth_path=/jkcgbj-18/ 2022-09-09T22:39:37.658 DEBUG:boto:path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:37.659 DEBUG:boto:auth_path=/jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:37.659 DEBUG:boto:Method: GET 2022-09-09T22:39:37.660 DEBUG:boto:Path: /jkcgbj-18/?prefix=obj-test-zone3 2022-09-09T22:39:37.660 DEBUG:boto:Data: 2022-09-09T22:39:37.661 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.661 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:39:37.662 DEBUG:boto:Port: 8000 2022-09-09T22:39:37.662 DEBUG:boto:Params: {} 2022-09-09T22:39:37.663 DEBUG:boto:Token: None 2022-09-09T22:39:37.663 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-18/ 2022-09-09T22:39:37.664 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sWd7mdST61RlwNhY8z36z8oGtP8= 2022-09-09T22:39:37.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sWd7mdST61RlwNhY8z36z8oGtP8=', 'Content-Length': '0'} 2022-09-09T22:39:37.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000121edc4007a0bebc-00631bc0a9-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.671 DEBUG:boto:b'jkcgbj-18obj-test-zone31000falseobj-test-zone32022-09-09T22:36:03.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:39:37.671 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-18 zones={test-zone3, test-zone2} 2022-09-09T22:39:37.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2022-09-09T22:39:37.676 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:39:37.677 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:39:37.677 DEBUG:boto:Method: PUT 2022-09-09T22:39:37.678 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:39:37.678 DEBUG:boto:Data: 2022-09-09T22:39:37.679 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.679 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:37.680 DEBUG:boto:Port: 8000 2022-09-09T22:39:37.680 DEBUG:boto:Params: {} 2022-09-09T22:39:37.681 DEBUG:boto:Token: None 2022-09-09T22:39:37.681 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-19/ 2022-09-09T22:39:37.682 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:mQlyBrsFDeZ1Z3lpPR5nZ40BudE= 2022-09-09T22:39:37.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:mQlyBrsFDeZ1Z3lpPR5nZ40BudE=', 'Content-Length': '0'} 2022-09-09T22:39:37.707 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000062733ed9ee7b2498-00631bc0a9-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.708 DEBUG:rgw_multi.tests:created bucket=jkcgbj-19 2022-09-09T22:39:37.708 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:39:37.709 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:39:37.709 DEBUG:boto:path=/jkcgbj-19/?versioning 2022-09-09T22:39:37.710 DEBUG:boto:auth_path=/jkcgbj-19/?versioning 2022-09-09T22:39:37.711 DEBUG:boto:Method: PUT 2022-09-09T22:39:37.711 DEBUG:boto:Path: /jkcgbj-19/?versioning 2022-09-09T22:39:37.712 DEBUG:boto:Data: Enabled Disabled 2022-09-09T22:39:37.712 DEBUG:boto:Headers: {} 2022-09-09T22:39:37.713 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:37.713 DEBUG:boto:Port: 8000 2022-09-09T22:39:37.714 DEBUG:boto:Params: {} 2022-09-09T22:39:37.714 DEBUG:boto:Token: None 2022-09-09T22:39:37.715 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:39:37 GMT /jkcgbj-19/?versioning 2022-09-09T22:39:37.715 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9/lb4rbMn7cokZHEVu7XJdV7qts= 2022-09-09T22:39:37.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:37 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9/lb4rbMn7cokZHEVu7XJdV7qts=', 'Content-Length': '228'} 2022-09-09T22:39:37.725 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000073d33f7d148f102e-00631bc0a9-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:37.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:39:37.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:39:37.727 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:39:37.773 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:39:37.774 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:39:37.827 INFO:teuthology.orchestra.run.smithi005.stderr:No --period given, using current period=503b1d61-4513-48c8-a308-17b76219610c 2022-09-09T22:39:37.854 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T22:39:37.855 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:39:37.856 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:39:37.856 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:28:06.146147Z" 2022-09-09T22:39:37.857 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:39:37.857 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:39:37.858 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:39:37.858 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:39:37.859 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:39:37.859 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:39:37.859 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:39:37.860 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:30:37.757014Z" 2022-09-09T22:39:37.860 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:39:37.861 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:39:37.861 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662763177.721058_92.1", 2022-09-09T22:39:37.861 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:39:37.721058Z" 2022-09-09T22:39:37.861 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:39:37.862 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T22:39:37.863 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763177.721058_92.1'} 2022-09-09T22:39:37.864 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-09T22:39:37.864 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:39:37.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:39:37.866 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:39:37.921 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:39:37.922 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:39:37.980 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:39:37.978+0000 7f5ce8dd9dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:39:37.982 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:39:37.978+0000 7f5ce8dd9dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:39:38.005 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:39:38.005 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:39:38.005 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:39:38.006 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:39:38.006 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:39:38.006 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:39:38.007 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:39:38.007 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:39:38.007 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:39:38.007 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:39:38.008 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:39:38.008 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:39:38.008 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:39:38.008 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:39:38.009 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:39:38.009 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:39:38.009 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:39:38.010 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:39:38.010 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:39:38.010 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:39:38.010 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:39:38.011 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:39:38.011 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:39:38.011 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:39:38.011 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:39:38.012 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:39:38.012 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:39:38.012 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:39:38.013 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:39:38.013 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:39:38.013 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:39:38.013 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:39:38.014 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:39:38.014 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:39:38.014 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:39:38.014 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:39:38.015 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:39:38.015 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:39:38.015 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:39:38.015 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:39:38.016 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:39:38.016 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:39:38.017 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:39:38.017 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:39:38.017 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:39:38.018 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:39:38.018 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:39:38.018 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:39:38.019 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:39:38.019 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763177.690963_87.1", 2022-09-09T22:39:38.019 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:39:38.019 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:39:38.020 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:39:38.020 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:39:37.690963Z", 2022-09-09T22:39:38.020 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:39:38.021 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:39:38.021 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:39:38.021 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:39:38.022 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:39:38.022 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:39:38.022 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:39:38.023 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:39:38.023 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:39:38.023 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:39:37Z" 2022-09-09T22:39:38.023 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:39:38.024 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763177.690963_87.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:39:37.690963Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:39:37Z" } 2022-09-09T22:39:38.025 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763177.690963_87.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:39:37.690963Z', 'realm_epoch': 2}}] 2022-09-09T22:39:38.026 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763177.721058_92.1'} 2022-09-09T22:39:38.026 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763177.690963_87.1'} 2022-09-09T22:39:38.027 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763177.721058_92.1 target=1_1662763177.690963_87.1 2022-09-09T22:39:43.028 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:39:43.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:39:43.029 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:39:43.078 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:39:43.078 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:39:43.139 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:39:43.134+0000 7fb5b882ddc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:39:43.141 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:39:43.138+0000 7fb5b882ddc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:39:43.168 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:39:43.169 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:39:43.169 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:39:43.169 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:39:43.169 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:39:43.170 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:39:43.170 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:39:43.170 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:39:43.171 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:39:43.171 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:39:43.171 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:39:43.171 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:39:43.171 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:39:43.172 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:39:43.172 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:39:43.172 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:39:43.172 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:39:43.173 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:39:43.173 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:39:43.173 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:39:43.173 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:39:43.174 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:39:43.174 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:39:43.174 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:39:43.175 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:39:43.175 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:39:43.175 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:39:43.175 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:39:43.176 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:39:43.176 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:39:43.176 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:39:43.176 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:39:43.177 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:39:43.177 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:39:43.177 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:39:43.177 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:39:43.178 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:39:43.178 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:39:43.178 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:39:43.178 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:39:43.179 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:39:43.179 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:39:43.179 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:39:43.179 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:39:43.180 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:39:43.180 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:39:43.180 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:39:43.180 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:39:43.181 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:39:43.181 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763177.721058_92.1", 2022-09-09T22:39:43.181 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:39:43.181 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:39:43.182 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:39:43.182 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:39:37.721058Z", 2022-09-09T22:39:43.182 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:39:43.183 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:39:43.183 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:39:43.183 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:39:43.183 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:39:43.183 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:39:43.184 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:39:43.184 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:39:43.184 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:39:43.184 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:39:43Z" 2022-09-09T22:39:43.185 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:39:43.186 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763177.721058_92.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:39:37.721058Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:39:43Z" } 2022-09-09T22:39:43.187 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763177.721058_92.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:39:37.721058Z', 'realm_epoch': 2}}] 2022-09-09T22:39:43.187 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763177.721058_92.1'} 2022-09-09T22:39:43.188 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763177.721058_92.1'} 2022-09-09T22:39:43.188 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-09T22:39:43.189 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-09T22:39:43.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:39:43.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:39:43.190 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:39:43.239 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:39:43.239 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:39:43.291 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:39:43.283+0000 7f2dc7643dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:39:43.293 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:39:43.287+0000 7f2dc7643dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:39:43.319 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:39:43.319 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T22:39:43.319 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T22:39:43.320 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T22:39:43.320 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T22:39:43.320 INFO:teuthology.orchestra.run.smithi005.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:39:43.320 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:39:43.321 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:39:43.321 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T22:39:43.321 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:39:43.321 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T22:39:43.322 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:39:43.322 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:39:43.322 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:39:43.322 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:39:43.323 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 1, 2022-09-09T22:39:43.323 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:39:43.323 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:39:43.324 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:39:43.324 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:39:43.324 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:39:43.324 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:39:43.325 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T22:39:43.325 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:39:43.325 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:39:43.325 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:39:43.326 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:39:43.326 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:39:43.326 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:39:43.326 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:39:43.327 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:39:43.327 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:39:43.327 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:39:43.327 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:39:43.328 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T22:39:43.328 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:39:43.328 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:39:43.328 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:39:43.329 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:39:43.329 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:39:43.329 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:39:43.329 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:39:43.330 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:39:43.330 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:39:43.330 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:39:43.330 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:39:43.331 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T22:39:43.331 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:39:43.331 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:39:43.331 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662763177.721058_92.1", 2022-09-09T22:39:43.332 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:39:43.332 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:39:43.332 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:39:43.333 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:39:37.721058Z", 2022-09-09T22:39:43.333 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:39:43.333 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:39:43.333 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:39:43.334 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:39:43.334 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:39:43.334 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T22:39:43.334 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 1, 2022-09-09T22:39:43.335 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 1 2022-09-09T22:39:43.335 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:39:43.335 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T22:39:43Z" 2022-09-09T22:39:43.335 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:39:43.337 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763177.721058_92.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:39:37.721058Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:39:43Z" } 2022-09-09T22:39:43.337 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763177.721058_92.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:39:37.721058Z', 'realm_epoch': 2}}] 2022-09-09T22:39:43.338 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763177.721058_92.1'} 2022-09-09T22:39:43.338 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763177.721058_92.1'} 2022-09-09T22:39:43.339 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-09T22:39:43.339 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-09T22:39:43.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:39:43.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:39:43.341 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:39:43.390 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:39:43.391 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:39:43.443 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:39:43.442+0000 7ffa37ed1dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:39:43.445 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:39:43.442+0000 7ffa37ed1dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:39:43.469 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:39:43.470 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:39:43.470 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:39:43.470 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:39:43.471 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:39:43.471 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:39:43.471 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:39:43.471 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:39:43.472 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:39:43.472 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:39:43.472 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:39:43.472 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:39:43.473 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:39:43.473 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:39:43.473 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:39:43.473 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:39:43.474 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:39:43.474 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:39:43.474 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:39:43.475 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:39:43.475 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:39:43.475 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:39:43.475 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:39:43.476 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:39:43.476 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:39:43.476 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:39:43.476 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:39:43.477 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:39:43.477 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:39:43.477 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:39:43.477 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:39:43.478 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:39:43.478 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:39:43.478 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:39:43.478 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:39:43.479 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:39:43.479 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:39:43.479 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:39:43.479 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:39:43.480 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:39:43.480 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:39:43.480 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:39:43.480 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:39:43.481 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:39:43.481 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:39:43.481 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:39:43.481 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:39:43.482 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:39:43.482 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:39:43.482 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763177.721058_92.1", 2022-09-09T22:39:43.483 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:39:43.483 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:39:43.483 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:39:43.483 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:39:37.721058Z", 2022-09-09T22:39:43.484 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:39:43.484 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:39:43.484 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:39:43.484 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:39:43.485 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:39:43.485 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:39:43.485 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:39:43.485 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:39:43.486 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:39:43.486 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:39:43Z" 2022-09-09T22:39:43.486 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:39:43.488 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763177.721058_92.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:39:37.721058Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:39:43Z" } 2022-09-09T22:39:43.489 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763177.721058_92.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:39:37.721058Z', 'realm_epoch': 2}}] 2022-09-09T22:39:43.489 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763177.721058_92.1'} 2022-09-09T22:39:43.490 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763177.721058_92.1'} 2022-09-09T22:39:43.490 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-09T22:39:43.491 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:39:43.492 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:39:43.492 DEBUG:boto:Method: HEAD 2022-09-09T22:39:43.493 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:39:43.493 DEBUG:boto:Data: 2022-09-09T22:39:43.494 DEBUG:boto:Headers: {} 2022-09-09T22:39:43.495 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:43.495 DEBUG:boto:Port: 8000 2022-09-09T22:39:43.496 DEBUG:boto:Params: {} 2022-09-09T22:39:43.496 DEBUG:boto:Token: None 2022-09-09T22:39:43.497 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:39:43 GMT /jkcgbj-19/ 2022-09-09T22:39:43.498 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0e15vsoq5/AfBAbarH4XIOx7atQ= 2022-09-09T22:39:43.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:39:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0e15vsoq5/AfBAbarH4XIOx7atQ=', 'Content-Length': '0'} 2022-09-09T22:39:43.504 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d309bad3fb6de63d-00631bc0af-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:43.504 DEBUG:boto:path=/jkcgbj-19/dummy 2022-09-09T22:39:43.505 DEBUG:boto:auth_path=/jkcgbj-19/dummy 2022-09-09T22:39:43.506 DEBUG:boto:Method: PUT 2022-09-09T22:39:43.506 DEBUG:boto:Path: /jkcgbj-19/dummy 2022-09-09T22:39:43.507 DEBUG:boto:Data: 2022-09-09T22:39:43.507 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-09T22:39:43.508 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:39:43.508 DEBUG:boto:Port: 8000 2022-09-09T22:39:43.509 DEBUG:boto:Params: {} 2022-09-09T22:39:43.509 DEBUG:boto:Token: None 2022-09-09T22:39:43.510 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 09 Sep 2022 22:39:43 GMT /jkcgbj-19/dummy 2022-09-09T22:39:43.511 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vCG3NFqwdzBhp7NO3UjPcEMW43g= 2022-09-09T22:39:43.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:39:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vCG3NFqwdzBhp7NO3UjPcEMW43g='} 2022-09-09T22:39:43.539 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Ux4yCo1Zw0Z6LcOb725cQEb153cMR29'), ('x-amz-request-id', 'tx00000f86af0ee88378163-00631bc0af-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:39:43.540 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jkcgbj-19 2022-09-09T22:39:43.541 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:43.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:39:43.542 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:39:43.597 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:39:43.598 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:39:43.660 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:39:43.658+0000 7f6b32d4fdc0 1 waiting to reach incremental sync.. 2022-09-09T22:39:48.662 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:39:48.658+0000 7f6b32d4fdc0 1 waiting to reach incremental sync.. 2022-09-09T22:39:53.663 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:39:53.658+0000 7f6b32d4fdc0 1 waiting to reach incremental sync.. 2022-09-09T22:39:58.664 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:39:58.662+0000 7f6b32d4fdc0 1 waiting to reach incremental sync.. 2022-09-09T22:40:03.666 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:40:03.662+0000 7f6b32d4fdc0 1 bucket sync caught up with source: 2022-09-09T22:40:03.667 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000002.6.10, , , , , ] 2022-09-09T22:40:03.667 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000002.6.10, , , , , ] 2022-09-09T22:40:03.667 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:40:03.662+0000 7f6b32d4fdc0 0 bucket checkpoint complete 2022-09-09T22:40:03.692 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jkcgbj-19 2022-09-09T22:40:03.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:40:03.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:40:03.694 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:40:03.743 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:40:03.744 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:40:03.811 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:40:03.802+0000 7f85ad7fddc0 1 bucket sync caught up with source: 2022-09-09T22:40:03.812 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , 00000000002.6.10, , , , , ] 2022-09-09T22:40:03.812 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , 00000000002.6.10, , , , , ] 2022-09-09T22:40:03.812 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:40:03.802+0000 7f85ad7fddc0 0 bucket checkpoint complete 2022-09-09T22:40:03.836 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jkcgbj-19 2022-09-09T22:40:03.837 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:40:03.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:40:03.838 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:40:03.893 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:40:03.894 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:40:03.981 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:40:03.978+0000 7f6d4bc87dc0 1 bucket sync caught up with source: 2022-09-09T22:40:03.982 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000002.6.10, , , , , ] 2022-09-09T22:40:03.982 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000002.6.10, , , , , ] 2022-09-09T22:40:03.982 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:40:03.978+0000 7f6d4bc87dc0 0 bucket checkpoint complete 2022-09-09T22:40:04.014 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jkcgbj-19 2022-09-09T22:40:04.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:40:04.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:40:04.016 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:40:04.071 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:40:04.071 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:40:04.139 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:40:04.130+0000 7f694348ddc0 1 waiting to reach incremental sync.. 2022-09-09T22:40:09.140 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:40:09.130+0000 7f694348ddc0 1 waiting to reach incremental sync.. 2022-09-09T22:40:14.141 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:40:14.134+0000 7f694348ddc0 1 waiting to reach incremental sync.. 2022-09-09T22:40:19.142 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:40:19.134+0000 7f694348ddc0 1 waiting to reach incremental sync.. 2022-09-09T22:40:24.146 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:40:24.138+0000 7f694348ddc0 1 bucket sync caught up with source: 2022-09-09T22:40:24.146 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , 00000000004.16.9, , , , , ] 2022-09-09T22:40:24.146 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , 00000000002.12.10, , , , , ] 2022-09-09T22:40:24.147 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:40:24.138+0000 7f694348ddc0 0 bucket checkpoint complete 2022-09-09T22:40:24.174 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jkcgbj-19 2022-09-09T22:40:24.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:40:24.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:40:24.176 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:40:24.230 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:40:24.231 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:40:24.294 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:40:24.286+0000 7fef864f0dc0 1 bucket sync caught up with source: 2022-09-09T22:40:24.295 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , 00000000004.16.9, , , , , ] 2022-09-09T22:40:24.295 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , 00000000004.16.9, , , , , ] 2022-09-09T22:40:24.296 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:40:24.286+0000 7fef864f0dc0 0 bucket checkpoint complete 2022-09-09T22:40:24.321 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jkcgbj-19 2022-09-09T22:40:24.322 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jkcgbj-19 2022-09-09T22:40:24.323 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:40:24.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:40:24.324 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:40:24.379 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:40:24.380 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:40:24.447 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:40:24.438+0000 7f46f4d18dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:40:24.448 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , 00000000003.15.9, , , , , ] 2022-09-09T22:40:24.448 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , 00000000004.17.9, , , , , ] 2022-09-09T22:40:29.450 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:40:29.442+0000 7f46f4d18dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:40:29.451 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , 00000000003.15.9, , , , , ] 2022-09-09T22:40:29.451 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , 00000000004.17.9, , , , , ] 2022-09-09T22:40:34.453 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:40:34.446+0000 7f46f4d18dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:40:34.454 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , 00000000003.15.9, , , , , ] 2022-09-09T22:40:34.454 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , 00000000004.17.9, , , , , ] 2022-09-09T22:40:39.456 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:40:39.450+0000 7f46f4d18dc0 1 bucket sync caught up with source: 2022-09-09T22:40:39.457 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , 00000000004.17.9, , , , , ] 2022-09-09T22:40:39.457 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , 00000000004.17.9, , , , , ] 2022-09-09T22:40:39.458 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:40:39.450+0000 7f46f4d18dc0 0 bucket checkpoint complete 2022-09-09T22:40:39.484 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jkcgbj-19 2022-09-09T22:40:39.485 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:40:39.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:40:39.486 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:40:39.541 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:40:39.541 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:40:39.618 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:40:39.613+0000 7fdea9283dc0 1 bucket sync caught up with source: 2022-09-09T22:40:39.618 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , , , ] 2022-09-09T22:40:39.619 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , , , ] 2022-09-09T22:40:39.619 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:40:39.613+0000 7fdea9283dc0 0 bucket checkpoint complete 2022-09-09T22:40:39.646 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jkcgbj-19 2022-09-09T22:40:39.647 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-19 zones={test-zone2, test-zone1} 2022-09-09T22:40:39.647 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:39.648 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:39.648 DEBUG:boto:Method: HEAD 2022-09-09T22:40:39.649 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:39.649 DEBUG:boto:Data: 2022-09-09T22:40:39.650 DEBUG:boto:Headers: {} 2022-09-09T22:40:39.650 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:39.651 DEBUG:boto:Port: 8000 2022-09-09T22:40:39.651 DEBUG:boto:Params: {} 2022-09-09T22:40:39.652 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T22:40:39.653 DEBUG:boto:Token: None 2022-09-09T22:40:39.654 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:39 GMT /jkcgbj-19/ 2022-09-09T22:40:39.654 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:M/WOO4vzeAzS1zypmFSpAvlI1Gc= 2022-09-09T22:40:39.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:39 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:M/WOO4vzeAzS1zypmFSpAvlI1Gc=', 'Content-Length': '0'} 2022-09-09T22:40:39.661 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000097644c0a15e97fb8-00631bc0e7-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:39.661 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:39.662 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:39.663 DEBUG:boto:Method: HEAD 2022-09-09T22:40:39.663 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:39.664 DEBUG:boto:Data: 2022-09-09T22:40:39.664 DEBUG:boto:Headers: {} 2022-09-09T22:40:39.665 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:39.666 DEBUG:boto:Port: 8000 2022-09-09T22:40:39.666 DEBUG:boto:Params: {} 2022-09-09T22:40:39.667 DEBUG:boto:Token: None 2022-09-09T22:40:39.668 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:39 GMT /jkcgbj-19/ 2022-09-09T22:40:39.668 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:M/WOO4vzeAzS1zypmFSpAvlI1Gc= 2022-09-09T22:40:39.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:39 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:M/WOO4vzeAzS1zypmFSpAvlI1Gc=', 'Content-Length': '0'} 2022-09-09T22:40:39.673 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004bb3d4b967d7be5e-00631bc0e7-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:39.673 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:40:39.674 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:39.675 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:39.675 DEBUG:boto:path=/jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:39.676 DEBUG:boto:auth_path=/jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:39.677 DEBUG:boto:Method: GET 2022-09-09T22:40:39.679 DEBUG:boto:Path: /jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:39.680 DEBUG:boto:Data: 2022-09-09T22:40:39.681 DEBUG:boto:Headers: {} 2022-09-09T22:40:39.682 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:39.683 DEBUG:boto:Port: 8000 2022-09-09T22:40:39.684 DEBUG:boto:Params: {} 2022-09-09T22:40:39.685 DEBUG:boto:Token: None 2022-09-09T22:40:39.686 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:39 GMT /jkcgbj-19/?versions 2022-09-09T22:40:39.687 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6Z+3KdQbUH8M2wDDDWTfLcSy57g= 2022-09-09T22:40:39.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:39 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6Z+3KdQbUH8M2wDDDWTfLcSy57g=', 'Content-Length': '0'} 2022-09-09T22:40:39.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b8ae0b0a71ff47a-00631bc0e7-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:39.695 DEBUG:boto:b'jkcgbj-19999falsedummyUx4yCo1Zw0Z6LcOb725cQEb153cMR29true2022-09-09T22:39:43.512Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:40:39.697 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:40:39.698 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:40:39.699 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:39.700 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:39.700 DEBUG:boto:path=/jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:39.701 DEBUG:boto:auth_path=/jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:39.702 DEBUG:boto:Method: GET 2022-09-09T22:40:39.703 DEBUG:boto:Path: /jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:39.704 DEBUG:boto:Data: 2022-09-09T22:40:39.705 DEBUG:boto:Headers: {} 2022-09-09T22:40:39.706 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:39.707 DEBUG:boto:Port: 8000 2022-09-09T22:40:39.708 DEBUG:boto:Params: {} 2022-09-09T22:40:39.710 DEBUG:boto:Token: None 2022-09-09T22:40:39.711 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:39 GMT /jkcgbj-19/?versions 2022-09-09T22:40:39.712 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6Z+3KdQbUH8M2wDDDWTfLcSy57g= 2022-09-09T22:40:39.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:39 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6Z+3KdQbUH8M2wDDDWTfLcSy57g=', 'Content-Length': '0'} 2022-09-09T22:40:39.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d57f8b87fb5b52c-00631bc0e7-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:39.720 DEBUG:boto:b'jkcgbj-19999falsedummyUx4yCo1Zw0Z6LcOb725cQEb153cMR29true2022-09-09T22:39:43.512Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:40:39.721 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:40:39.722 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:39.723 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:39.724 DEBUG:boto:path=/jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:39.725 DEBUG:boto:auth_path=/jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:39.726 DEBUG:boto:Method: GET 2022-09-09T22:40:39.727 DEBUG:boto:Path: /jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:39.728 DEBUG:boto:Data: 2022-09-09T22:40:39.729 DEBUG:boto:Headers: {} 2022-09-09T22:40:39.730 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:39.731 DEBUG:boto:Port: 8000 2022-09-09T22:40:39.732 DEBUG:boto:Params: {} 2022-09-09T22:40:39.733 DEBUG:boto:Token: None 2022-09-09T22:40:39.734 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:39 GMT /jkcgbj-19/?versions 2022-09-09T22:40:39.735 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6Z+3KdQbUH8M2wDDDWTfLcSy57g= 2022-09-09T22:40:39.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:39 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6Z+3KdQbUH8M2wDDDWTfLcSy57g=', 'Content-Length': '0'} 2022-09-09T22:40:39.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b090f591379aa2a-00631bc0e7-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:39.742 DEBUG:boto:b'jkcgbj-19999falsedummyUx4yCo1Zw0Z6LcOb725cQEb153cMR29true2022-09-09T22:39:43.512Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:40:39.743 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:39.744 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:39.744 DEBUG:boto:path=/jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:39.745 DEBUG:boto:auth_path=/jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:39.745 DEBUG:boto:Method: GET 2022-09-09T22:40:39.746 DEBUG:boto:Path: /jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:39.746 DEBUG:boto:Data: 2022-09-09T22:40:39.747 DEBUG:boto:Headers: {} 2022-09-09T22:40:39.747 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:39.748 DEBUG:boto:Port: 8000 2022-09-09T22:40:39.748 DEBUG:boto:Params: {} 2022-09-09T22:40:39.749 DEBUG:boto:Token: None 2022-09-09T22:40:39.750 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:39 GMT /jkcgbj-19/?versions 2022-09-09T22:40:39.750 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6Z+3KdQbUH8M2wDDDWTfLcSy57g= 2022-09-09T22:40:39.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:39 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6Z+3KdQbUH8M2wDDDWTfLcSy57g=', 'Content-Length': '0'} 2022-09-09T22:40:39.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d32124fefd990e7-00631bc0e7-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:39.798 DEBUG:boto:b'jkcgbj-19999falsedummyUx4yCo1Zw0Z6LcOb725cQEb153cMR29true2022-09-09T22:39:43.512Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:40:39.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:40:39.800 DEBUG:boto:path=/jkcgbj-19/dummy 2022-09-09T22:40:39.800 DEBUG:boto:auth_path=/jkcgbj-19/dummy 2022-09-09T22:40:39.801 DEBUG:boto:path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.802 DEBUG:boto:auth_path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.802 DEBUG:boto:Method: GET 2022-09-09T22:40:39.803 DEBUG:boto:Path: /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.803 DEBUG:boto:Data: 2022-09-09T22:40:39.804 DEBUG:boto:Headers: {} 2022-09-09T22:40:39.804 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:39.805 DEBUG:boto:Port: 8000 2022-09-09T22:40:39.805 DEBUG:boto:Params: {} 2022-09-09T22:40:39.806 DEBUG:boto:Token: None 2022-09-09T22:40:39.807 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:39 GMT /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.807 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:mON28ovblDW8wPIad8vwG2OnmH4= 2022-09-09T22:40:39.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:39 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:mON28ovblDW8wPIad8vwG2OnmH4=', 'Content-Length': '0'} 2022-09-09T22:40:39.811 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('x-amz-version-id', 'Ux4yCo1Zw0Z6LcOb725cQEb153cMR29'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001d5244ca92fc2368-00631bc0e7-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:39.812 DEBUG:boto:path=/jkcgbj-19/dummy 2022-09-09T22:40:39.813 DEBUG:boto:auth_path=/jkcgbj-19/dummy 2022-09-09T22:40:39.813 DEBUG:boto:path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.814 DEBUG:boto:auth_path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.815 DEBUG:boto:Method: GET 2022-09-09T22:40:39.815 DEBUG:boto:Path: /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.816 DEBUG:boto:Data: 2022-09-09T22:40:39.816 DEBUG:boto:Headers: {} 2022-09-09T22:40:39.817 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:39.817 DEBUG:boto:Port: 8000 2022-09-09T22:40:39.818 DEBUG:boto:Params: {} 2022-09-09T22:40:39.818 DEBUG:boto:Token: None 2022-09-09T22:40:39.819 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:39 GMT /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.820 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:mON28ovblDW8wPIad8vwG2OnmH4= 2022-09-09T22:40:39.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:39 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:mON28ovblDW8wPIad8vwG2OnmH4=', 'Content-Length': '0'} 2022-09-09T22:40:39.823 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('x-amz-version-id', 'Ux4yCo1Zw0Z6LcOb725cQEb153cMR29'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000012f83b70022bd37b-00631bc0e7-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:39.824 DEBUG:boto:path=/jkcgbj-19/dummy 2022-09-09T22:40:39.825 DEBUG:boto:auth_path=/jkcgbj-19/dummy 2022-09-09T22:40:39.825 DEBUG:boto:path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.826 DEBUG:boto:auth_path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.826 DEBUG:boto:Method: HEAD 2022-09-09T22:40:39.827 DEBUG:boto:Path: /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.828 DEBUG:boto:Data: 2022-09-09T22:40:39.828 DEBUG:boto:Headers: {} 2022-09-09T22:40:39.828 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:39.829 DEBUG:boto:Port: 8000 2022-09-09T22:40:39.829 DEBUG:boto:Params: {} 2022-09-09T22:40:39.830 DEBUG:boto:Token: None 2022-09-09T22:40:39.831 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:39 GMT /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.831 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:DKqfNTiy2VGmIbOYSlF+mFQkdzU= 2022-09-09T22:40:39.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:39 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:DKqfNTiy2VGmIbOYSlF+mFQkdzU=', 'Content-Length': '0'} 2022-09-09T22:40:39.835 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('x-amz-version-id', 'Ux4yCo1Zw0Z6LcOb725cQEb153cMR29'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000024d54ac14ae907dd-00631bc0e7-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:39.836 DEBUG:boto:path=/jkcgbj-19/dummy 2022-09-09T22:40:39.836 DEBUG:boto:auth_path=/jkcgbj-19/dummy 2022-09-09T22:40:39.837 DEBUG:boto:path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.837 DEBUG:boto:auth_path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.838 DEBUG:boto:Method: HEAD 2022-09-09T22:40:39.838 DEBUG:boto:Path: /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.839 DEBUG:boto:Data: 2022-09-09T22:40:39.839 DEBUG:boto:Headers: {} 2022-09-09T22:40:39.840 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:39.840 DEBUG:boto:Port: 8000 2022-09-09T22:40:39.841 DEBUG:boto:Params: {} 2022-09-09T22:40:39.841 DEBUG:boto:Token: None 2022-09-09T22:40:39.842 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:39 GMT /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.842 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:DKqfNTiy2VGmIbOYSlF+mFQkdzU= 2022-09-09T22:40:39.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:39 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:DKqfNTiy2VGmIbOYSlF+mFQkdzU=', 'Content-Length': '0'} 2022-09-09T22:40:39.846 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('x-amz-version-id', 'Ux4yCo1Zw0Z6LcOb725cQEb153cMR29'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008986b5a62fe6cc97-00631bc0e7-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:39.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:40:39.847 DEBUG:boto:path=/jkcgbj-19/dummy 2022-09-09T22:40:39.848 DEBUG:boto:auth_path=/jkcgbj-19/dummy 2022-09-09T22:40:39.848 DEBUG:boto:path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.849 DEBUG:boto:auth_path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.849 DEBUG:boto:Method: GET 2022-09-09T22:40:39.850 DEBUG:boto:Path: /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.850 DEBUG:boto:Data: 2022-09-09T22:40:39.851 DEBUG:boto:Headers: {} 2022-09-09T22:40:39.851 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:39.852 DEBUG:boto:Port: 8000 2022-09-09T22:40:39.852 DEBUG:boto:Params: {} 2022-09-09T22:40:39.853 DEBUG:boto:Token: None 2022-09-09T22:40:39.854 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:39 GMT /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.854 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:mON28ovblDW8wPIad8vwG2OnmH4= 2022-09-09T22:40:39.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:39 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:mON28ovblDW8wPIad8vwG2OnmH4=', 'Content-Length': '0'} 2022-09-09T22:40:39.858 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('x-amz-version-id', 'Ux4yCo1Zw0Z6LcOb725cQEb153cMR29'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001976a814944f6f31-00631bc0e7-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:39.859 DEBUG:boto:path=/jkcgbj-19/dummy 2022-09-09T22:40:39.859 DEBUG:boto:auth_path=/jkcgbj-19/dummy 2022-09-09T22:40:39.860 DEBUG:boto:path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.860 DEBUG:boto:auth_path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.861 DEBUG:boto:Method: GET 2022-09-09T22:40:39.861 DEBUG:boto:Path: /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.862 DEBUG:boto:Data: 2022-09-09T22:40:39.862 DEBUG:boto:Headers: {} 2022-09-09T22:40:39.863 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:39.863 DEBUG:boto:Port: 8000 2022-09-09T22:40:39.864 DEBUG:boto:Params: {} 2022-09-09T22:40:39.864 DEBUG:boto:Token: None 2022-09-09T22:40:39.865 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:39 GMT /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.866 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:mON28ovblDW8wPIad8vwG2OnmH4= 2022-09-09T22:40:39.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:39 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:mON28ovblDW8wPIad8vwG2OnmH4=', 'Content-Length': '0'} 2022-09-09T22:40:39.869 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('x-amz-version-id', 'Ux4yCo1Zw0Z6LcOb725cQEb153cMR29'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000066d7a4f393bffa9e-00631bc0e7-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:39.870 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:39.871 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:39.871 DEBUG:boto:path=/jkcgbj-19/?prefix=dummy 2022-09-09T22:40:39.872 DEBUG:boto:auth_path=/jkcgbj-19/?prefix=dummy 2022-09-09T22:40:39.872 DEBUG:boto:Method: GET 2022-09-09T22:40:39.873 DEBUG:boto:Path: /jkcgbj-19/?prefix=dummy 2022-09-09T22:40:39.873 DEBUG:boto:Data: 2022-09-09T22:40:39.874 DEBUG:boto:Headers: {} 2022-09-09T22:40:39.875 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:39.875 DEBUG:boto:Port: 8000 2022-09-09T22:40:39.876 DEBUG:boto:Params: {} 2022-09-09T22:40:39.876 DEBUG:boto:Token: None 2022-09-09T22:40:39.877 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:39 GMT /jkcgbj-19/ 2022-09-09T22:40:39.877 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qWmJX//iMkk+PA0x68HE3/Dd2SQ= 2022-09-09T22:40:39.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:39 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qWmJX//iMkk+PA0x68HE3/Dd2SQ=', 'Content-Length': '0'} 2022-09-09T22:40:39.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049318341fe7c08c0-00631bc0e7-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:39.926 DEBUG:boto:b'jkcgbj-19dummy1000falsedummy2022-09-09T22:39:43.512Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:40:39.927 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:39.928 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:39.928 DEBUG:boto:path=/jkcgbj-19/?prefix=dummy 2022-09-09T22:40:39.929 DEBUG:boto:auth_path=/jkcgbj-19/?prefix=dummy 2022-09-09T22:40:39.929 DEBUG:boto:Method: GET 2022-09-09T22:40:39.930 DEBUG:boto:Path: /jkcgbj-19/?prefix=dummy 2022-09-09T22:40:39.930 DEBUG:boto:Data: 2022-09-09T22:40:39.931 DEBUG:boto:Headers: {} 2022-09-09T22:40:39.931 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:39.932 DEBUG:boto:Port: 8000 2022-09-09T22:40:39.932 DEBUG:boto:Params: {} 2022-09-09T22:40:39.933 DEBUG:boto:Token: None 2022-09-09T22:40:39.934 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:39 GMT /jkcgbj-19/ 2022-09-09T22:40:39.934 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qWmJX//iMkk+PA0x68HE3/Dd2SQ= 2022-09-09T22:40:39.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:39 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qWmJX//iMkk+PA0x68HE3/Dd2SQ=', 'Content-Length': '0'} 2022-09-09T22:40:39.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002863a11fa8405e54-00631bc0e7-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:39.941 DEBUG:boto:b'jkcgbj-19dummy1000falsedummy2022-09-09T22:39:43.512Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:40:39.941 DEBUG:boto:path=/jkcgbj-19/dummy 2022-09-09T22:40:39.942 DEBUG:boto:auth_path=/jkcgbj-19/dummy 2022-09-09T22:40:39.943 DEBUG:boto:Method: HEAD 2022-09-09T22:40:39.944 DEBUG:boto:Path: /jkcgbj-19/dummy 2022-09-09T22:40:39.944 DEBUG:boto:Data: 2022-09-09T22:40:39.945 DEBUG:boto:Headers: {} 2022-09-09T22:40:39.945 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:39.945 DEBUG:boto:Port: 8000 2022-09-09T22:40:39.946 DEBUG:boto:Params: {} 2022-09-09T22:40:39.947 DEBUG:boto:Token: None 2022-09-09T22:40:39.947 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:39 GMT /jkcgbj-19/dummy 2022-09-09T22:40:39.948 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TDrM5DDjY+rVrDlM7u9YVjh412Q= 2022-09-09T22:40:39.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:39 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TDrM5DDjY+rVrDlM7u9YVjh412Q=', 'Content-Length': '0'} 2022-09-09T22:40:39.952 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('x-amz-version-id', 'Ux4yCo1Zw0Z6LcOb725cQEb153cMR29'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004bd133d277ff06b6-00631bc0e7-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:39.953 DEBUG:boto:path=/jkcgbj-19/dummy 2022-09-09T22:40:39.953 DEBUG:boto:auth_path=/jkcgbj-19/dummy 2022-09-09T22:40:39.954 DEBUG:boto:Method: HEAD 2022-09-09T22:40:39.954 DEBUG:boto:Path: /jkcgbj-19/dummy 2022-09-09T22:40:39.955 DEBUG:boto:Data: 2022-09-09T22:40:39.955 DEBUG:boto:Headers: {} 2022-09-09T22:40:39.956 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:39.956 DEBUG:boto:Port: 8000 2022-09-09T22:40:39.957 DEBUG:boto:Params: {} 2022-09-09T22:40:39.957 DEBUG:boto:Token: None 2022-09-09T22:40:39.958 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:39 GMT /jkcgbj-19/dummy 2022-09-09T22:40:39.958 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TDrM5DDjY+rVrDlM7u9YVjh412Q= 2022-09-09T22:40:39.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:39 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TDrM5DDjY+rVrDlM7u9YVjh412Q=', 'Content-Length': '0'} 2022-09-09T22:40:39.969 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('x-amz-version-id', 'Ux4yCo1Zw0Z6LcOb725cQEb153cMR29'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004582dc6a8e9f842e-00631bc0e7-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:39.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:40:39.970 DEBUG:boto:path=/jkcgbj-19/dummy 2022-09-09T22:40:39.971 DEBUG:boto:auth_path=/jkcgbj-19/dummy 2022-09-09T22:40:39.971 DEBUG:boto:path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.972 DEBUG:boto:auth_path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.972 DEBUG:boto:Method: GET 2022-09-09T22:40:39.973 DEBUG:boto:Path: /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.973 DEBUG:boto:Data: 2022-09-09T22:40:39.974 DEBUG:boto:Headers: {} 2022-09-09T22:40:39.974 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:39.975 DEBUG:boto:Port: 8000 2022-09-09T22:40:39.975 DEBUG:boto:Params: {} 2022-09-09T22:40:39.976 DEBUG:boto:Token: None 2022-09-09T22:40:39.977 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:39 GMT /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.977 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:mON28ovblDW8wPIad8vwG2OnmH4= 2022-09-09T22:40:39.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:39 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:mON28ovblDW8wPIad8vwG2OnmH4=', 'Content-Length': '0'} 2022-09-09T22:40:39.981 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('x-amz-version-id', 'Ux4yCo1Zw0Z6LcOb725cQEb153cMR29'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000854c94843824ce6d-00631bc0e7-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:39.982 DEBUG:boto:path=/jkcgbj-19/dummy 2022-09-09T22:40:39.982 DEBUG:boto:auth_path=/jkcgbj-19/dummy 2022-09-09T22:40:39.983 DEBUG:boto:path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.984 DEBUG:boto:auth_path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.984 DEBUG:boto:Method: GET 2022-09-09T22:40:39.985 DEBUG:boto:Path: /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.985 DEBUG:boto:Data: 2022-09-09T22:40:39.986 DEBUG:boto:Headers: {} 2022-09-09T22:40:39.987 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:39.987 DEBUG:boto:Port: 8000 2022-09-09T22:40:39.988 DEBUG:boto:Params: {} 2022-09-09T22:40:39.988 DEBUG:boto:Token: None 2022-09-09T22:40:39.989 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:39 GMT /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:39.989 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:mON28ovblDW8wPIad8vwG2OnmH4= 2022-09-09T22:40:39.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:39 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:mON28ovblDW8wPIad8vwG2OnmH4=', 'Content-Length': '0'} 2022-09-09T22:40:39.993 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('x-amz-version-id', 'Ux4yCo1Zw0Z6LcOb725cQEb153cMR29'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000084fdcd7d5186b116-00631bc0e7-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:39.995 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:39.996 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:39.997 DEBUG:boto:path=/jkcgbj-19/?prefix=dummy 2022-09-09T22:40:39.998 DEBUG:boto:auth_path=/jkcgbj-19/?prefix=dummy 2022-09-09T22:40:39.999 DEBUG:boto:Method: GET 2022-09-09T22:40:40.000 DEBUG:boto:Path: /jkcgbj-19/?prefix=dummy 2022-09-09T22:40:40.000 DEBUG:boto:Data: 2022-09-09T22:40:40.001 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.002 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:40.003 DEBUG:boto:Port: 8000 2022-09-09T22:40:40.004 DEBUG:boto:Params: {} 2022-09-09T22:40:40.004 DEBUG:boto:Token: None 2022-09-09T22:40:40.005 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/ 2022-09-09T22:40:40.006 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:axjiiwtiq6LnRVW1o/AfG7Vbmn0= 2022-09-09T22:40:40.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:axjiiwtiq6LnRVW1o/AfG7Vbmn0=', 'Content-Length': '0'} 2022-09-09T22:40:40.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af4881cbc563a9f4-00631bc0e8-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.013 DEBUG:boto:b'jkcgbj-19dummy1000falsedummy2022-09-09T22:39:43.512Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:40:40.014 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:40.015 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:40.015 DEBUG:boto:path=/jkcgbj-19/?prefix=dummy 2022-09-09T22:40:40.016 DEBUG:boto:auth_path=/jkcgbj-19/?prefix=dummy 2022-09-09T22:40:40.016 DEBUG:boto:Method: GET 2022-09-09T22:40:40.017 DEBUG:boto:Path: /jkcgbj-19/?prefix=dummy 2022-09-09T22:40:40.017 DEBUG:boto:Data: 2022-09-09T22:40:40.018 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.019 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:40.019 DEBUG:boto:Port: 8000 2022-09-09T22:40:40.020 DEBUG:boto:Params: {} 2022-09-09T22:40:40.020 DEBUG:boto:Token: None 2022-09-09T22:40:40.021 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/ 2022-09-09T22:40:40.022 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:axjiiwtiq6LnRVW1o/AfG7Vbmn0= 2022-09-09T22:40:40.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:axjiiwtiq6LnRVW1o/AfG7Vbmn0=', 'Content-Length': '0'} 2022-09-09T22:40:40.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d821bc36bd13fb15-00631bc0e8-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.070 DEBUG:boto:b'jkcgbj-19dummy1000falsedummy2022-09-09T22:39:43.512Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:40:40.071 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-19 zones={test-zone2, test-zone1} 2022-09-09T22:40:40.072 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-19 zones={test-zone3, test-zone1} 2022-09-09T22:40:40.072 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:40.073 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:40.073 DEBUG:boto:Method: HEAD 2022-09-09T22:40:40.074 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:40.074 DEBUG:boto:Data: 2022-09-09T22:40:40.075 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.075 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:40.076 DEBUG:boto:Port: 8001 2022-09-09T22:40:40.076 DEBUG:boto:Params: {} 2022-09-09T22:40:40.077 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-09T22:40:40.077 DEBUG:boto:Token: None 2022-09-09T22:40:40.078 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/ 2022-09-09T22:40:40.079 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rEmwpALPt3PBB+6o7cVMUhVjf50= 2022-09-09T22:40:40.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rEmwpALPt3PBB+6o7cVMUhVjf50=', 'Content-Length': '0'} 2022-09-09T22:40:40.085 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bca3d034dae81d01-00631bc0e8-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.085 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:40.086 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:40.086 DEBUG:boto:Method: HEAD 2022-09-09T22:40:40.087 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:40.087 DEBUG:boto:Data: 2022-09-09T22:40:40.088 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.088 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:40.089 DEBUG:boto:Port: 8000 2022-09-09T22:40:40.089 DEBUG:boto:Params: {} 2022-09-09T22:40:40.090 DEBUG:boto:Token: None 2022-09-09T22:40:40.091 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/ 2022-09-09T22:40:40.091 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rEmwpALPt3PBB+6o7cVMUhVjf50= 2022-09-09T22:40:40.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rEmwpALPt3PBB+6o7cVMUhVjf50=', 'Content-Length': '0'} 2022-09-09T22:40:40.096 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a630541ac515c54-00631bc0e8-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.097 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:40:40.097 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:40.098 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:40.098 DEBUG:boto:path=/jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:40.099 DEBUG:boto:auth_path=/jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:40.099 DEBUG:boto:Method: GET 2022-09-09T22:40:40.100 DEBUG:boto:Path: /jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:40.100 DEBUG:boto:Data: 2022-09-09T22:40:40.101 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.101 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:40.102 DEBUG:boto:Port: 8001 2022-09-09T22:40:40.102 DEBUG:boto:Params: {} 2022-09-09T22:40:40.103 DEBUG:boto:Token: None 2022-09-09T22:40:40.104 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/?versions 2022-09-09T22:40:40.104 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JL/DQGu+mjWi+Z95nm4mFw69GxM= 2022-09-09T22:40:40.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JL/DQGu+mjWi+Z95nm4mFw69GxM=', 'Content-Length': '0'} 2022-09-09T22:40:40.110 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028add4f43639178d-00631bc0e8-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.111 DEBUG:boto:b'jkcgbj-19999falsedummyUx4yCo1Zw0Z6LcOb725cQEb153cMR29true2022-09-09T22:39:43.512Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:40:40.111 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:40:40.112 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:40:40.113 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:40.113 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:40.114 DEBUG:boto:path=/jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:40.114 DEBUG:boto:auth_path=/jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:40.115 DEBUG:boto:Method: GET 2022-09-09T22:40:40.115 DEBUG:boto:Path: /jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:40.116 DEBUG:boto:Data: 2022-09-09T22:40:40.116 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.117 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:40.117 DEBUG:boto:Port: 8000 2022-09-09T22:40:40.118 DEBUG:boto:Params: {} 2022-09-09T22:40:40.118 DEBUG:boto:Token: None 2022-09-09T22:40:40.119 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/?versions 2022-09-09T22:40:40.119 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JL/DQGu+mjWi+Z95nm4mFw69GxM= 2022-09-09T22:40:40.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JL/DQGu+mjWi+Z95nm4mFw69GxM=', 'Content-Length': '0'} 2022-09-09T22:40:40.125 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b489d046b56d4fe6-00631bc0e8-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.126 DEBUG:boto:b'jkcgbj-19999falsedummyUx4yCo1Zw0Z6LcOb725cQEb153cMR29true2022-09-09T22:39:43.512Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:40:40.126 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:40:40.127 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:40.127 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:40.128 DEBUG:boto:path=/jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:40.128 DEBUG:boto:auth_path=/jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:40.129 DEBUG:boto:Method: GET 2022-09-09T22:40:40.129 DEBUG:boto:Path: /jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:40.130 DEBUG:boto:Data: 2022-09-09T22:40:40.130 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.131 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:40.131 DEBUG:boto:Port: 8001 2022-09-09T22:40:40.132 DEBUG:boto:Params: {} 2022-09-09T22:40:40.133 DEBUG:boto:Token: None 2022-09-09T22:40:40.133 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/?versions 2022-09-09T22:40:40.134 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JL/DQGu+mjWi+Z95nm4mFw69GxM= 2022-09-09T22:40:40.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JL/DQGu+mjWi+Z95nm4mFw69GxM=', 'Content-Length': '0'} 2022-09-09T22:40:40.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1a22578b20f37e8-00631bc0e8-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.140 DEBUG:boto:b'jkcgbj-19999falsedummyUx4yCo1Zw0Z6LcOb725cQEb153cMR29true2022-09-09T22:39:43.512Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:40:40.141 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:40.141 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:40.142 DEBUG:boto:path=/jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:40.142 DEBUG:boto:auth_path=/jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:40.143 DEBUG:boto:Method: GET 2022-09-09T22:40:40.143 DEBUG:boto:Path: /jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:40.144 DEBUG:boto:Data: 2022-09-09T22:40:40.144 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.145 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:40.145 DEBUG:boto:Port: 8000 2022-09-09T22:40:40.146 DEBUG:boto:Params: {} 2022-09-09T22:40:40.146 DEBUG:boto:Token: None 2022-09-09T22:40:40.147 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/?versions 2022-09-09T22:40:40.148 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JL/DQGu+mjWi+Z95nm4mFw69GxM= 2022-09-09T22:40:40.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JL/DQGu+mjWi+Z95nm4mFw69GxM=', 'Content-Length': '0'} 2022-09-09T22:40:40.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5d3a3fd3c1a956a-00631bc0e8-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.194 DEBUG:boto:b'jkcgbj-19999falsedummyUx4yCo1Zw0Z6LcOb725cQEb153cMR29true2022-09-09T22:39:43.512Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:40:40.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:40:40.196 DEBUG:boto:path=/jkcgbj-19/dummy 2022-09-09T22:40:40.196 DEBUG:boto:auth_path=/jkcgbj-19/dummy 2022-09-09T22:40:40.197 DEBUG:boto:path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.197 DEBUG:boto:auth_path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.198 DEBUG:boto:Method: GET 2022-09-09T22:40:40.198 DEBUG:boto:Path: /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.199 DEBUG:boto:Data: 2022-09-09T22:40:40.199 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.200 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:40.200 DEBUG:boto:Port: 8001 2022-09-09T22:40:40.201 DEBUG:boto:Params: {} 2022-09-09T22:40:40.202 DEBUG:boto:Token: None 2022-09-09T22:40:40.202 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.203 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:26EcXc7XMqNtbQXOFqPpS3uYWv0= 2022-09-09T22:40:40.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:26EcXc7XMqNtbQXOFqPpS3uYWv0=', 'Content-Length': '0'} 2022-09-09T22:40:40.207 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('x-amz-version-id', 'Ux4yCo1Zw0Z6LcOb725cQEb153cMR29'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cfdccb98120507a3-00631bc0e8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.207 DEBUG:boto:path=/jkcgbj-19/dummy 2022-09-09T22:40:40.208 DEBUG:boto:auth_path=/jkcgbj-19/dummy 2022-09-09T22:40:40.208 DEBUG:boto:path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.209 DEBUG:boto:auth_path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.210 DEBUG:boto:Method: GET 2022-09-09T22:40:40.210 DEBUG:boto:Path: /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.211 DEBUG:boto:Data: 2022-09-09T22:40:40.211 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.212 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:40.212 DEBUG:boto:Port: 8000 2022-09-09T22:40:40.213 DEBUG:boto:Params: {} 2022-09-09T22:40:40.213 DEBUG:boto:Token: None 2022-09-09T22:40:40.214 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.215 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:26EcXc7XMqNtbQXOFqPpS3uYWv0= 2022-09-09T22:40:40.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:26EcXc7XMqNtbQXOFqPpS3uYWv0=', 'Content-Length': '0'} 2022-09-09T22:40:40.218 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('x-amz-version-id', 'Ux4yCo1Zw0Z6LcOb725cQEb153cMR29'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000096a51a30082287f0-00631bc0e8-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.219 DEBUG:boto:path=/jkcgbj-19/dummy 2022-09-09T22:40:40.219 DEBUG:boto:auth_path=/jkcgbj-19/dummy 2022-09-09T22:40:40.220 DEBUG:boto:path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.220 DEBUG:boto:auth_path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.221 DEBUG:boto:Method: HEAD 2022-09-09T22:40:40.221 DEBUG:boto:Path: /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.222 DEBUG:boto:Data: 2022-09-09T22:40:40.222 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.223 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:40.224 DEBUG:boto:Port: 8001 2022-09-09T22:40:40.224 DEBUG:boto:Params: {} 2022-09-09T22:40:40.225 DEBUG:boto:Token: None 2022-09-09T22:40:40.225 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.226 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0qzIywn2B/jU1Tf4uL2tIefpup0= 2022-09-09T22:40:40.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0qzIywn2B/jU1Tf4uL2tIefpup0=', 'Content-Length': '0'} 2022-09-09T22:40:40.229 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('x-amz-version-id', 'Ux4yCo1Zw0Z6LcOb725cQEb153cMR29'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b7ca043980916ed4-00631bc0e8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.230 DEBUG:boto:path=/jkcgbj-19/dummy 2022-09-09T22:40:40.230 DEBUG:boto:auth_path=/jkcgbj-19/dummy 2022-09-09T22:40:40.231 DEBUG:boto:path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.231 DEBUG:boto:auth_path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.232 DEBUG:boto:Method: HEAD 2022-09-09T22:40:40.232 DEBUG:boto:Path: /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.233 DEBUG:boto:Data: 2022-09-09T22:40:40.234 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.234 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:40.235 DEBUG:boto:Port: 8000 2022-09-09T22:40:40.235 DEBUG:boto:Params: {} 2022-09-09T22:40:40.236 DEBUG:boto:Token: None 2022-09-09T22:40:40.237 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.237 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0qzIywn2B/jU1Tf4uL2tIefpup0= 2022-09-09T22:40:40.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0qzIywn2B/jU1Tf4uL2tIefpup0=', 'Content-Length': '0'} 2022-09-09T22:40:40.241 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('x-amz-version-id', 'Ux4yCo1Zw0Z6LcOb725cQEb153cMR29'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003524b385a479eee5-00631bc0e8-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:40:40.242 DEBUG:boto:path=/jkcgbj-19/dummy 2022-09-09T22:40:40.243 DEBUG:boto:auth_path=/jkcgbj-19/dummy 2022-09-09T22:40:40.244 DEBUG:boto:path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.244 DEBUG:boto:auth_path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.245 DEBUG:boto:Method: GET 2022-09-09T22:40:40.246 DEBUG:boto:Path: /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.246 DEBUG:boto:Data: 2022-09-09T22:40:40.247 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.248 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:40.248 DEBUG:boto:Port: 8001 2022-09-09T22:40:40.249 DEBUG:boto:Params: {} 2022-09-09T22:40:40.250 DEBUG:boto:Token: None 2022-09-09T22:40:40.250 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.251 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:26EcXc7XMqNtbQXOFqPpS3uYWv0= 2022-09-09T22:40:40.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:26EcXc7XMqNtbQXOFqPpS3uYWv0=', 'Content-Length': '0'} 2022-09-09T22:40:40.254 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('x-amz-version-id', 'Ux4yCo1Zw0Z6LcOb725cQEb153cMR29'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d6437e1ff9104184-00631bc0e8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.255 DEBUG:boto:path=/jkcgbj-19/dummy 2022-09-09T22:40:40.256 DEBUG:boto:auth_path=/jkcgbj-19/dummy 2022-09-09T22:40:40.256 DEBUG:boto:path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.257 DEBUG:boto:auth_path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.257 DEBUG:boto:Method: GET 2022-09-09T22:40:40.258 DEBUG:boto:Path: /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.258 DEBUG:boto:Data: 2022-09-09T22:40:40.259 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.260 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:40.260 DEBUG:boto:Port: 8000 2022-09-09T22:40:40.261 DEBUG:boto:Params: {} 2022-09-09T22:40:40.261 DEBUG:boto:Token: None 2022-09-09T22:40:40.262 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.262 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:26EcXc7XMqNtbQXOFqPpS3uYWv0= 2022-09-09T22:40:40.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:26EcXc7XMqNtbQXOFqPpS3uYWv0=', 'Content-Length': '0'} 2022-09-09T22:40:40.266 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('x-amz-version-id', 'Ux4yCo1Zw0Z6LcOb725cQEb153cMR29'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f5ffd65286a3f9d0-00631bc0e8-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.267 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:40.268 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:40.268 DEBUG:boto:path=/jkcgbj-19/?prefix=dummy 2022-09-09T22:40:40.269 DEBUG:boto:auth_path=/jkcgbj-19/?prefix=dummy 2022-09-09T22:40:40.269 DEBUG:boto:Method: GET 2022-09-09T22:40:40.270 DEBUG:boto:Path: /jkcgbj-19/?prefix=dummy 2022-09-09T22:40:40.270 DEBUG:boto:Data: 2022-09-09T22:40:40.271 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.271 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:40.272 DEBUG:boto:Port: 8001 2022-09-09T22:40:40.272 DEBUG:boto:Params: {} 2022-09-09T22:40:40.273 DEBUG:boto:Token: None 2022-09-09T22:40:40.274 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/ 2022-09-09T22:40:40.274 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:axjiiwtiq6LnRVW1o/AfG7Vbmn0= 2022-09-09T22:40:40.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:axjiiwtiq6LnRVW1o/AfG7Vbmn0=', 'Content-Length': '0'} 2022-09-09T22:40:40.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032b93ed5f4b4632f-00631bc0e8-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.326 DEBUG:boto:b'jkcgbj-19dummy1000falsedummy2022-09-09T22:39:43.512Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:40:40.327 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:40.328 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:40.329 DEBUG:boto:path=/jkcgbj-19/?prefix=dummy 2022-09-09T22:40:40.329 DEBUG:boto:auth_path=/jkcgbj-19/?prefix=dummy 2022-09-09T22:40:40.330 DEBUG:boto:Method: GET 2022-09-09T22:40:40.330 DEBUG:boto:Path: /jkcgbj-19/?prefix=dummy 2022-09-09T22:40:40.331 DEBUG:boto:Data: 2022-09-09T22:40:40.331 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.332 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:40.333 DEBUG:boto:Port: 8000 2022-09-09T22:40:40.333 DEBUG:boto:Params: {} 2022-09-09T22:40:40.334 DEBUG:boto:Token: None 2022-09-09T22:40:40.334 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/ 2022-09-09T22:40:40.335 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:axjiiwtiq6LnRVW1o/AfG7Vbmn0= 2022-09-09T22:40:40.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:axjiiwtiq6LnRVW1o/AfG7Vbmn0=', 'Content-Length': '0'} 2022-09-09T22:40:40.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebe59963ade0055d-00631bc0e8-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.342 DEBUG:boto:b'jkcgbj-19dummy1000falsedummy2022-09-09T22:39:43.512Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:40:40.343 DEBUG:boto:path=/jkcgbj-19/dummy 2022-09-09T22:40:40.343 DEBUG:boto:auth_path=/jkcgbj-19/dummy 2022-09-09T22:40:40.344 DEBUG:boto:Method: HEAD 2022-09-09T22:40:40.344 DEBUG:boto:Path: /jkcgbj-19/dummy 2022-09-09T22:40:40.345 DEBUG:boto:Data: 2022-09-09T22:40:40.346 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.346 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:40.347 DEBUG:boto:Port: 8001 2022-09-09T22:40:40.347 DEBUG:boto:Params: {} 2022-09-09T22:40:40.348 DEBUG:boto:Token: None 2022-09-09T22:40:40.348 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/dummy 2022-09-09T22:40:40.349 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jazPL921vp3U50N1/6k/qsrgwZA= 2022-09-09T22:40:40.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jazPL921vp3U50N1/6k/qsrgwZA=', 'Content-Length': '0'} 2022-09-09T22:40:40.353 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('x-amz-version-id', 'Ux4yCo1Zw0Z6LcOb725cQEb153cMR29'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000052644e40250ab99b-00631bc0e8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.354 DEBUG:boto:path=/jkcgbj-19/dummy 2022-09-09T22:40:40.355 DEBUG:boto:auth_path=/jkcgbj-19/dummy 2022-09-09T22:40:40.356 DEBUG:boto:Method: HEAD 2022-09-09T22:40:40.357 DEBUG:boto:Path: /jkcgbj-19/dummy 2022-09-09T22:40:40.358 DEBUG:boto:Data: 2022-09-09T22:40:40.359 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.359 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:40.360 DEBUG:boto:Port: 8000 2022-09-09T22:40:40.361 DEBUG:boto:Params: {} 2022-09-09T22:40:40.362 DEBUG:boto:Token: None 2022-09-09T22:40:40.363 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/dummy 2022-09-09T22:40:40.364 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jazPL921vp3U50N1/6k/qsrgwZA= 2022-09-09T22:40:40.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jazPL921vp3U50N1/6k/qsrgwZA=', 'Content-Length': '0'} 2022-09-09T22:40:40.370 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('x-amz-version-id', 'Ux4yCo1Zw0Z6LcOb725cQEb153cMR29'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c487a5e5b91dbeea-00631bc0e8-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:40:40.372 DEBUG:boto:path=/jkcgbj-19/dummy 2022-09-09T22:40:40.373 DEBUG:boto:auth_path=/jkcgbj-19/dummy 2022-09-09T22:40:40.374 DEBUG:boto:path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.375 DEBUG:boto:auth_path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.376 DEBUG:boto:Method: GET 2022-09-09T22:40:40.377 DEBUG:boto:Path: /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.378 DEBUG:boto:Data: 2022-09-09T22:40:40.379 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.380 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:40.381 DEBUG:boto:Port: 8001 2022-09-09T22:40:40.382 DEBUG:boto:Params: {} 2022-09-09T22:40:40.383 DEBUG:boto:Token: None 2022-09-09T22:40:40.384 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.385 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:26EcXc7XMqNtbQXOFqPpS3uYWv0= 2022-09-09T22:40:40.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:26EcXc7XMqNtbQXOFqPpS3uYWv0=', 'Content-Length': '0'} 2022-09-09T22:40:40.389 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('x-amz-version-id', 'Ux4yCo1Zw0Z6LcOb725cQEb153cMR29'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c7bce93167f31a2d-00631bc0e8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.390 DEBUG:boto:path=/jkcgbj-19/dummy 2022-09-09T22:40:40.391 DEBUG:boto:auth_path=/jkcgbj-19/dummy 2022-09-09T22:40:40.391 DEBUG:boto:path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.392 DEBUG:boto:auth_path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.392 DEBUG:boto:Method: GET 2022-09-09T22:40:40.393 DEBUG:boto:Path: /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.393 DEBUG:boto:Data: 2022-09-09T22:40:40.394 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.394 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:40.395 DEBUG:boto:Port: 8000 2022-09-09T22:40:40.396 DEBUG:boto:Params: {} 2022-09-09T22:40:40.396 DEBUG:boto:Token: None 2022-09-09T22:40:40.397 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.397 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:26EcXc7XMqNtbQXOFqPpS3uYWv0= 2022-09-09T22:40:40.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:26EcXc7XMqNtbQXOFqPpS3uYWv0=', 'Content-Length': '0'} 2022-09-09T22:40:40.401 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('x-amz-version-id', 'Ux4yCo1Zw0Z6LcOb725cQEb153cMR29'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ddff0f8430aebe00-00631bc0e8-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.402 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:40.402 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:40.403 DEBUG:boto:path=/jkcgbj-19/?prefix=dummy 2022-09-09T22:40:40.403 DEBUG:boto:auth_path=/jkcgbj-19/?prefix=dummy 2022-09-09T22:40:40.404 DEBUG:boto:Method: GET 2022-09-09T22:40:40.404 DEBUG:boto:Path: /jkcgbj-19/?prefix=dummy 2022-09-09T22:40:40.405 DEBUG:boto:Data: 2022-09-09T22:40:40.405 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.406 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:40.406 DEBUG:boto:Port: 8001 2022-09-09T22:40:40.407 DEBUG:boto:Params: {} 2022-09-09T22:40:40.408 DEBUG:boto:Token: None 2022-09-09T22:40:40.408 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/ 2022-09-09T22:40:40.409 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:axjiiwtiq6LnRVW1o/AfG7Vbmn0= 2022-09-09T22:40:40.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:axjiiwtiq6LnRVW1o/AfG7Vbmn0=', 'Content-Length': '0'} 2022-09-09T22:40:40.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000265dc4f712e5ff0c-00631bc0e8-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.417 DEBUG:boto:b'jkcgbj-19dummy1000falsedummy2022-09-09T22:39:43.512Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:40:40.418 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:40.418 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:40.419 DEBUG:boto:path=/jkcgbj-19/?prefix=dummy 2022-09-09T22:40:40.419 DEBUG:boto:auth_path=/jkcgbj-19/?prefix=dummy 2022-09-09T22:40:40.420 DEBUG:boto:Method: GET 2022-09-09T22:40:40.420 DEBUG:boto:Path: /jkcgbj-19/?prefix=dummy 2022-09-09T22:40:40.421 DEBUG:boto:Data: 2022-09-09T22:40:40.421 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.422 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:40.423 DEBUG:boto:Port: 8000 2022-09-09T22:40:40.423 DEBUG:boto:Params: {} 2022-09-09T22:40:40.424 DEBUG:boto:Token: None 2022-09-09T22:40:40.424 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/ 2022-09-09T22:40:40.425 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:axjiiwtiq6LnRVW1o/AfG7Vbmn0= 2022-09-09T22:40:40.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:axjiiwtiq6LnRVW1o/AfG7Vbmn0=', 'Content-Length': '0'} 2022-09-09T22:40:40.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd4e0798e674f0b7-00631bc0e8-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.474 DEBUG:boto:b'jkcgbj-19dummy1000falsedummy2022-09-09T22:39:43.512Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:40:40.475 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-19 zones={test-zone3, test-zone1} 2022-09-09T22:40:40.476 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-19 zones={test-zone3, test-zone2} 2022-09-09T22:40:40.477 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:40.477 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:40.478 DEBUG:boto:Method: HEAD 2022-09-09T22:40:40.478 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:40.479 DEBUG:boto:Data: 2022-09-09T22:40:40.479 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.480 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:40.480 DEBUG:boto:Port: 8001 2022-09-09T22:40:40.481 DEBUG:boto:Params: {} 2022-09-09T22:40:40.481 DEBUG:boto:Token: None 2022-09-09T22:40:40.482 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/ 2022-09-09T22:40:40.483 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rEmwpALPt3PBB+6o7cVMUhVjf50= 2022-09-09T22:40:40.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rEmwpALPt3PBB+6o7cVMUhVjf50=', 'Content-Length': '0'} 2022-09-09T22:40:40.490 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000731e3c05bf762a0f-00631bc0e8-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.491 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:40.492 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:40.493 DEBUG:boto:Method: HEAD 2022-09-09T22:40:40.494 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:40.495 DEBUG:boto:Data: 2022-09-09T22:40:40.495 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.496 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:40.497 DEBUG:boto:Port: 8000 2022-09-09T22:40:40.498 DEBUG:boto:Params: {} 2022-09-09T22:40:40.499 DEBUG:boto:Token: None 2022-09-09T22:40:40.500 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/ 2022-09-09T22:40:40.501 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rEmwpALPt3PBB+6o7cVMUhVjf50= 2022-09-09T22:40:40.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rEmwpALPt3PBB+6o7cVMUhVjf50=', 'Content-Length': '0'} 2022-09-09T22:40:40.507 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000063f35600e95acc68-00631bc0e8-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.508 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:40:40.509 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:40.510 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:40.511 DEBUG:boto:path=/jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:40.511 DEBUG:boto:auth_path=/jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:40.512 DEBUG:boto:Method: GET 2022-09-09T22:40:40.513 DEBUG:boto:Path: /jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:40.513 DEBUG:boto:Data: 2022-09-09T22:40:40.514 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.515 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:40.516 DEBUG:boto:Port: 8001 2022-09-09T22:40:40.517 DEBUG:boto:Params: {} 2022-09-09T22:40:40.518 DEBUG:boto:Token: None 2022-09-09T22:40:40.519 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/?versions 2022-09-09T22:40:40.520 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JL/DQGu+mjWi+Z95nm4mFw69GxM= 2022-09-09T22:40:40.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JL/DQGu+mjWi+Z95nm4mFw69GxM=', 'Content-Length': '0'} 2022-09-09T22:40:40.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae7ed0c5d5806b69-00631bc0e8-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.574 DEBUG:boto:b'jkcgbj-19999falsedummyUx4yCo1Zw0Z6LcOb725cQEb153cMR29true2022-09-09T22:39:43.512Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:40:40.575 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:40:40.576 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:40:40.576 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:40.577 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:40.577 DEBUG:boto:path=/jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:40.578 DEBUG:boto:auth_path=/jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:40.578 DEBUG:boto:Method: GET 2022-09-09T22:40:40.579 DEBUG:boto:Path: /jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:40.579 DEBUG:boto:Data: 2022-09-09T22:40:40.580 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.581 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:40.581 DEBUG:boto:Port: 8000 2022-09-09T22:40:40.582 DEBUG:boto:Params: {} 2022-09-09T22:40:40.582 DEBUG:boto:Token: None 2022-09-09T22:40:40.583 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/?versions 2022-09-09T22:40:40.583 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JL/DQGu+mjWi+Z95nm4mFw69GxM= 2022-09-09T22:40:40.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JL/DQGu+mjWi+Z95nm4mFw69GxM=', 'Content-Length': '0'} 2022-09-09T22:40:40.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f0d8965e73b582c-00631bc0e8-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.589 DEBUG:boto:b'jkcgbj-19999falsedummyUx4yCo1Zw0Z6LcOb725cQEb153cMR29true2022-09-09T22:39:43.512Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:40:40.590 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-09T22:40:40.590 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:40.591 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:40.591 DEBUG:boto:path=/jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:40.592 DEBUG:boto:auth_path=/jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:40.592 DEBUG:boto:Method: GET 2022-09-09T22:40:40.593 DEBUG:boto:Path: /jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:40.593 DEBUG:boto:Data: 2022-09-09T22:40:40.594 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.594 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:40.595 DEBUG:boto:Port: 8001 2022-09-09T22:40:40.595 DEBUG:boto:Params: {} 2022-09-09T22:40:40.596 DEBUG:boto:Token: None 2022-09-09T22:40:40.597 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/?versions 2022-09-09T22:40:40.597 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JL/DQGu+mjWi+Z95nm4mFw69GxM= 2022-09-09T22:40:40.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JL/DQGu+mjWi+Z95nm4mFw69GxM=', 'Content-Length': '0'} 2022-09-09T22:40:40.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000342990d8592360e2-00631bc0e8-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.604 DEBUG:boto:b'jkcgbj-19999falsedummyUx4yCo1Zw0Z6LcOb725cQEb153cMR29true2022-09-09T22:39:43.512Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:40:40.605 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:40.606 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:40.606 DEBUG:boto:path=/jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:40.607 DEBUG:boto:auth_path=/jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:40.607 DEBUG:boto:Method: GET 2022-09-09T22:40:40.608 DEBUG:boto:Path: /jkcgbj-19/?versions&max-keys=999 2022-09-09T22:40:40.608 DEBUG:boto:Data: 2022-09-09T22:40:40.609 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.609 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:40.610 DEBUG:boto:Port: 8000 2022-09-09T22:40:40.610 DEBUG:boto:Params: {} 2022-09-09T22:40:40.611 DEBUG:boto:Token: None 2022-09-09T22:40:40.612 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/?versions 2022-09-09T22:40:40.612 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JL/DQGu+mjWi+Z95nm4mFw69GxM= 2022-09-09T22:40:40.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JL/DQGu+mjWi+Z95nm4mFw69GxM=', 'Content-Length': '0'} 2022-09-09T22:40:40.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001553b4737f899b69-00631bc0e8-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.666 DEBUG:boto:b'jkcgbj-19999falsedummyUx4yCo1Zw0Z6LcOb725cQEb153cMR29true2022-09-09T22:39:43.512Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:40:40.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:40:40.668 DEBUG:boto:path=/jkcgbj-19/dummy 2022-09-09T22:40:40.668 DEBUG:boto:auth_path=/jkcgbj-19/dummy 2022-09-09T22:40:40.669 DEBUG:boto:path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.669 DEBUG:boto:auth_path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.670 DEBUG:boto:Method: GET 2022-09-09T22:40:40.670 DEBUG:boto:Path: /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.671 DEBUG:boto:Data: 2022-09-09T22:40:40.671 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.672 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:40.672 DEBUG:boto:Port: 8001 2022-09-09T22:40:40.673 DEBUG:boto:Params: {} 2022-09-09T22:40:40.674 DEBUG:boto:Token: None 2022-09-09T22:40:40.674 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.675 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:26EcXc7XMqNtbQXOFqPpS3uYWv0= 2022-09-09T22:40:40.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:26EcXc7XMqNtbQXOFqPpS3uYWv0=', 'Content-Length': '0'} 2022-09-09T22:40:40.678 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('x-amz-version-id', 'Ux4yCo1Zw0Z6LcOb725cQEb153cMR29'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004d39bf72dfaa860d-00631bc0e8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.679 DEBUG:boto:path=/jkcgbj-19/dummy 2022-09-09T22:40:40.680 DEBUG:boto:auth_path=/jkcgbj-19/dummy 2022-09-09T22:40:40.680 DEBUG:boto:path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.681 DEBUG:boto:auth_path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.681 DEBUG:boto:Method: GET 2022-09-09T22:40:40.682 DEBUG:boto:Path: /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.682 DEBUG:boto:Data: 2022-09-09T22:40:40.683 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.683 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:40.684 DEBUG:boto:Port: 8000 2022-09-09T22:40:40.684 DEBUG:boto:Params: {} 2022-09-09T22:40:40.685 DEBUG:boto:Token: None 2022-09-09T22:40:40.686 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.686 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:26EcXc7XMqNtbQXOFqPpS3uYWv0= 2022-09-09T22:40:40.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:26EcXc7XMqNtbQXOFqPpS3uYWv0=', 'Content-Length': '0'} 2022-09-09T22:40:40.690 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('x-amz-version-id', 'Ux4yCo1Zw0Z6LcOb725cQEb153cMR29'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ce8cf27254d210f4-00631bc0e8-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.691 DEBUG:boto:path=/jkcgbj-19/dummy 2022-09-09T22:40:40.691 DEBUG:boto:auth_path=/jkcgbj-19/dummy 2022-09-09T22:40:40.692 DEBUG:boto:path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.692 DEBUG:boto:auth_path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.693 DEBUG:boto:Method: HEAD 2022-09-09T22:40:40.693 DEBUG:boto:Path: /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.694 DEBUG:boto:Data: 2022-09-09T22:40:40.694 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.695 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:40.695 DEBUG:boto:Port: 8001 2022-09-09T22:40:40.696 DEBUG:boto:Params: {} 2022-09-09T22:40:40.696 DEBUG:boto:Token: None 2022-09-09T22:40:40.697 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.697 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0qzIywn2B/jU1Tf4uL2tIefpup0= 2022-09-09T22:40:40.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0qzIywn2B/jU1Tf4uL2tIefpup0=', 'Content-Length': '0'} 2022-09-09T22:40:40.700 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('x-amz-version-id', 'Ux4yCo1Zw0Z6LcOb725cQEb153cMR29'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ae59b8fbcf9b2cdf-00631bc0e8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.701 DEBUG:boto:path=/jkcgbj-19/dummy 2022-09-09T22:40:40.702 DEBUG:boto:auth_path=/jkcgbj-19/dummy 2022-09-09T22:40:40.702 DEBUG:boto:path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.703 DEBUG:boto:auth_path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.703 DEBUG:boto:Method: HEAD 2022-09-09T22:40:40.704 DEBUG:boto:Path: /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.704 DEBUG:boto:Data: 2022-09-09T22:40:40.705 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.705 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:40.706 DEBUG:boto:Port: 8000 2022-09-09T22:40:40.706 DEBUG:boto:Params: {} 2022-09-09T22:40:40.707 DEBUG:boto:Token: None 2022-09-09T22:40:40.708 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.708 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0qzIywn2B/jU1Tf4uL2tIefpup0= 2022-09-09T22:40:40.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0qzIywn2B/jU1Tf4uL2tIefpup0=', 'Content-Length': '0'} 2022-09-09T22:40:40.712 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('x-amz-version-id', 'Ux4yCo1Zw0Z6LcOb725cQEb153cMR29'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009696183e55cd6392-00631bc0e8-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:40:40.713 DEBUG:boto:path=/jkcgbj-19/dummy 2022-09-09T22:40:40.714 DEBUG:boto:auth_path=/jkcgbj-19/dummy 2022-09-09T22:40:40.714 DEBUG:boto:path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.715 DEBUG:boto:auth_path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.715 DEBUG:boto:Method: GET 2022-09-09T22:40:40.716 DEBUG:boto:Path: /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.716 DEBUG:boto:Data: 2022-09-09T22:40:40.717 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.717 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:40.718 DEBUG:boto:Port: 8001 2022-09-09T22:40:40.718 DEBUG:boto:Params: {} 2022-09-09T22:40:40.719 DEBUG:boto:Token: None 2022-09-09T22:40:40.720 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.720 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:26EcXc7XMqNtbQXOFqPpS3uYWv0= 2022-09-09T22:40:40.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:26EcXc7XMqNtbQXOFqPpS3uYWv0=', 'Content-Length': '0'} 2022-09-09T22:40:40.724 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('x-amz-version-id', 'Ux4yCo1Zw0Z6LcOb725cQEb153cMR29'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bee50c60e38e0809-00631bc0e8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.724 DEBUG:boto:path=/jkcgbj-19/dummy 2022-09-09T22:40:40.725 DEBUG:boto:auth_path=/jkcgbj-19/dummy 2022-09-09T22:40:40.725 DEBUG:boto:path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.726 DEBUG:boto:auth_path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.726 DEBUG:boto:Method: GET 2022-09-09T22:40:40.727 DEBUG:boto:Path: /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.727 DEBUG:boto:Data: 2022-09-09T22:40:40.728 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.728 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:40.729 DEBUG:boto:Port: 8000 2022-09-09T22:40:40.729 DEBUG:boto:Params: {} 2022-09-09T22:40:40.730 DEBUG:boto:Token: None 2022-09-09T22:40:40.731 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.731 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:26EcXc7XMqNtbQXOFqPpS3uYWv0= 2022-09-09T22:40:40.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:26EcXc7XMqNtbQXOFqPpS3uYWv0=', 'Content-Length': '0'} 2022-09-09T22:40:40.735 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('x-amz-version-id', 'Ux4yCo1Zw0Z6LcOb725cQEb153cMR29'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000204ab25d8ede0bf2-00631bc0e8-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.736 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:40.736 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:40.737 DEBUG:boto:path=/jkcgbj-19/?prefix=dummy 2022-09-09T22:40:40.737 DEBUG:boto:auth_path=/jkcgbj-19/?prefix=dummy 2022-09-09T22:40:40.738 DEBUG:boto:Method: GET 2022-09-09T22:40:40.738 DEBUG:boto:Path: /jkcgbj-19/?prefix=dummy 2022-09-09T22:40:40.739 DEBUG:boto:Data: 2022-09-09T22:40:40.739 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.740 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:40.740 DEBUG:boto:Port: 8001 2022-09-09T22:40:40.741 DEBUG:boto:Params: {} 2022-09-09T22:40:40.741 DEBUG:boto:Token: None 2022-09-09T22:40:40.742 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/ 2022-09-09T22:40:40.742 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:axjiiwtiq6LnRVW1o/AfG7Vbmn0= 2022-09-09T22:40:40.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:axjiiwtiq6LnRVW1o/AfG7Vbmn0=', 'Content-Length': '0'} 2022-09-09T22:40:40.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006fe2166108d224e-00631bc0e8-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.794 DEBUG:boto:b'jkcgbj-19dummy1000falsedummy2022-09-09T22:39:43.512Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:40:40.795 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:40.796 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:40.797 DEBUG:boto:path=/jkcgbj-19/?prefix=dummy 2022-09-09T22:40:40.797 DEBUG:boto:auth_path=/jkcgbj-19/?prefix=dummy 2022-09-09T22:40:40.798 DEBUG:boto:Method: GET 2022-09-09T22:40:40.798 DEBUG:boto:Path: /jkcgbj-19/?prefix=dummy 2022-09-09T22:40:40.799 DEBUG:boto:Data: 2022-09-09T22:40:40.799 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.800 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:40.800 DEBUG:boto:Port: 8000 2022-09-09T22:40:40.801 DEBUG:boto:Params: {} 2022-09-09T22:40:40.801 DEBUG:boto:Token: None 2022-09-09T22:40:40.802 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/ 2022-09-09T22:40:40.803 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:axjiiwtiq6LnRVW1o/AfG7Vbmn0= 2022-09-09T22:40:40.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:axjiiwtiq6LnRVW1o/AfG7Vbmn0=', 'Content-Length': '0'} 2022-09-09T22:40:40.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e51554ce22a7699-00631bc0e8-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.809 DEBUG:boto:b'jkcgbj-19dummy1000falsedummy2022-09-09T22:39:43.512Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:40:40.810 DEBUG:boto:path=/jkcgbj-19/dummy 2022-09-09T22:40:40.811 DEBUG:boto:auth_path=/jkcgbj-19/dummy 2022-09-09T22:40:40.811 DEBUG:boto:Method: HEAD 2022-09-09T22:40:40.812 DEBUG:boto:Path: /jkcgbj-19/dummy 2022-09-09T22:40:40.812 DEBUG:boto:Data: 2022-09-09T22:40:40.813 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.813 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:40.814 DEBUG:boto:Port: 8001 2022-09-09T22:40:40.814 DEBUG:boto:Params: {} 2022-09-09T22:40:40.815 DEBUG:boto:Token: None 2022-09-09T22:40:40.815 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/dummy 2022-09-09T22:40:40.816 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jazPL921vp3U50N1/6k/qsrgwZA= 2022-09-09T22:40:40.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jazPL921vp3U50N1/6k/qsrgwZA=', 'Content-Length': '0'} 2022-09-09T22:40:40.820 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('x-amz-version-id', 'Ux4yCo1Zw0Z6LcOb725cQEb153cMR29'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000055aa0452435cd5d0-00631bc0e8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.821 DEBUG:boto:path=/jkcgbj-19/dummy 2022-09-09T22:40:40.822 DEBUG:boto:auth_path=/jkcgbj-19/dummy 2022-09-09T22:40:40.822 DEBUG:boto:Method: HEAD 2022-09-09T22:40:40.823 DEBUG:boto:Path: /jkcgbj-19/dummy 2022-09-09T22:40:40.823 DEBUG:boto:Data: 2022-09-09T22:40:40.824 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.824 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:40.825 DEBUG:boto:Port: 8000 2022-09-09T22:40:40.825 DEBUG:boto:Params: {} 2022-09-09T22:40:40.826 DEBUG:boto:Token: None 2022-09-09T22:40:40.827 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/dummy 2022-09-09T22:40:40.827 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jazPL921vp3U50N1/6k/qsrgwZA= 2022-09-09T22:40:40.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jazPL921vp3U50N1/6k/qsrgwZA=', 'Content-Length': '0'} 2022-09-09T22:40:40.832 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('x-amz-version-id', 'Ux4yCo1Zw0Z6LcOb725cQEb153cMR29'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000038b61f24169529df-00631bc0e8-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-09T22:40:40.833 DEBUG:boto:path=/jkcgbj-19/dummy 2022-09-09T22:40:40.833 DEBUG:boto:auth_path=/jkcgbj-19/dummy 2022-09-09T22:40:40.834 DEBUG:boto:path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.834 DEBUG:boto:auth_path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.835 DEBUG:boto:Method: GET 2022-09-09T22:40:40.835 DEBUG:boto:Path: /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.836 DEBUG:boto:Data: 2022-09-09T22:40:40.837 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.837 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:40.838 DEBUG:boto:Port: 8001 2022-09-09T22:40:40.838 DEBUG:boto:Params: {} 2022-09-09T22:40:40.839 DEBUG:boto:Token: None 2022-09-09T22:40:40.839 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.840 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:26EcXc7XMqNtbQXOFqPpS3uYWv0= 2022-09-09T22:40:40.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:26EcXc7XMqNtbQXOFqPpS3uYWv0=', 'Content-Length': '0'} 2022-09-09T22:40:40.844 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('x-amz-version-id', 'Ux4yCo1Zw0Z6LcOb725cQEb153cMR29'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000607e04f660f8fc53-00631bc0e8-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.845 DEBUG:boto:path=/jkcgbj-19/dummy 2022-09-09T22:40:40.845 DEBUG:boto:auth_path=/jkcgbj-19/dummy 2022-09-09T22:40:40.846 DEBUG:boto:path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.847 DEBUG:boto:auth_path=/jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.847 DEBUG:boto:Method: GET 2022-09-09T22:40:40.848 DEBUG:boto:Path: /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.849 DEBUG:boto:Data: 2022-09-09T22:40:40.849 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.850 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:40.850 DEBUG:boto:Port: 8000 2022-09-09T22:40:40.851 DEBUG:boto:Params: {} 2022-09-09T22:40:40.852 DEBUG:boto:Token: None 2022-09-09T22:40:40.852 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/dummy?versionId=Ux4yCo1Zw0Z6LcOb725cQEb153cMR29 2022-09-09T22:40:40.853 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:26EcXc7XMqNtbQXOFqPpS3uYWv0= 2022-09-09T22:40:40.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:26EcXc7XMqNtbQXOFqPpS3uYWv0=', 'Content-Length': '0'} 2022-09-09T22:40:40.857 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:39:43 GMT'), ('x-amz-version-id', 'Ux4yCo1Zw0Z6LcOb725cQEb153cMR29'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008ffa8db954f54359-00631bc0e8-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.858 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:40.859 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:40.860 DEBUG:boto:path=/jkcgbj-19/?prefix=dummy 2022-09-09T22:40:40.861 DEBUG:boto:auth_path=/jkcgbj-19/?prefix=dummy 2022-09-09T22:40:40.862 DEBUG:boto:Method: GET 2022-09-09T22:40:40.862 DEBUG:boto:Path: /jkcgbj-19/?prefix=dummy 2022-09-09T22:40:40.863 DEBUG:boto:Data: 2022-09-09T22:40:40.864 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.865 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:40.865 DEBUG:boto:Port: 8001 2022-09-09T22:40:40.866 DEBUG:boto:Params: {} 2022-09-09T22:40:40.867 DEBUG:boto:Token: None 2022-09-09T22:40:40.867 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/ 2022-09-09T22:40:40.868 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:axjiiwtiq6LnRVW1o/AfG7Vbmn0= 2022-09-09T22:40:40.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:axjiiwtiq6LnRVW1o/AfG7Vbmn0=', 'Content-Length': '0'} 2022-09-09T22:40:40.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e3c69dd07a30122-00631bc0e8-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.875 DEBUG:boto:b'jkcgbj-19dummy1000falsedummy2022-09-09T22:39:43.512Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:40:40.876 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:40.876 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:40.877 DEBUG:boto:path=/jkcgbj-19/?prefix=dummy 2022-09-09T22:40:40.877 DEBUG:boto:auth_path=/jkcgbj-19/?prefix=dummy 2022-09-09T22:40:40.878 DEBUG:boto:Method: GET 2022-09-09T22:40:40.878 DEBUG:boto:Path: /jkcgbj-19/?prefix=dummy 2022-09-09T22:40:40.879 DEBUG:boto:Data: 2022-09-09T22:40:40.880 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.880 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:40.881 DEBUG:boto:Port: 8000 2022-09-09T22:40:40.881 DEBUG:boto:Params: {} 2022-09-09T22:40:40.882 DEBUG:boto:Token: None 2022-09-09T22:40:40.883 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/ 2022-09-09T22:40:40.883 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:axjiiwtiq6LnRVW1o/AfG7Vbmn0= 2022-09-09T22:40:40.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:axjiiwtiq6LnRVW1o/AfG7Vbmn0=', 'Content-Length': '0'} 2022-09-09T22:40:40.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000233e0d76f0d42f32-00631bc0e8-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.930 DEBUG:boto:b'jkcgbj-19dummy1000falsedummy2022-09-09T22:39:43.512Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:40:40.931 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-19 zones={test-zone3, test-zone2} 2022-09-09T22:40:40.932 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:40.932 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:40.933 DEBUG:boto:Method: HEAD 2022-09-09T22:40:40.933 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:40.934 DEBUG:boto:Data: 2022-09-09T22:40:40.935 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.936 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:40.937 DEBUG:boto:Port: 8000 2022-09-09T22:40:40.938 DEBUG:boto:Params: {} 2022-09-09T22:40:40.939 DEBUG:boto:Token: None 2022-09-09T22:40:40.940 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/ 2022-09-09T22:40:40.941 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rEmwpALPt3PBB+6o7cVMUhVjf50= 2022-09-09T22:40:40.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rEmwpALPt3PBB+6o7cVMUhVjf50=', 'Content-Length': '0'} 2022-09-09T22:40:40.947 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006fd89bd02c5a3517-00631bc0e8-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.948 DEBUG:boto:path=/jkcgbj-19/obj 2022-09-09T22:40:40.949 DEBUG:boto:auth_path=/jkcgbj-19/obj 2022-09-09T22:40:40.949 DEBUG:boto:Method: PUT 2022-09-09T22:40:40.950 DEBUG:boto:Path: /jkcgbj-19/obj 2022-09-09T22:40:40.950 DEBUG:boto:Data: 2022-09-09T22:40:40.951 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:40:40.951 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:40.952 DEBUG:boto:Port: 8000 2022-09-09T22:40:40.952 DEBUG:boto:Params: {} 2022-09-09T22:40:40.953 DEBUG:boto:Token: None 2022-09-09T22:40:40.953 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/obj 2022-09-09T22:40:40.954 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ChBhKUvaDpOmdIB+q+efFkbGb1k= 2022-09-09T22:40:40.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ChBhKUvaDpOmdIB+q+efFkbGb1k='} 2022-09-09T22:40:40.981 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'YBT3W9ZEvXJBtzLcP3Sbp-MrMWn31Gi'), ('x-amz-request-id', 'tx000008c15906787230fa6-00631bc0e8-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.981 DEBUG:rgw_multi.tests:zone=test-zone1 version=YBT3W9ZEvXJBtzLcP3Sbp-MrMWn31Gi 2022-09-09T22:40:40.982 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:40.982 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:40.983 DEBUG:boto:Method: HEAD 2022-09-09T22:40:40.983 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:40.984 DEBUG:boto:Data: 2022-09-09T22:40:40.984 DEBUG:boto:Headers: {} 2022-09-09T22:40:40.985 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:40.985 DEBUG:boto:Port: 8000 2022-09-09T22:40:40.986 DEBUG:boto:Params: {} 2022-09-09T22:40:40.987 DEBUG:boto:Token: None 2022-09-09T22:40:40.987 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/ 2022-09-09T22:40:40.988 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rEmwpALPt3PBB+6o7cVMUhVjf50= 2022-09-09T22:40:40.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rEmwpALPt3PBB+6o7cVMUhVjf50=', 'Content-Length': '0'} 2022-09-09T22:40:40.993 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010dd9a4bc603d0eb-00631bc0e8-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:40.993 DEBUG:boto:path=/jkcgbj-19/obj 2022-09-09T22:40:40.994 DEBUG:boto:auth_path=/jkcgbj-19/obj 2022-09-09T22:40:40.995 DEBUG:boto:Method: PUT 2022-09-09T22:40:40.995 DEBUG:boto:Path: /jkcgbj-19/obj 2022-09-09T22:40:40.996 DEBUG:boto:Data: 2022-09-09T22:40:40.996 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:40:40.997 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:40.997 DEBUG:boto:Port: 8000 2022-09-09T22:40:40.998 DEBUG:boto:Params: {} 2022-09-09T22:40:40.998 DEBUG:boto:Token: None 2022-09-09T22:40:40.999 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:40:40 GMT /jkcgbj-19/obj 2022-09-09T22:40:41.000 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ChBhKUvaDpOmdIB+q+efFkbGb1k= 2022-09-09T22:40:41.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:40:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ChBhKUvaDpOmdIB+q+efFkbGb1k='} 2022-09-09T22:40:41.026 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'qRXnhy0E17ZdwGS3jZcRg1EAvhqwYZo'), ('x-amz-request-id', 'tx00000735c4d94946328ff-00631bc0e8-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.027 DEBUG:rgw_multi.tests:zone=test-zone2 version=qRXnhy0E17ZdwGS3jZcRg1EAvhqwYZo 2022-09-09T22:40:41.028 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:41.028 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:41.029 DEBUG:boto:Method: HEAD 2022-09-09T22:40:41.029 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:41.030 DEBUG:boto:Data: 2022-09-09T22:40:41.030 DEBUG:boto:Headers: {} 2022-09-09T22:40:41.031 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:41.031 DEBUG:boto:Port: 8001 2022-09-09T22:40:41.032 DEBUG:boto:Params: {} 2022-09-09T22:40:41.033 DEBUG:boto:Token: None 2022-09-09T22:40:41.033 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/ 2022-09-09T22:40:41.034 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8= 2022-09-09T22:40:41.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8=', 'Content-Length': '0'} 2022-09-09T22:40:41.039 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000598acfa46a62957e-00631bc0e9-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.040 DEBUG:boto:path=/jkcgbj-19/obj 2022-09-09T22:40:41.040 DEBUG:boto:auth_path=/jkcgbj-19/obj 2022-09-09T22:40:41.041 DEBUG:boto:Method: PUT 2022-09-09T22:40:41.041 DEBUG:boto:Path: /jkcgbj-19/obj 2022-09-09T22:40:41.042 DEBUG:boto:Data: 2022-09-09T22:40:41.042 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:40:41.043 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:41.044 DEBUG:boto:Port: 8001 2022-09-09T22:40:41.044 DEBUG:boto:Params: {} 2022-09-09T22:40:41.045 DEBUG:boto:Token: None 2022-09-09T22:40:41.045 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/obj 2022-09-09T22:40:41.046 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ= 2022-09-09T22:40:41.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ='} 2022-09-09T22:40:41.091 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'kCktnM5aVDufoybJWTVKjew9CQRswsq'), ('x-amz-request-id', 'tx000003e859b77f12e1c1d-00631bc0e9-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.092 DEBUG:rgw_multi.tests:zone=test-zone3 version=kCktnM5aVDufoybJWTVKjew9CQRswsq 2022-09-09T22:40:41.093 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:41.093 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:41.094 DEBUG:boto:Method: HEAD 2022-09-09T22:40:41.094 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:41.095 DEBUG:boto:Data: 2022-09-09T22:40:41.095 DEBUG:boto:Headers: {} 2022-09-09T22:40:41.096 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:41.097 DEBUG:boto:Port: 8000 2022-09-09T22:40:41.097 DEBUG:boto:Params: {} 2022-09-09T22:40:41.098 DEBUG:boto:Token: None 2022-09-09T22:40:41.098 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/ 2022-09-09T22:40:41.099 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8= 2022-09-09T22:40:41.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8=', 'Content-Length': '0'} 2022-09-09T22:40:41.105 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a144c07acc6d12f-00631bc0e9-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.106 DEBUG:boto:path=/jkcgbj-19/obj 2022-09-09T22:40:41.106 DEBUG:boto:auth_path=/jkcgbj-19/obj 2022-09-09T22:40:41.107 DEBUG:boto:Method: PUT 2022-09-09T22:40:41.107 DEBUG:boto:Path: /jkcgbj-19/obj 2022-09-09T22:40:41.108 DEBUG:boto:Data: 2022-09-09T22:40:41.108 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:40:41.109 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:41.109 DEBUG:boto:Port: 8000 2022-09-09T22:40:41.110 DEBUG:boto:Params: {} 2022-09-09T22:40:41.111 DEBUG:boto:Token: None 2022-09-09T22:40:41.111 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/obj 2022-09-09T22:40:41.112 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ= 2022-09-09T22:40:41.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ='} 2022-09-09T22:40:41.136 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'tjFEg2VPMCUVe-4mXFePBfYlGSUmTEg'), ('x-amz-request-id', 'tx00000e904b2ab42cce794-00631bc0e9-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.137 DEBUG:rgw_multi.tests:zone=test-zone1 version=tjFEg2VPMCUVe-4mXFePBfYlGSUmTEg 2022-09-09T22:40:41.138 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:41.138 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:41.139 DEBUG:boto:Method: HEAD 2022-09-09T22:40:41.140 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:41.140 DEBUG:boto:Data: 2022-09-09T22:40:41.141 DEBUG:boto:Headers: {} 2022-09-09T22:40:41.141 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:41.142 DEBUG:boto:Port: 8000 2022-09-09T22:40:41.142 DEBUG:boto:Params: {} 2022-09-09T22:40:41.143 DEBUG:boto:Token: None 2022-09-09T22:40:41.144 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/ 2022-09-09T22:40:41.144 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8= 2022-09-09T22:40:41.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8=', 'Content-Length': '0'} 2022-09-09T22:40:41.150 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001408d111cd88fb67-00631bc0e9-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.151 DEBUG:boto:path=/jkcgbj-19/obj 2022-09-09T22:40:41.152 DEBUG:boto:auth_path=/jkcgbj-19/obj 2022-09-09T22:40:41.153 DEBUG:boto:Method: PUT 2022-09-09T22:40:41.154 DEBUG:boto:Path: /jkcgbj-19/obj 2022-09-09T22:40:41.154 DEBUG:boto:Data: 2022-09-09T22:40:41.155 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:40:41.156 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:41.157 DEBUG:boto:Port: 8000 2022-09-09T22:40:41.158 DEBUG:boto:Params: {} 2022-09-09T22:40:41.158 DEBUG:boto:Token: None 2022-09-09T22:40:41.159 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/obj 2022-09-09T22:40:41.160 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ= 2022-09-09T22:40:41.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ='} 2022-09-09T22:40:41.185 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'tFD3QI1AfR3.mOMdPi-L94TckfetkhR'), ('x-amz-request-id', 'tx000002039a2e8cf06b50e-00631bc0e9-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.187 DEBUG:rgw_multi.tests:zone=test-zone2 version=tFD3QI1AfR3.mOMdPi-L94TckfetkhR 2022-09-09T22:40:41.188 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:41.189 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:41.190 DEBUG:boto:Method: HEAD 2022-09-09T22:40:41.191 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:41.192 DEBUG:boto:Data: 2022-09-09T22:40:41.193 DEBUG:boto:Headers: {} 2022-09-09T22:40:41.193 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:41.194 DEBUG:boto:Port: 8001 2022-09-09T22:40:41.195 DEBUG:boto:Params: {} 2022-09-09T22:40:41.196 DEBUG:boto:Token: None 2022-09-09T22:40:41.196 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/ 2022-09-09T22:40:41.197 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8= 2022-09-09T22:40:41.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8=', 'Content-Length': '0'} 2022-09-09T22:40:41.203 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c088040941651b93-00631bc0e9-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.204 DEBUG:boto:path=/jkcgbj-19/obj 2022-09-09T22:40:41.205 DEBUG:boto:auth_path=/jkcgbj-19/obj 2022-09-09T22:40:41.206 DEBUG:boto:Method: PUT 2022-09-09T22:40:41.207 DEBUG:boto:Path: /jkcgbj-19/obj 2022-09-09T22:40:41.208 DEBUG:boto:Data: 2022-09-09T22:40:41.208 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:40:41.209 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:41.210 DEBUG:boto:Port: 8001 2022-09-09T22:40:41.211 DEBUG:boto:Params: {} 2022-09-09T22:40:41.211 DEBUG:boto:Token: None 2022-09-09T22:40:41.212 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/obj 2022-09-09T22:40:41.212 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ= 2022-09-09T22:40:41.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ='} 2022-09-09T22:40:41.236 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'vrfOUyccGLacwbK-gMfsKuBypX5FLXG'), ('x-amz-request-id', 'tx00000946ad30210dd685b-00631bc0e9-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.237 DEBUG:rgw_multi.tests:zone=test-zone3 version=vrfOUyccGLacwbK-gMfsKuBypX5FLXG 2022-09-09T22:40:41.238 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:41.238 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:41.239 DEBUG:boto:Method: HEAD 2022-09-09T22:40:41.240 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:41.240 DEBUG:boto:Data: 2022-09-09T22:40:41.241 DEBUG:boto:Headers: {} 2022-09-09T22:40:41.241 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:41.242 DEBUG:boto:Port: 8000 2022-09-09T22:40:41.242 DEBUG:boto:Params: {} 2022-09-09T22:40:41.243 DEBUG:boto:Token: None 2022-09-09T22:40:41.243 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/ 2022-09-09T22:40:41.244 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8= 2022-09-09T22:40:41.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8=', 'Content-Length': '0'} 2022-09-09T22:40:41.250 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ed2a05ea8f307ab0-00631bc0e9-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.251 DEBUG:boto:path=/jkcgbj-19/obj 2022-09-09T22:40:41.251 DEBUG:boto:auth_path=/jkcgbj-19/obj 2022-09-09T22:40:41.252 DEBUG:boto:Method: PUT 2022-09-09T22:40:41.252 DEBUG:boto:Path: /jkcgbj-19/obj 2022-09-09T22:40:41.253 DEBUG:boto:Data: 2022-09-09T22:40:41.253 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:40:41.254 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:41.255 DEBUG:boto:Port: 8000 2022-09-09T22:40:41.255 DEBUG:boto:Params: {} 2022-09-09T22:40:41.256 DEBUG:boto:Token: None 2022-09-09T22:40:41.256 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/obj 2022-09-09T22:40:41.257 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ= 2022-09-09T22:40:41.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ='} 2022-09-09T22:40:41.283 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'bPGipS.9xn3v2Tr7GKV6QcIcgZ7apW5'), ('x-amz-request-id', 'tx000006d9f8921534b80df-00631bc0e9-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.284 DEBUG:rgw_multi.tests:zone=test-zone1 version=bPGipS.9xn3v2Tr7GKV6QcIcgZ7apW5 2022-09-09T22:40:41.285 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:41.285 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:41.286 DEBUG:boto:Method: HEAD 2022-09-09T22:40:41.287 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:41.287 DEBUG:boto:Data: 2022-09-09T22:40:41.288 DEBUG:boto:Headers: {} 2022-09-09T22:40:41.289 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:41.289 DEBUG:boto:Port: 8000 2022-09-09T22:40:41.290 DEBUG:boto:Params: {} 2022-09-09T22:40:41.291 DEBUG:boto:Token: None 2022-09-09T22:40:41.291 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/ 2022-09-09T22:40:41.292 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8= 2022-09-09T22:40:41.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8=', 'Content-Length': '0'} 2022-09-09T22:40:41.298 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d49210343086324-00631bc0e9-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.299 DEBUG:boto:path=/jkcgbj-19/obj 2022-09-09T22:40:41.299 DEBUG:boto:auth_path=/jkcgbj-19/obj 2022-09-09T22:40:41.300 DEBUG:boto:Method: PUT 2022-09-09T22:40:41.300 DEBUG:boto:Path: /jkcgbj-19/obj 2022-09-09T22:40:41.301 DEBUG:boto:Data: 2022-09-09T22:40:41.301 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:40:41.302 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:41.302 DEBUG:boto:Port: 8000 2022-09-09T22:40:41.303 DEBUG:boto:Params: {} 2022-09-09T22:40:41.303 DEBUG:boto:Token: None 2022-09-09T22:40:41.304 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/obj 2022-09-09T22:40:41.305 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ= 2022-09-09T22:40:41.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ='} 2022-09-09T22:40:41.328 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'b877gxmcaiM.3HSgVao.tK14vZ6AxyO'), ('x-amz-request-id', 'tx00000666ce7f1562d0ff7-00631bc0e9-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.329 DEBUG:rgw_multi.tests:zone=test-zone2 version=b877gxmcaiM.3HSgVao.tK14vZ6AxyO 2022-09-09T22:40:41.330 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:41.331 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:41.332 DEBUG:boto:Method: HEAD 2022-09-09T22:40:41.332 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:41.333 DEBUG:boto:Data: 2022-09-09T22:40:41.334 DEBUG:boto:Headers: {} 2022-09-09T22:40:41.334 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:41.335 DEBUG:boto:Port: 8001 2022-09-09T22:40:41.335 DEBUG:boto:Params: {} 2022-09-09T22:40:41.336 DEBUG:boto:Token: None 2022-09-09T22:40:41.336 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/ 2022-09-09T22:40:41.337 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8= 2022-09-09T22:40:41.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8=', 'Content-Length': '0'} 2022-09-09T22:40:41.343 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000911093656a45b8-00631bc0e9-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.344 DEBUG:boto:path=/jkcgbj-19/obj 2022-09-09T22:40:41.344 DEBUG:boto:auth_path=/jkcgbj-19/obj 2022-09-09T22:40:41.345 DEBUG:boto:Method: PUT 2022-09-09T22:40:41.345 DEBUG:boto:Path: /jkcgbj-19/obj 2022-09-09T22:40:41.346 DEBUG:boto:Data: 2022-09-09T22:40:41.346 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:40:41.347 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:41.347 DEBUG:boto:Port: 8001 2022-09-09T22:40:41.348 DEBUG:boto:Params: {} 2022-09-09T22:40:41.348 DEBUG:boto:Token: None 2022-09-09T22:40:41.349 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/obj 2022-09-09T22:40:41.355 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ= 2022-09-09T22:40:41.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ='} 2022-09-09T22:40:41.379 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ANPmfmuNWlkjG2kbYaSmipyRrJVIL6K'), ('x-amz-request-id', 'tx0000057475c163ec18220-00631bc0e9-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.380 DEBUG:rgw_multi.tests:zone=test-zone3 version=ANPmfmuNWlkjG2kbYaSmipyRrJVIL6K 2022-09-09T22:40:41.381 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:41.381 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:41.382 DEBUG:boto:Method: HEAD 2022-09-09T22:40:41.382 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:41.383 DEBUG:boto:Data: 2022-09-09T22:40:41.383 DEBUG:boto:Headers: {} 2022-09-09T22:40:41.384 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:41.384 DEBUG:boto:Port: 8000 2022-09-09T22:40:41.385 DEBUG:boto:Params: {} 2022-09-09T22:40:41.385 DEBUG:boto:Token: None 2022-09-09T22:40:41.386 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/ 2022-09-09T22:40:41.386 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8= 2022-09-09T22:40:41.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8=', 'Content-Length': '0'} 2022-09-09T22:40:41.392 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000228908021ab5fd4c-00631bc0e9-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.393 DEBUG:boto:path=/jkcgbj-19/obj 2022-09-09T22:40:41.394 DEBUG:boto:auth_path=/jkcgbj-19/obj 2022-09-09T22:40:41.394 DEBUG:boto:Method: PUT 2022-09-09T22:40:41.395 DEBUG:boto:Path: /jkcgbj-19/obj 2022-09-09T22:40:41.395 DEBUG:boto:Data: 2022-09-09T22:40:41.396 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:40:41.396 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:41.397 DEBUG:boto:Port: 8000 2022-09-09T22:40:41.397 DEBUG:boto:Params: {} 2022-09-09T22:40:41.398 DEBUG:boto:Token: None 2022-09-09T22:40:41.398 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/obj 2022-09-09T22:40:41.399 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ= 2022-09-09T22:40:41.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ='} 2022-09-09T22:40:41.442 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'DhdLm8POu-hSdL6Zcgs8jpkcH0XgpwV'), ('x-amz-request-id', 'tx00000d28856b59fcb5967-00631bc0e9-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.443 DEBUG:rgw_multi.tests:zone=test-zone1 version=DhdLm8POu-hSdL6Zcgs8jpkcH0XgpwV 2022-09-09T22:40:41.444 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:41.445 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:41.445 DEBUG:boto:Method: HEAD 2022-09-09T22:40:41.446 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:41.447 DEBUG:boto:Data: 2022-09-09T22:40:41.448 DEBUG:boto:Headers: {} 2022-09-09T22:40:41.449 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:41.449 DEBUG:boto:Port: 8000 2022-09-09T22:40:41.450 DEBUG:boto:Params: {} 2022-09-09T22:40:41.450 DEBUG:boto:Token: None 2022-09-09T22:40:41.451 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/ 2022-09-09T22:40:41.452 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8= 2022-09-09T22:40:41.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8=', 'Content-Length': '0'} 2022-09-09T22:40:41.457 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000483882194ae9b6e6-00631bc0e9-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.458 DEBUG:boto:path=/jkcgbj-19/obj 2022-09-09T22:40:41.459 DEBUG:boto:auth_path=/jkcgbj-19/obj 2022-09-09T22:40:41.460 DEBUG:boto:Method: PUT 2022-09-09T22:40:41.460 DEBUG:boto:Path: /jkcgbj-19/obj 2022-09-09T22:40:41.461 DEBUG:boto:Data: 2022-09-09T22:40:41.462 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:40:41.462 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:41.463 DEBUG:boto:Port: 8000 2022-09-09T22:40:41.464 DEBUG:boto:Params: {} 2022-09-09T22:40:41.464 DEBUG:boto:Token: None 2022-09-09T22:40:41.465 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/obj 2022-09-09T22:40:41.466 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ= 2022-09-09T22:40:41.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ='} 2022-09-09T22:40:41.490 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'SodkxDO9pIwQlDyf8Bf-tb4Y9DF9.1I'), ('x-amz-request-id', 'tx00000c59b40a390f7b4f1-00631bc0e9-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.491 DEBUG:rgw_multi.tests:zone=test-zone2 version=SodkxDO9pIwQlDyf8Bf-tb4Y9DF9.1I 2022-09-09T22:40:41.492 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:41.492 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:41.493 DEBUG:boto:Method: HEAD 2022-09-09T22:40:41.493 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:41.494 DEBUG:boto:Data: 2022-09-09T22:40:41.494 DEBUG:boto:Headers: {} 2022-09-09T22:40:41.495 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:41.496 DEBUG:boto:Port: 8001 2022-09-09T22:40:41.496 DEBUG:boto:Params: {} 2022-09-09T22:40:41.497 DEBUG:boto:Token: None 2022-09-09T22:40:41.497 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/ 2022-09-09T22:40:41.498 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8= 2022-09-09T22:40:41.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8=', 'Content-Length': '0'} 2022-09-09T22:40:41.504 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007eacd7b31058fe91-00631bc0e9-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.505 DEBUG:boto:path=/jkcgbj-19/obj 2022-09-09T22:40:41.506 DEBUG:boto:auth_path=/jkcgbj-19/obj 2022-09-09T22:40:41.506 DEBUG:boto:Method: PUT 2022-09-09T22:40:41.507 DEBUG:boto:Path: /jkcgbj-19/obj 2022-09-09T22:40:41.508 DEBUG:boto:Data: 2022-09-09T22:40:41.508 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:40:41.508 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:41.509 DEBUG:boto:Port: 8001 2022-09-09T22:40:41.509 DEBUG:boto:Params: {} 2022-09-09T22:40:41.510 DEBUG:boto:Token: None 2022-09-09T22:40:41.511 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/obj 2022-09-09T22:40:41.512 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ= 2022-09-09T22:40:41.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ='} 2022-09-09T22:40:41.558 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'yLEh1mMindNgGKWthBAOIY5KsdJyWR5'), ('x-amz-request-id', 'tx00000c6dcc92ef6c8910e-00631bc0e9-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.559 DEBUG:rgw_multi.tests:zone=test-zone3 version=yLEh1mMindNgGKWthBAOIY5KsdJyWR5 2022-09-09T22:40:41.560 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:41.560 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:41.560 DEBUG:boto:Method: HEAD 2022-09-09T22:40:41.561 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:41.561 DEBUG:boto:Data: 2022-09-09T22:40:41.562 DEBUG:boto:Headers: {} 2022-09-09T22:40:41.562 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:41.563 DEBUG:boto:Port: 8000 2022-09-09T22:40:41.563 DEBUG:boto:Params: {} 2022-09-09T22:40:41.564 DEBUG:boto:Token: None 2022-09-09T22:40:41.565 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/ 2022-09-09T22:40:41.565 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8= 2022-09-09T22:40:41.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8=', 'Content-Length': '0'} 2022-09-09T22:40:41.571 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000caf30d518a15eeb8-00631bc0e9-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.572 DEBUG:boto:path=/jkcgbj-19/obj 2022-09-09T22:40:41.572 DEBUG:boto:auth_path=/jkcgbj-19/obj 2022-09-09T22:40:41.573 DEBUG:boto:Method: PUT 2022-09-09T22:40:41.573 DEBUG:boto:Path: /jkcgbj-19/obj 2022-09-09T22:40:41.574 DEBUG:boto:Data: 2022-09-09T22:40:41.574 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:40:41.575 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:41.575 DEBUG:boto:Port: 8000 2022-09-09T22:40:41.575 DEBUG:boto:Params: {} 2022-09-09T22:40:41.576 DEBUG:boto:Token: None 2022-09-09T22:40:41.577 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/obj 2022-09-09T22:40:41.577 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ= 2022-09-09T22:40:41.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ='} 2022-09-09T22:40:41.601 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Kmke451TNZJapv.bQCV9eAg4htoo0B-'), ('x-amz-request-id', 'tx00000f08a92de713a18af-00631bc0e9-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.603 DEBUG:rgw_multi.tests:zone=test-zone1 version=Kmke451TNZJapv.bQCV9eAg4htoo0B- 2022-09-09T22:40:41.604 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:41.604 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:41.605 DEBUG:boto:Method: HEAD 2022-09-09T22:40:41.605 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:41.606 DEBUG:boto:Data: 2022-09-09T22:40:41.606 DEBUG:boto:Headers: {} 2022-09-09T22:40:41.607 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:41.607 DEBUG:boto:Port: 8000 2022-09-09T22:40:41.608 DEBUG:boto:Params: {} 2022-09-09T22:40:41.608 DEBUG:boto:Token: None 2022-09-09T22:40:41.609 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/ 2022-09-09T22:40:41.612 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8= 2022-09-09T22:40:41.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8=', 'Content-Length': '0'} 2022-09-09T22:40:41.619 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f056365416bc78f0-00631bc0e9-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.621 DEBUG:boto:path=/jkcgbj-19/obj 2022-09-09T22:40:41.621 DEBUG:boto:auth_path=/jkcgbj-19/obj 2022-09-09T22:40:41.622 DEBUG:boto:Method: PUT 2022-09-09T22:40:41.623 DEBUG:boto:Path: /jkcgbj-19/obj 2022-09-09T22:40:41.623 DEBUG:boto:Data: 2022-09-09T22:40:41.624 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:40:41.625 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:41.625 DEBUG:boto:Port: 8000 2022-09-09T22:40:41.626 DEBUG:boto:Params: {} 2022-09-09T22:40:41.626 DEBUG:boto:Token: None 2022-09-09T22:40:41.627 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/obj 2022-09-09T22:40:41.628 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ= 2022-09-09T22:40:41.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ='} 2022-09-09T22:40:41.670 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'uHirdu1s6x9nbyvRYumWBfN7BhOGGVF'), ('x-amz-request-id', 'tx0000009a31393bc286eab-00631bc0e9-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.672 DEBUG:rgw_multi.tests:zone=test-zone2 version=uHirdu1s6x9nbyvRYumWBfN7BhOGGVF 2022-09-09T22:40:41.673 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:41.673 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:41.674 DEBUG:boto:Method: HEAD 2022-09-09T22:40:41.675 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:41.676 DEBUG:boto:Data: 2022-09-09T22:40:41.677 DEBUG:boto:Headers: {} 2022-09-09T22:40:41.678 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:41.678 DEBUG:boto:Port: 8001 2022-09-09T22:40:41.679 DEBUG:boto:Params: {} 2022-09-09T22:40:41.679 DEBUG:boto:Token: None 2022-09-09T22:40:41.680 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/ 2022-09-09T22:40:41.681 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8= 2022-09-09T22:40:41.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8=', 'Content-Length': '0'} 2022-09-09T22:40:41.687 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000049f276e272ed976b-00631bc0e9-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.688 DEBUG:boto:path=/jkcgbj-19/obj 2022-09-09T22:40:41.689 DEBUG:boto:auth_path=/jkcgbj-19/obj 2022-09-09T22:40:41.689 DEBUG:boto:Method: PUT 2022-09-09T22:40:41.690 DEBUG:boto:Path: /jkcgbj-19/obj 2022-09-09T22:40:41.690 DEBUG:boto:Data: 2022-09-09T22:40:41.691 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:40:41.691 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:41.692 DEBUG:boto:Port: 8001 2022-09-09T22:40:41.692 DEBUG:boto:Params: {} 2022-09-09T22:40:41.693 DEBUG:boto:Token: None 2022-09-09T22:40:41.693 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/obj 2022-09-09T22:40:41.694 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ= 2022-09-09T22:40:41.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ='} 2022-09-09T22:40:41.713 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'zPHONCiW8rBkBE9g53hktsnhLaSTP6z'), ('x-amz-request-id', 'tx00000ab6428d14bd97c67-00631bc0e9-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.714 DEBUG:rgw_multi.tests:zone=test-zone3 version=zPHONCiW8rBkBE9g53hktsnhLaSTP6z 2022-09-09T22:40:41.714 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:41.715 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:41.715 DEBUG:boto:Method: HEAD 2022-09-09T22:40:41.716 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:41.716 DEBUG:boto:Data: 2022-09-09T22:40:41.717 DEBUG:boto:Headers: {} 2022-09-09T22:40:41.717 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:41.718 DEBUG:boto:Port: 8000 2022-09-09T22:40:41.718 DEBUG:boto:Params: {} 2022-09-09T22:40:41.719 DEBUG:boto:Token: None 2022-09-09T22:40:41.719 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/ 2022-09-09T22:40:41.720 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8= 2022-09-09T22:40:41.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8=', 'Content-Length': '0'} 2022-09-09T22:40:41.725 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c81d716a698fa60a-00631bc0e9-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.726 DEBUG:boto:path=/jkcgbj-19/obj 2022-09-09T22:40:41.726 DEBUG:boto:auth_path=/jkcgbj-19/obj 2022-09-09T22:40:41.727 DEBUG:boto:Method: PUT 2022-09-09T22:40:41.727 DEBUG:boto:Path: /jkcgbj-19/obj 2022-09-09T22:40:41.728 DEBUG:boto:Data: 2022-09-09T22:40:41.728 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:40:41.729 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:41.729 DEBUG:boto:Port: 8000 2022-09-09T22:40:41.730 DEBUG:boto:Params: {} 2022-09-09T22:40:41.731 DEBUG:boto:Token: None 2022-09-09T22:40:41.731 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/obj 2022-09-09T22:40:41.732 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ= 2022-09-09T22:40:41.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ='} 2022-09-09T22:40:41.756 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'z0U59Aim37JoMUmFgPJS-5RFF1sKze0'), ('x-amz-request-id', 'tx00000bdb01a4c725f8f5f-00631bc0e9-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.756 DEBUG:rgw_multi.tests:zone=test-zone1 version=z0U59Aim37JoMUmFgPJS-5RFF1sKze0 2022-09-09T22:40:41.757 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:41.758 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:41.758 DEBUG:boto:Method: HEAD 2022-09-09T22:40:41.759 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:41.760 DEBUG:boto:Data: 2022-09-09T22:40:41.760 DEBUG:boto:Headers: {} 2022-09-09T22:40:41.761 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:41.762 DEBUG:boto:Port: 8000 2022-09-09T22:40:41.762 DEBUG:boto:Params: {} 2022-09-09T22:40:41.763 DEBUG:boto:Token: None 2022-09-09T22:40:41.764 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/ 2022-09-09T22:40:41.764 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8= 2022-09-09T22:40:41.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8=', 'Content-Length': '0'} 2022-09-09T22:40:41.770 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b2629ba48628ef37-00631bc0e9-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.771 DEBUG:boto:path=/jkcgbj-19/obj 2022-09-09T22:40:41.772 DEBUG:boto:auth_path=/jkcgbj-19/obj 2022-09-09T22:40:41.773 DEBUG:boto:Method: PUT 2022-09-09T22:40:41.773 DEBUG:boto:Path: /jkcgbj-19/obj 2022-09-09T22:40:41.774 DEBUG:boto:Data: 2022-09-09T22:40:41.775 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:40:41.775 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:41.776 DEBUG:boto:Port: 8000 2022-09-09T22:40:41.777 DEBUG:boto:Params: {} 2022-09-09T22:40:41.777 DEBUG:boto:Token: None 2022-09-09T22:40:41.778 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/obj 2022-09-09T22:40:41.778 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ= 2022-09-09T22:40:41.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ='} 2022-09-09T22:40:41.803 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '5W67jsgyJWq6oKkqLKNneQ1enkrdjsm'), ('x-amz-request-id', 'tx00000e193c9f1d512add3-00631bc0e9-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.804 DEBUG:rgw_multi.tests:zone=test-zone2 version=5W67jsgyJWq6oKkqLKNneQ1enkrdjsm 2022-09-09T22:40:41.805 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:41.805 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:41.806 DEBUG:boto:Method: HEAD 2022-09-09T22:40:41.807 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:41.807 DEBUG:boto:Data: 2022-09-09T22:40:41.808 DEBUG:boto:Headers: {} 2022-09-09T22:40:41.808 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:41.809 DEBUG:boto:Port: 8001 2022-09-09T22:40:41.809 DEBUG:boto:Params: {} 2022-09-09T22:40:41.810 DEBUG:boto:Token: None 2022-09-09T22:40:41.811 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/ 2022-09-09T22:40:41.811 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8= 2022-09-09T22:40:41.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8=', 'Content-Length': '0'} 2022-09-09T22:40:41.817 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d48700bd846a99c3-00631bc0e9-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.819 DEBUG:boto:path=/jkcgbj-19/obj 2022-09-09T22:40:41.819 DEBUG:boto:auth_path=/jkcgbj-19/obj 2022-09-09T22:40:41.820 DEBUG:boto:Method: PUT 2022-09-09T22:40:41.820 DEBUG:boto:Path: /jkcgbj-19/obj 2022-09-09T22:40:41.821 DEBUG:boto:Data: 2022-09-09T22:40:41.821 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:40:41.822 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:41.823 DEBUG:boto:Port: 8001 2022-09-09T22:40:41.824 DEBUG:boto:Params: {} 2022-09-09T22:40:41.825 DEBUG:boto:Token: None 2022-09-09T22:40:41.826 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/obj 2022-09-09T22:40:41.827 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ= 2022-09-09T22:40:41.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ='} 2022-09-09T22:40:41.851 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'A4J7M3-ded8tPFTb8qmsxW1QnLbEFuR'), ('x-amz-request-id', 'tx00000770d406db1ceac28-00631bc0e9-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.852 DEBUG:rgw_multi.tests:zone=test-zone3 version=A4J7M3-ded8tPFTb8qmsxW1QnLbEFuR 2022-09-09T22:40:41.852 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:41.853 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:41.853 DEBUG:boto:Method: HEAD 2022-09-09T22:40:41.854 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:41.854 DEBUG:boto:Data: 2022-09-09T22:40:41.855 DEBUG:boto:Headers: {} 2022-09-09T22:40:41.855 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:41.856 DEBUG:boto:Port: 8000 2022-09-09T22:40:41.856 DEBUG:boto:Params: {} 2022-09-09T22:40:41.857 DEBUG:boto:Token: None 2022-09-09T22:40:41.857 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/ 2022-09-09T22:40:41.858 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8= 2022-09-09T22:40:41.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8=', 'Content-Length': '0'} 2022-09-09T22:40:41.863 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009dcaf1fd7681192c-00631bc0e9-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.864 DEBUG:boto:path=/jkcgbj-19/obj 2022-09-09T22:40:41.864 DEBUG:boto:auth_path=/jkcgbj-19/obj 2022-09-09T22:40:41.865 DEBUG:boto:Method: PUT 2022-09-09T22:40:41.865 DEBUG:boto:Path: /jkcgbj-19/obj 2022-09-09T22:40:41.866 DEBUG:boto:Data: 2022-09-09T22:40:41.866 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:40:41.867 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:41.867 DEBUG:boto:Port: 8000 2022-09-09T22:40:41.868 DEBUG:boto:Params: {} 2022-09-09T22:40:41.868 DEBUG:boto:Token: None 2022-09-09T22:40:41.869 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/obj 2022-09-09T22:40:41.869 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ= 2022-09-09T22:40:41.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ='} 2022-09-09T22:40:41.914 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Zkehe6bhb4TUKnt33fybBsFSNgQZLrH'), ('x-amz-request-id', 'tx00000fe88906d75659f5f-00631bc0e9-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.915 DEBUG:rgw_multi.tests:zone=test-zone1 version=Zkehe6bhb4TUKnt33fybBsFSNgQZLrH 2022-09-09T22:40:41.916 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:41.916 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:41.917 DEBUG:boto:Method: HEAD 2022-09-09T22:40:41.917 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:41.918 DEBUG:boto:Data: 2022-09-09T22:40:41.918 DEBUG:boto:Headers: {} 2022-09-09T22:40:41.919 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:41.919 DEBUG:boto:Port: 8000 2022-09-09T22:40:41.920 DEBUG:boto:Params: {} 2022-09-09T22:40:41.920 DEBUG:boto:Token: None 2022-09-09T22:40:41.921 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/ 2022-09-09T22:40:41.921 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8= 2022-09-09T22:40:41.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8=', 'Content-Length': '0'} 2022-09-09T22:40:41.926 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d46466cf45a550d4-00631bc0e9-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.927 DEBUG:boto:path=/jkcgbj-19/obj 2022-09-09T22:40:41.927 DEBUG:boto:auth_path=/jkcgbj-19/obj 2022-09-09T22:40:41.928 DEBUG:boto:Method: PUT 2022-09-09T22:40:41.928 DEBUG:boto:Path: /jkcgbj-19/obj 2022-09-09T22:40:41.929 DEBUG:boto:Data: 2022-09-09T22:40:41.930 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:40:41.930 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:41.931 DEBUG:boto:Port: 8000 2022-09-09T22:40:41.931 DEBUG:boto:Params: {} 2022-09-09T22:40:41.932 DEBUG:boto:Token: None 2022-09-09T22:40:41.932 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/obj 2022-09-09T22:40:41.933 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ= 2022-09-09T22:40:41.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ='} 2022-09-09T22:40:41.958 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '2J7HF9LB--ZSe-YuOw.aDXPzP7sEg3b'), ('x-amz-request-id', 'tx000002d22e6a7e81f5628-00631bc0e9-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.959 DEBUG:rgw_multi.tests:zone=test-zone2 version=2J7HF9LB--ZSe-YuOw.aDXPzP7sEg3b 2022-09-09T22:40:41.959 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:41.960 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:41.960 DEBUG:boto:Method: HEAD 2022-09-09T22:40:41.961 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:41.961 DEBUG:boto:Data: 2022-09-09T22:40:41.962 DEBUG:boto:Headers: {} 2022-09-09T22:40:41.962 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:41.963 DEBUG:boto:Port: 8001 2022-09-09T22:40:41.963 DEBUG:boto:Params: {} 2022-09-09T22:40:41.964 DEBUG:boto:Token: None 2022-09-09T22:40:41.965 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/ 2022-09-09T22:40:41.965 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8= 2022-09-09T22:40:41.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:n5LmdLS+3S1UTMm4/ZxUmaAzfi8=', 'Content-Length': '0'} 2022-09-09T22:40:41.970 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c686fd9676e6939-00631bc0e9-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:41.971 DEBUG:boto:path=/jkcgbj-19/obj 2022-09-09T22:40:41.972 DEBUG:boto:auth_path=/jkcgbj-19/obj 2022-09-09T22:40:41.972 DEBUG:boto:Method: PUT 2022-09-09T22:40:41.973 DEBUG:boto:Path: /jkcgbj-19/obj 2022-09-09T22:40:41.973 DEBUG:boto:Data: 2022-09-09T22:40:41.974 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:40:41.974 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:41.975 DEBUG:boto:Port: 8001 2022-09-09T22:40:41.975 DEBUG:boto:Params: {} 2022-09-09T22:40:41.976 DEBUG:boto:Token: None 2022-09-09T22:40:41.976 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:40:41 GMT /jkcgbj-19/obj 2022-09-09T22:40:41.977 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ= 2022-09-09T22:40:41.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:40:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pfiHcy/reJ0ZX79PW6o/4Cb0frQ='} 2022-09-09T22:40:42.026 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'jazHCvOkGnlGOeRvdUICqMSC7euPzig'), ('x-amz-request-id', 'tx00000e3b86bc3b277ecba-00631bc0e9-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:40:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:42.027 DEBUG:rgw_multi.tests:zone=test-zone3 version=jazHCvOkGnlGOeRvdUICqMSC7euPzig 2022-09-09T22:40:42.028 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:42.028 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:42.029 DEBUG:boto:Method: HEAD 2022-09-09T22:40:42.029 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:42.030 DEBUG:boto:Data: 2022-09-09T22:40:42.030 DEBUG:boto:Headers: {} 2022-09-09T22:40:42.031 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:42.031 DEBUG:boto:Port: 8000 2022-09-09T22:40:42.032 DEBUG:boto:Params: {} 2022-09-09T22:40:42.032 DEBUG:boto:Token: None 2022-09-09T22:40:42.033 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:42 GMT /jkcgbj-19/ 2022-09-09T22:40:42.033 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:V6NC6d93MOv9dW5mLpsKmhD6tBw= 2022-09-09T22:40:42.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:V6NC6d93MOv9dW5mLpsKmhD6tBw=', 'Content-Length': '0'} 2022-09-09T22:40:42.039 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b93cd521accfc79-00631bc0ea-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:42.040 DEBUG:boto:path=/jkcgbj-19/obj 2022-09-09T22:40:42.040 DEBUG:boto:auth_path=/jkcgbj-19/obj 2022-09-09T22:40:42.041 DEBUG:boto:Method: PUT 2022-09-09T22:40:42.041 DEBUG:boto:Path: /jkcgbj-19/obj 2022-09-09T22:40:42.042 DEBUG:boto:Data: 2022-09-09T22:40:42.042 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:40:42.043 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:42.043 DEBUG:boto:Port: 8000 2022-09-09T22:40:42.044 DEBUG:boto:Params: {} 2022-09-09T22:40:42.045 DEBUG:boto:Token: None 2022-09-09T22:40:42.045 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:40:42 GMT /jkcgbj-19/obj 2022-09-09T22:40:42.046 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qhy/zjvHdGtu/p1VanLKmlERxuA= 2022-09-09T22:40:42.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:40:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qhy/zjvHdGtu/p1VanLKmlERxuA='} 2022-09-09T22:40:42.072 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'SGmB9jICu3I.Fx74pgovYWN.35m-Ddn'), ('x-amz-request-id', 'tx00000b7c285c928b3926d-00631bc0ea-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:40:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:42.072 DEBUG:rgw_multi.tests:zone=test-zone1 version=SGmB9jICu3I.Fx74pgovYWN.35m-Ddn 2022-09-09T22:40:42.073 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:42.073 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:42.074 DEBUG:boto:Method: HEAD 2022-09-09T22:40:42.074 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:42.075 DEBUG:boto:Data: 2022-09-09T22:40:42.075 DEBUG:boto:Headers: {} 2022-09-09T22:40:42.076 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:42.076 DEBUG:boto:Port: 8000 2022-09-09T22:40:42.077 DEBUG:boto:Params: {} 2022-09-09T22:40:42.077 DEBUG:boto:Token: None 2022-09-09T22:40:42.078 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:42 GMT /jkcgbj-19/ 2022-09-09T22:40:42.079 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:V6NC6d93MOv9dW5mLpsKmhD6tBw= 2022-09-09T22:40:42.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:V6NC6d93MOv9dW5mLpsKmhD6tBw=', 'Content-Length': '0'} 2022-09-09T22:40:42.084 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088c1cd36b40a8d12-00631bc0ea-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:42.085 DEBUG:boto:path=/jkcgbj-19/obj 2022-09-09T22:40:42.085 DEBUG:boto:auth_path=/jkcgbj-19/obj 2022-09-09T22:40:42.086 DEBUG:boto:Method: PUT 2022-09-09T22:40:42.086 DEBUG:boto:Path: /jkcgbj-19/obj 2022-09-09T22:40:42.087 DEBUG:boto:Data: 2022-09-09T22:40:42.087 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:40:42.088 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:42.088 DEBUG:boto:Port: 8000 2022-09-09T22:40:42.089 DEBUG:boto:Params: {} 2022-09-09T22:40:42.089 DEBUG:boto:Token: None 2022-09-09T22:40:42.090 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:40:42 GMT /jkcgbj-19/obj 2022-09-09T22:40:42.090 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qhy/zjvHdGtu/p1VanLKmlERxuA= 2022-09-09T22:40:42.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:40:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qhy/zjvHdGtu/p1VanLKmlERxuA='} 2022-09-09T22:40:42.134 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ACb5CwaFMEkNgP8u9zt2zO4TmQlDmD-'), ('x-amz-request-id', 'tx0000039064184af56bd3c-00631bc0ea-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:40:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:42.135 DEBUG:rgw_multi.tests:zone=test-zone2 version=ACb5CwaFMEkNgP8u9zt2zO4TmQlDmD- 2022-09-09T22:40:42.136 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:42.136 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:42.137 DEBUG:boto:Method: HEAD 2022-09-09T22:40:42.137 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:42.138 DEBUG:boto:Data: 2022-09-09T22:40:42.138 DEBUG:boto:Headers: {} 2022-09-09T22:40:42.139 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:42.139 DEBUG:boto:Port: 8001 2022-09-09T22:40:42.140 DEBUG:boto:Params: {} 2022-09-09T22:40:42.140 DEBUG:boto:Token: None 2022-09-09T22:40:42.141 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:42 GMT /jkcgbj-19/ 2022-09-09T22:40:42.141 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:V6NC6d93MOv9dW5mLpsKmhD6tBw= 2022-09-09T22:40:42.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:V6NC6d93MOv9dW5mLpsKmhD6tBw=', 'Content-Length': '0'} 2022-09-09T22:40:42.148 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000daafea8f3be6785-00631bc0ea-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:42.149 DEBUG:boto:path=/jkcgbj-19/obj 2022-09-09T22:40:42.149 DEBUG:boto:auth_path=/jkcgbj-19/obj 2022-09-09T22:40:42.150 DEBUG:boto:Method: PUT 2022-09-09T22:40:42.151 DEBUG:boto:Path: /jkcgbj-19/obj 2022-09-09T22:40:42.151 DEBUG:boto:Data: 2022-09-09T22:40:42.152 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:40:42.152 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:42.153 DEBUG:boto:Port: 8001 2022-09-09T22:40:42.153 DEBUG:boto:Params: {} 2022-09-09T22:40:42.154 DEBUG:boto:Token: None 2022-09-09T22:40:42.154 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:40:42 GMT /jkcgbj-19/obj 2022-09-09T22:40:42.155 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qhy/zjvHdGtu/p1VanLKmlERxuA= 2022-09-09T22:40:42.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:40:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qhy/zjvHdGtu/p1VanLKmlERxuA='} 2022-09-09T22:40:42.179 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '2Cj8.D8Y0F95S5eWfrqsfmvAfpkh1gV'), ('x-amz-request-id', 'tx000007e8bab948d2ea21e-00631bc0ea-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:40:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:42.180 DEBUG:rgw_multi.tests:zone=test-zone3 version=2Cj8.D8Y0F95S5eWfrqsfmvAfpkh1gV 2022-09-09T22:40:42.180 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:42.181 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:42.181 DEBUG:boto:Method: HEAD 2022-09-09T22:40:42.182 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:42.182 DEBUG:boto:Data: 2022-09-09T22:40:42.183 DEBUG:boto:Headers: {} 2022-09-09T22:40:42.183 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:42.184 DEBUG:boto:Port: 8000 2022-09-09T22:40:42.185 DEBUG:boto:Params: {} 2022-09-09T22:40:42.186 DEBUG:boto:Token: None 2022-09-09T22:40:42.187 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:42 GMT /jkcgbj-19/ 2022-09-09T22:40:42.188 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:V6NC6d93MOv9dW5mLpsKmhD6tBw= 2022-09-09T22:40:42.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:V6NC6d93MOv9dW5mLpsKmhD6tBw=', 'Content-Length': '0'} 2022-09-09T22:40:42.194 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000928d850947feb31d-00631bc0ea-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:42.195 DEBUG:boto:path=/jkcgbj-19/obj 2022-09-09T22:40:42.196 DEBUG:boto:auth_path=/jkcgbj-19/obj 2022-09-09T22:40:42.196 DEBUG:boto:Method: PUT 2022-09-09T22:40:42.197 DEBUG:boto:Path: /jkcgbj-19/obj 2022-09-09T22:40:42.197 DEBUG:boto:Data: 2022-09-09T22:40:42.198 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:40:42.198 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:42.199 DEBUG:boto:Port: 8000 2022-09-09T22:40:42.199 DEBUG:boto:Params: {} 2022-09-09T22:40:42.200 DEBUG:boto:Token: None 2022-09-09T22:40:42.200 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:40:42 GMT /jkcgbj-19/obj 2022-09-09T22:40:42.201 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qhy/zjvHdGtu/p1VanLKmlERxuA= 2022-09-09T22:40:42.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:40:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qhy/zjvHdGtu/p1VanLKmlERxuA='} 2022-09-09T22:40:42.226 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'azmSi4BeBwbPLh8u.YZx--ZDeqpqHSt'), ('x-amz-request-id', 'tx000009de54e037b36c503-00631bc0ea-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:40:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:42.227 DEBUG:rgw_multi.tests:zone=test-zone1 version=azmSi4BeBwbPLh8u.YZx--ZDeqpqHSt 2022-09-09T22:40:42.228 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:42.229 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:42.230 DEBUG:boto:Method: HEAD 2022-09-09T22:40:42.230 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:42.231 DEBUG:boto:Data: 2022-09-09T22:40:42.232 DEBUG:boto:Headers: {} 2022-09-09T22:40:42.233 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:42.233 DEBUG:boto:Port: 8000 2022-09-09T22:40:42.234 DEBUG:boto:Params: {} 2022-09-09T22:40:42.235 DEBUG:boto:Token: None 2022-09-09T22:40:42.236 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:42 GMT /jkcgbj-19/ 2022-09-09T22:40:42.237 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:V6NC6d93MOv9dW5mLpsKmhD6tBw= 2022-09-09T22:40:42.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:V6NC6d93MOv9dW5mLpsKmhD6tBw=', 'Content-Length': '0'} 2022-09-09T22:40:42.242 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1bec8bc21e19190-00631bc0ea-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:42.243 DEBUG:boto:path=/jkcgbj-19/obj 2022-09-09T22:40:42.244 DEBUG:boto:auth_path=/jkcgbj-19/obj 2022-09-09T22:40:42.244 DEBUG:boto:Method: PUT 2022-09-09T22:40:42.245 DEBUG:boto:Path: /jkcgbj-19/obj 2022-09-09T22:40:42.245 DEBUG:boto:Data: 2022-09-09T22:40:42.246 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:40:42.247 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:42.247 DEBUG:boto:Port: 8000 2022-09-09T22:40:42.248 DEBUG:boto:Params: {} 2022-09-09T22:40:42.248 DEBUG:boto:Token: None 2022-09-09T22:40:42.249 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:40:42 GMT /jkcgbj-19/obj 2022-09-09T22:40:42.250 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qhy/zjvHdGtu/p1VanLKmlERxuA= 2022-09-09T22:40:42.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:40:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qhy/zjvHdGtu/p1VanLKmlERxuA='} 2022-09-09T22:40:42.276 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'j.y8xHxKdsNotHvLmJpAUyF0q7o9M01'), ('x-amz-request-id', 'tx0000058e26f1a90500eed-00631bc0ea-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:40:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:42.277 DEBUG:rgw_multi.tests:zone=test-zone2 version=j.y8xHxKdsNotHvLmJpAUyF0q7o9M01 2022-09-09T22:40:42.278 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:42.278 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:42.279 DEBUG:boto:Method: HEAD 2022-09-09T22:40:42.280 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:42.281 DEBUG:boto:Data: 2022-09-09T22:40:42.281 DEBUG:boto:Headers: {} 2022-09-09T22:40:42.281 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:42.282 DEBUG:boto:Port: 8001 2022-09-09T22:40:42.283 DEBUG:boto:Params: {} 2022-09-09T22:40:42.284 DEBUG:boto:Token: None 2022-09-09T22:40:42.285 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:42 GMT /jkcgbj-19/ 2022-09-09T22:40:42.286 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:V6NC6d93MOv9dW5mLpsKmhD6tBw= 2022-09-09T22:40:42.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:V6NC6d93MOv9dW5mLpsKmhD6tBw=', 'Content-Length': '0'} 2022-09-09T22:40:42.292 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a7e4a1e206f02abb-00631bc0ea-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:42.294 DEBUG:boto:path=/jkcgbj-19/obj 2022-09-09T22:40:42.295 DEBUG:boto:auth_path=/jkcgbj-19/obj 2022-09-09T22:40:42.296 DEBUG:boto:Method: PUT 2022-09-09T22:40:42.297 DEBUG:boto:Path: /jkcgbj-19/obj 2022-09-09T22:40:42.298 DEBUG:boto:Data: 2022-09-09T22:40:42.299 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:40:42.300 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:42.301 DEBUG:boto:Port: 8001 2022-09-09T22:40:42.302 DEBUG:boto:Params: {} 2022-09-09T22:40:42.303 DEBUG:boto:Token: None 2022-09-09T22:40:42.304 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:40:42 GMT /jkcgbj-19/obj 2022-09-09T22:40:42.305 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qhy/zjvHdGtu/p1VanLKmlERxuA= 2022-09-09T22:40:42.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:40:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qhy/zjvHdGtu/p1VanLKmlERxuA='} 2022-09-09T22:40:42.330 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'xVpgKZZ4WaWJL6oorArms71ay0GqVG.'), ('x-amz-request-id', 'tx0000035e0b1d60df4a46b-00631bc0ea-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:40:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:42.331 DEBUG:rgw_multi.tests:zone=test-zone3 version=xVpgKZZ4WaWJL6oorArms71ay0GqVG. 2022-09-09T22:40:42.332 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:42.333 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:42.334 DEBUG:boto:Method: HEAD 2022-09-09T22:40:42.334 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:42.335 DEBUG:boto:Data: 2022-09-09T22:40:42.335 DEBUG:boto:Headers: {} 2022-09-09T22:40:42.336 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:42.336 DEBUG:boto:Port: 8000 2022-09-09T22:40:42.337 DEBUG:boto:Params: {} 2022-09-09T22:40:42.337 DEBUG:boto:Token: None 2022-09-09T22:40:42.338 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:42 GMT /jkcgbj-19/ 2022-09-09T22:40:42.339 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:V6NC6d93MOv9dW5mLpsKmhD6tBw= 2022-09-09T22:40:42.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:V6NC6d93MOv9dW5mLpsKmhD6tBw=', 'Content-Length': '0'} 2022-09-09T22:40:42.345 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007becf0bf11b15d37-00631bc0ea-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:42.346 DEBUG:boto:path=/jkcgbj-19/obj 2022-09-09T22:40:42.346 DEBUG:boto:auth_path=/jkcgbj-19/obj 2022-09-09T22:40:42.347 DEBUG:boto:Method: PUT 2022-09-09T22:40:42.348 DEBUG:boto:Path: /jkcgbj-19/obj 2022-09-09T22:40:42.348 DEBUG:boto:Data: 2022-09-09T22:40:42.349 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:40:42.349 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:40:42.350 DEBUG:boto:Port: 8000 2022-09-09T22:40:42.350 DEBUG:boto:Params: {} 2022-09-09T22:40:42.351 DEBUG:boto:Token: None 2022-09-09T22:40:42.351 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:40:42 GMT /jkcgbj-19/obj 2022-09-09T22:40:42.352 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qhy/zjvHdGtu/p1VanLKmlERxuA= 2022-09-09T22:40:42.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:40:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qhy/zjvHdGtu/p1VanLKmlERxuA='} 2022-09-09T22:40:42.402 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'SMzjiECzC4fO29rmBSIjBkC0Ca8iUcg'), ('x-amz-request-id', 'tx000007c8e387c4805d17e-00631bc0ea-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:40:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:42.403 DEBUG:rgw_multi.tests:zone=test-zone1 version=SMzjiECzC4fO29rmBSIjBkC0Ca8iUcg 2022-09-09T22:40:42.404 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:42.404 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:42.405 DEBUG:boto:Method: HEAD 2022-09-09T22:40:42.405 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:42.406 DEBUG:boto:Data: 2022-09-09T22:40:42.406 DEBUG:boto:Headers: {} 2022-09-09T22:40:42.407 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:42.407 DEBUG:boto:Port: 8000 2022-09-09T22:40:42.408 DEBUG:boto:Params: {} 2022-09-09T22:40:42.408 DEBUG:boto:Token: None 2022-09-09T22:40:42.409 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:42 GMT /jkcgbj-19/ 2022-09-09T22:40:42.409 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:V6NC6d93MOv9dW5mLpsKmhD6tBw= 2022-09-09T22:40:42.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:V6NC6d93MOv9dW5mLpsKmhD6tBw=', 'Content-Length': '0'} 2022-09-09T22:40:42.415 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f63345bf678ea7f-00631bc0ea-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:42.415 DEBUG:boto:path=/jkcgbj-19/obj 2022-09-09T22:40:42.416 DEBUG:boto:auth_path=/jkcgbj-19/obj 2022-09-09T22:40:42.416 DEBUG:boto:Method: PUT 2022-09-09T22:40:42.417 DEBUG:boto:Path: /jkcgbj-19/obj 2022-09-09T22:40:42.417 DEBUG:boto:Data: 2022-09-09T22:40:42.418 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:40:42.418 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:40:42.419 DEBUG:boto:Port: 8000 2022-09-09T22:40:42.419 DEBUG:boto:Params: {} 2022-09-09T22:40:42.420 DEBUG:boto:Token: None 2022-09-09T22:40:42.421 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:40:42 GMT /jkcgbj-19/obj 2022-09-09T22:40:42.421 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qhy/zjvHdGtu/p1VanLKmlERxuA= 2022-09-09T22:40:42.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:40:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qhy/zjvHdGtu/p1VanLKmlERxuA='} 2022-09-09T22:40:42.446 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'BTjb4sYKId6rGQlbCSwrmWlW433zzBq'), ('x-amz-request-id', 'tx000004316977c740dcae6-00631bc0ea-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:40:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:42.447 DEBUG:rgw_multi.tests:zone=test-zone2 version=BTjb4sYKId6rGQlbCSwrmWlW433zzBq 2022-09-09T22:40:42.447 DEBUG:boto:path=/jkcgbj-19/ 2022-09-09T22:40:42.448 DEBUG:boto:auth_path=/jkcgbj-19/ 2022-09-09T22:40:42.448 DEBUG:boto:Method: HEAD 2022-09-09T22:40:42.449 DEBUG:boto:Path: /jkcgbj-19/ 2022-09-09T22:40:42.449 DEBUG:boto:Data: 2022-09-09T22:40:42.450 DEBUG:boto:Headers: {} 2022-09-09T22:40:42.450 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:42.451 DEBUG:boto:Port: 8001 2022-09-09T22:40:42.451 DEBUG:boto:Params: {} 2022-09-09T22:40:42.452 DEBUG:boto:Token: None 2022-09-09T22:40:42.452 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:40:42 GMT /jkcgbj-19/ 2022-09-09T22:40:42.453 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:V6NC6d93MOv9dW5mLpsKmhD6tBw= 2022-09-09T22:40:42.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:40:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:V6NC6d93MOv9dW5mLpsKmhD6tBw=', 'Content-Length': '0'} 2022-09-09T22:40:42.459 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da17cc6fb881d892-00631bc0ea-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:40:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:42.459 DEBUG:boto:path=/jkcgbj-19/obj 2022-09-09T22:40:42.460 DEBUG:boto:auth_path=/jkcgbj-19/obj 2022-09-09T22:40:42.460 DEBUG:boto:Method: PUT 2022-09-09T22:40:42.461 DEBUG:boto:Path: /jkcgbj-19/obj 2022-09-09T22:40:42.461 DEBUG:boto:Data: 2022-09-09T22:40:42.462 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-09T22:40:42.462 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:40:42.463 DEBUG:boto:Port: 8001 2022-09-09T22:40:42.464 DEBUG:boto:Params: {} 2022-09-09T22:40:42.464 DEBUG:boto:Token: None 2022-09-09T22:40:42.465 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 09 Sep 2022 22:40:42 GMT /jkcgbj-19/obj 2022-09-09T22:40:42.465 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qhy/zjvHdGtu/p1VanLKmlERxuA= 2022-09-09T22:40:42.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:40:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qhy/zjvHdGtu/p1VanLKmlERxuA='} 2022-09-09T22:40:42.510 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'wuQ8QzhPSSeYSBTMG3vMW6krRpkD9oX'), ('x-amz-request-id', 'tx000009557aa86079c4176-00631bc0ea-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:40:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:40:42.511 DEBUG:rgw_multi.tests:zone=test-zone3 version=wuQ8QzhPSSeYSBTMG3vMW6krRpkD9oX 2022-09-09T22:40:42.512 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jkcgbj-19 2022-09-09T22:40:42.512 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:40:42.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:40:42.514 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:40:42.567 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:40:42.567 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:40:42.655 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:40:42.648+0000 7fed41621dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:40:42.656 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.10.9, , , , , ] 2022-09-09T22:40:42.656 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.10.9, , , 00000000020.50.11, , ] 2022-09-09T22:40:47.654 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:40:47.648+0000 7fed41621dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:40:47.654 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.10.9, , , , , ] 2022-09-09T22:40:47.655 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.10.9, , , 00000000020.50.11, , ] 2022-09-09T22:40:52.657 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:40:52.652+0000 7fed41621dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:40:52.658 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.10.9, , , , , ] 2022-09-09T22:40:52.658 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.10.9, , , 00000000020.50.11, , ] 2022-09-09T22:40:57.660 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:40:57.656+0000 7fed41621dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:40:57.660 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.10.9, , , , , ] 2022-09-09T22:40:57.661 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.10.9, , , 00000000020.50.11, , ] 2022-09-09T22:41:02.663 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:41:02.656+0000 7fed41621dc0 1 bucket sync caught up with source: 2022-09-09T22:41:02.664 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.10.9, , , 00000000032.86.5, , ] 2022-09-09T22:41:02.664 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.10.9, , , 00000000020.50.11, , ] 2022-09-09T22:41:02.664 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:41:02.656+0000 7fed41621dc0 0 bucket checkpoint complete 2022-09-09T22:41:02.693 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jkcgbj-19 2022-09-09T22:41:02.694 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:41:02.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:41:02.695 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:41:02.753 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:41:02.754 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:41:02.815 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:41:02.809+0000 7f7e8a496dc0 1 bucket sync caught up with source: 2022-09-09T22:41:02.815 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , 00000000004.10.9, , , 00000000034.92.5, , ] 2022-09-09T22:41:02.815 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , 00000000004.10.9, , , 00000000034.92.5, , ] 2022-09-09T22:41:02.816 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:41:02.809+0000 7f7e8a496dc0 0 bucket checkpoint complete 2022-09-09T22:41:02.842 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jkcgbj-19 2022-09-09T22:41:02.843 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:41:02.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:41:02.844 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:41:02.899 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:41:02.900 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:41:02.972 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:41:02.968+0000 7f7b4d651dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:41:02.973 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.10.9, , , 00000000020.50.11, , ] 2022-09-09T22:41:02.973 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.10.9, , , 00000000034.92.5, , ] 2022-09-09T22:41:07.975 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:41:07.968+0000 7f7b4d651dc0 1 bucket sync caught up with source: 2022-09-09T22:41:07.976 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.10.9, , , 00000000034.92.5, , ] 2022-09-09T22:41:07.976 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.10.9, , , 00000000034.92.5, , ] 2022-09-09T22:41:07.976 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:41:07.968+0000 7f7b4d651dc0 0 bucket checkpoint complete 2022-09-09T22:41:08.006 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jkcgbj-19 2022-09-09T22:41:08.007 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:41:08.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:41:08.008 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:41:08.063 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:41:08.063 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:41:08.128 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:41:08.121+0000 7f2ea0062dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:41:08.128 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000015.47.5, , ] 2022-09-09T22:41:08.129 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000042.118.8, , ] 2022-09-09T22:41:13.131 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:41:13.125+0000 7f2ea0062dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:41:13.131 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000015.47.5, , ] 2022-09-09T22:41:13.132 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000042.118.8, , ] 2022-09-09T22:41:18.134 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:41:18.129+0000 7f2ea0062dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:41:18.134 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000015.47.5, , ] 2022-09-09T22:41:18.135 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000042.118.8, , ] 2022-09-09T22:41:23.137 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:41:23.133+0000 7f2ea0062dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:41:23.138 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000020.56.11, , ] 2022-09-09T22:41:23.138 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000042.118.8, , ] 2022-09-09T22:41:28.140 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:41:28.137+0000 7f2ea0062dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:41:28.141 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000020.56.11, , ] 2022-09-09T22:41:28.141 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000042.118.8, , ] 2022-09-09T22:41:33.144 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:41:33.136+0000 7f2ea0062dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:41:33.144 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000020.56.11, , ] 2022-09-09T22:41:33.145 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000042.118.8, , ] 2022-09-09T22:41:38.146 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:41:38.140+0000 7f2ea0062dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:41:38.147 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000020.56.11, , ] 2022-09-09T22:41:38.147 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000042.118.8, , ] 2022-09-09T22:41:43.149 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:41:43.144+0000 7f2ea0062dc0 1 bucket sync caught up with source: 2022-09-09T22:41:43.150 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000045.124.9, , ] 2022-09-09T22:41:43.150 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000042.118.8, , ] 2022-09-09T22:41:43.150 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:41:43.144+0000 7f2ea0062dc0 0 bucket checkpoint complete 2022-09-09T22:41:43.178 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jkcgbj-19 2022-09-09T22:41:43.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:41:43.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:41:43.181 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:41:43.234 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:41:43.234 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:41:43.299 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:41:43.292+0000 7f51db68cdc0 1 bucket sync caught up with source: 2022-09-09T22:41:43.300 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000045.124.9, , ] 2022-09-09T22:41:43.300 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000045.124.9, , ] 2022-09-09T22:41:43.301 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:41:43.292+0000 7f51db68cdc0 0 bucket checkpoint complete 2022-09-09T22:41:43.326 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jkcgbj-19 2022-09-09T22:41:43.327 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jkcgbj-19 2022-09-09T22:41:43.328 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:41:43.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:41:43.329 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:41:43.382 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:41:43.383 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:41:43.448 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:41:43.444+0000 7f31f3684dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:41:43.449 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000043.121.9, , ] 2022-09-09T22:41:43.449 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:41:48.451 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:41:48.444+0000 7f31f3684dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:41:48.451 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000043.121.9, , ] 2022-09-09T22:41:48.452 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:41:53.454 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:41:53.448+0000 7f31f3684dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:41:53.455 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000043.121.9, , ] 2022-09-09T22:41:53.455 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:41:58.457 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:41:58.452+0000 7f31f3684dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:41:58.458 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000043.121.9, , ] 2022-09-09T22:41:58.459 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:42:03.460 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:42:03.452+0000 7f31f3684dc0 1 bucket sync caught up with source: 2022-09-09T22:42:03.460 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:42:03.461 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:42:03.461 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:42:03.452+0000 7f31f3684dc0 0 bucket checkpoint complete 2022-09-09T22:42:03.491 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jkcgbj-19 2022-09-09T22:42:03.491 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:42:03.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:42:03.493 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:42:03.546 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:42:03.546 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:42:03.625 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:42:03.622+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:42:03.625 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:42:03.626 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:42:08.628 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:42:08.622+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:42:08.628 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:42:08.629 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:42:13.626 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:42:13.622+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:42:13.627 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:42:13.627 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:42:18.629 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:42:18.626+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:42:18.630 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:42:18.631 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:42:23.631 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:42:23.626+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:42:23.632 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:42:23.632 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:42:28.633 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:42:28.630+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:42:28.634 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:42:28.634 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:42:33.635 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:42:33.629+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:42:33.636 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:42:33.636 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:42:38.638 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:42:38.633+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:42:38.638 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:42:38.639 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:42:43.641 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:42:43.637+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:42:43.641 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:42:43.641 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:42:48.643 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:42:48.637+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:42:48.644 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:42:48.644 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:42:53.646 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:42:53.641+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:42:53.647 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:42:53.647 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:42:58.649 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:42:58.645+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:42:58.649 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:42:58.650 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:43:03.651 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:43:03.645+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:43:03.652 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:43:03.652 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:43:08.654 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:43:08.649+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:43:08.655 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:43:08.655 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:43:13.656 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:43:13.652+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:43:13.657 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:43:13.657 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:43:18.659 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:43:18.652+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:43:18.659 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:43:18.660 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:43:23.661 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:43:23.656+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:43:23.661 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:43:23.661 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:43:28.664 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:43:28.656+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:43:28.664 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:43:28.665 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:43:33.665 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:43:33.660+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:43:33.666 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:43:33.666 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:43:38.668 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:43:38.660+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:43:38.669 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:43:38.669 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:43:43.671 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:43:43.664+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:43:43.671 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:43:43.671 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:43:48.673 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:43:48.668+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:43:48.674 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:43:48.674 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:43:53.676 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:43:53.671+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:43:53.677 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:43:53.677 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:43:58.679 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:43:58.671+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:43:58.680 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:43:58.680 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:44:03.684 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:44:03.675+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:44:03.684 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:44:03.685 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:44:08.685 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:44:08.679+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:44:08.685 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:44:08.686 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:44:13.687 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:44:13.679+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:44:13.688 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:44:13.688 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:44:18.690 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:44:18.687+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:44:18.691 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:44:18.692 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:44:23.693 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:44:23.691+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:44:23.694 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:44:23.694 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:44:28.696 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:44:28.691+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:44:28.697 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:44:28.697 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:44:33.699 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:44:33.695+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:44:33.700 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:44:33.701 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:44:38.701 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:44:38.698+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:44:38.702 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:44:38.703 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:44:43.704 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:44:43.698+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:44:43.704 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:44:43.705 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:44:48.706 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:44:48.702+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:44:48.707 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:44:48.707 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:44:53.709 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:44:53.706+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:44:53.710 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:44:53.710 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:44:58.711 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:44:58.706+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:44:58.712 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:44:58.712 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:45:03.714 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:45:03.710+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:45:03.715 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:45:03.715 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:45:08.717 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:45:08.714+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:45:08.717 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:45:08.718 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:45:13.719 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:45:13.714+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:45:13.720 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:45:13.720 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:45:18.722 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:45:18.717+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:45:18.723 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:45:18.723 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:45:23.725 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:45:23.721+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:45:23.726 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:45:23.726 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:45:28.728 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:45:28.721+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:45:28.728 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:45:28.729 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:45:33.730 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:45:33.725+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:45:33.731 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:45:33.731 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:45:38.733 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:45:38.729+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:45:38.733 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:45:38.734 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:45:43.735 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:45:43.729+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:45:43.735 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:45:43.736 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:45:48.737 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:45:48.733+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:45:48.738 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:45:48.738 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:45:53.740 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:45:53.737+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:45:53.741 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:45:53.741 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:45:58.744 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:45:58.737+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:45:58.745 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:45:58.745 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:46:03.747 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:46:03.740+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:46:03.748 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:46:03.748 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:46:08.750 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:46:08.744+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:46:08.751 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:46:08.752 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:46:13.752 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:46:13.744+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:46:13.753 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:46:13.753 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:46:18.756 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:46:18.748+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:46:18.756 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:46:18.757 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:46:23.759 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:46:23.752+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:46:23.760 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:46:23.760 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:46:28.762 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:46:28.756+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:46:28.763 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:46:28.764 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:46:33.766 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:46:33.760+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:46:33.767 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:46:33.767 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:46:38.769 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:46:38.764+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:46:38.769 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:46:38.770 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:46:43.771 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:46:43.764+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:46:43.771 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:46:43.772 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:46:48.773 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:46:48.767+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:46:48.774 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:46:48.774 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:46:53.776 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:46:53.771+0000 7f7161639dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:46:53.777 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000020.57.11, , ] 2022-09-09T22:46:53.777 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000044.123.9, , ] 2022-09-09T22:46:58.779 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:46:58.771+0000 7f7161639dc0 0 bucket checkpoint timed out waiting for incremental sync to catch up 2022-09-09T22:46:58.780 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:46:58.771+0000 7f7161639dc0 0 bucket sync checkpoint failed: (110) Connection timed out 2022-09-09T22:46:58.781 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:46:58.771+0000 7f7161639dc0 -1 bucket sync checkpoint failed: (110) Connection timed out 2022-09-09T22:46:58.806 DEBUG:teuthology.orchestra.run:got remote process result: 110 2022-09-09T22:46:58.813 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... FAIL 2022-09-09T22:46:58.818 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:46:58.819 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:46:58.820 DEBUG:boto:Method: PUT 2022-09-09T22:46:58.822 DEBUG:boto:Path: /jkcgbj-20/ 2022-09-09T22:46:58.823 DEBUG:boto:Data: 2022-09-09T22:46:58.824 DEBUG:boto:Headers: {} 2022-09-09T22:46:58.825 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:46:58.826 DEBUG:boto:Port: 8000 2022-09-09T22:46:58.827 DEBUG:boto:Params: {} 2022-09-09T22:46:58.828 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T22:46:58.829 DEBUG:boto:Token: None 2022-09-09T22:46:58.830 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:46:58 GMT /jkcgbj-20/ 2022-09-09T22:46:58.830 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:5Tll2fx0HFS/LIG5JuXJUB9Bh1s= 2022-09-09T22:46:58.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:46:58 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:5Tll2fx0HFS/LIG5JuXJUB9Bh1s=', 'Content-Length': '0'} 2022-09-09T22:46:58.863 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000749dde80fe801426-00631bc262-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:46:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:46:58.864 DEBUG:rgw_multi.tests:created bucket=jkcgbj-20 2022-09-09T22:46:58.865 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:46:58.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:46:58.867 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:46:58.911 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:46:58.911 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:46:58.962 INFO:teuthology.orchestra.run.smithi005.stderr:No --period given, using current period=503b1d61-4513-48c8-a308-17b76219610c 2022-09-09T22:46:58.983 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T22:46:58.983 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:46:58.984 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:46:58.984 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:28:06.146147Z" 2022-09-09T22:46:58.984 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:46:58.985 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:46:58.985 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:46:58.985 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:46:58.985 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:46:58.986 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:46:58.986 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:46:58.987 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:30:37.757014Z" 2022-09-09T22:46:58.987 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:46:58.987 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:46:58.988 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662763618.858372_97.1", 2022-09-09T22:46:58.988 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:46:58.858372Z" 2022-09-09T22:46:58.988 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:46:58.989 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T22:46:58.990 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763618.858372_97.1'} 2022-09-09T22:46:58.991 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-09T22:46:58.991 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:46:58.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:46:58.992 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:46:59.039 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:46:59.040 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:46:59.116 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:46:59.107+0000 7fb339540dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:46:59.118 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:46:59.111+0000 7fb339540dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:46:59.147 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:46:59.148 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:46:59.149 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:46:59.150 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:46:59.150 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:46:59.150 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:46:59.151 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:46:59.151 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:46:59.152 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:46:59.152 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:46:59.152 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:46:59.153 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:46:59.153 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:46:59.153 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:46:59.153 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:46:59.154 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:46:59.154 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:46:59.154 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:46:59.155 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:46:59.155 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:46:59.155 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:46:59.155 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:46:59.156 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:46:59.156 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:46:59.156 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:46:59.156 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:46:59.157 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:46:59.157 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:46:59.157 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:46:59.157 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:46:59.158 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:46:59.158 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:46:59.158 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:46:59.159 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:46:59.159 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:46:59.159 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:46:59.159 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:46:59.160 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:46:59.160 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:46:59.160 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:46:59.161 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:46:59.161 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:46:59.161 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:46:59.161 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:46:59.162 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:46:59.162 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:46:59.162 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:46:59.162 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:46:59.163 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:46:59.163 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763618.858372_97.1", 2022-09-09T22:46:59.163 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:46:59.164 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:46:59.164 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:46:59.164 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:46:58.858372Z", 2022-09-09T22:46:59.165 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:46:59.165 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:46:59.165 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:46:59.166 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:46:59.166 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:46:59.167 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:46:59.167 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:46:59.167 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:46:59.167 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:46:59.168 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:46:59Z" 2022-09-09T22:46:59.168 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:46:59.169 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763618.858372_97.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:46:58.858372Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:46:59Z" } 2022-09-09T22:46:59.170 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763618.858372_97.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:46:58.858372Z', 'realm_epoch': 2}}] 2022-09-09T22:46:59.171 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763618.858372_97.1'} 2022-09-09T22:46:59.171 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763618.858372_97.1'} 2022-09-09T22:46:59.172 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-09T22:46:59.172 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-09T22:46:59.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:46:59.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:46:59.175 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:46:59.222 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:46:59.222 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:46:59.273 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:46:59.266+0000 7f3fad1f2dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:46:59.275 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:46:59.266+0000 7f3fad1f2dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:46:59.307 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:46:59.308 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T22:46:59.308 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T22:46:59.308 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T22:46:59.309 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T22:46:59.309 INFO:teuthology.orchestra.run.smithi005.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:46:59.309 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:46:59.309 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:46:59.310 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T22:46:59.310 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:46:59.310 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T22:46:59.311 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:46:59.311 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:46:59.311 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:46:59.311 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:46:59.312 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 1, 2022-09-09T22:46:59.312 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:46:59.312 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:46:59.312 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:46:59.313 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:46:59.313 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:46:59.313 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:46:59.314 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T22:46:59.314 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:46:59.315 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:46:59.315 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:46:59.315 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:46:59.316 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:46:59.316 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:46:59.316 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:46:59.317 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:46:59.317 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:46:59.318 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:46:59.318 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:46:59.319 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T22:46:59.319 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:46:59.319 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:46:59.320 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:46:59.320 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:46:59.320 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:46:59.321 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:46:59.321 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:46:59.321 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:46:59.322 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:46:59.322 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:46:59.322 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:46:59.323 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T22:46:59.323 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:46:59.323 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:46:59.324 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662763618.858372_97.1", 2022-09-09T22:46:59.324 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:46:59.324 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:46:59.324 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:46:59.325 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:46:58.858372Z", 2022-09-09T22:46:59.325 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:46:59.326 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:46:59.326 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:46:59.327 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:46:59.327 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:46:59.327 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T22:46:59.328 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 1, 2022-09-09T22:46:59.328 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 1 2022-09-09T22:46:59.329 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:46:59.329 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T22:46:59Z" 2022-09-09T22:46:59.330 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:46:59.330 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763618.858372_97.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:46:58.858372Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:46:59Z" } 2022-09-09T22:46:59.331 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763618.858372_97.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:46:58.858372Z', 'realm_epoch': 2}}] 2022-09-09T22:46:59.332 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763618.858372_97.1'} 2022-09-09T22:46:59.333 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763618.858372_97.1'} 2022-09-09T22:46:59.334 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-09T22:46:59.335 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-09T22:46:59.336 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:46:59.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:46:59.337 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:46:59.385 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:46:59.386 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:46:59.456 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:46:59.451+0000 7f547682cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:46:59.458 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:46:59.451+0000 7f547682cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:46:59.484 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:46:59.484 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:46:59.485 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:46:59.485 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:46:59.486 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:46:59.486 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:46:59.488 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:46:59.488 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:46:59.488 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:46:59.489 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:46:59.489 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:46:59.489 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:46:59.489 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:46:59.490 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:46:59.490 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:46:59.491 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:46:59.491 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:46:59.492 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:46:59.492 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:46:59.493 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:46:59.493 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:46:59.494 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:46:59.494 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:46:59.495 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:46:59.496 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:46:59.497 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:46:59.498 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:46:59.498 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:46:59.499 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:46:59.499 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:46:59.500 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:46:59.501 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:46:59.502 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:46:59.502 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:46:59.503 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:46:59.503 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:46:59.504 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:46:59.504 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:46:59.505 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:46:59.506 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:46:59.507 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:46:59.507 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:46:59.508 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:46:59.508 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:46:59.509 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:46:59.509 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:46:59.510 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:46:59.510 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:46:59.511 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:46:59.511 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763177.721058_92.1", 2022-09-09T22:46:59.511 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:46:59.512 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:46:59.512 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:46:59.513 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:39:37.721058Z", 2022-09-09T22:46:59.513 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:46:59.513 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:46:59.514 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:46:59.514 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:46:59.514 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:46:59.515 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:46:59.515 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:46:59.515 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:46:59.515 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:46:59.516 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:46:59Z" 2022-09-09T22:46:59.516 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:46:59.518 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763177.721058_92.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:39:37.721058Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:46:59Z" } 2022-09-09T22:46:59.519 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763177.721058_92.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:39:37.721058Z', 'realm_epoch': 2}}] 2022-09-09T22:46:59.520 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763618.858372_97.1'} 2022-09-09T22:46:59.521 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763177.721058_92.1'} 2022-09-09T22:46:59.522 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662763618.858372_97.1 target=1_1662763177.721058_92.1 2022-09-09T22:47:04.524 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:47:04.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:47:04.525 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:47:04.575 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:47:04.576 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:47:04.634 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:47:04.627+0000 7ffa87c59dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:47:04.636 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:47:04.631+0000 7ffa87c59dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:47:04.663 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:47:04.664 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:47:04.664 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:47:04.665 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:47:04.665 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:47:04.665 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:47:04.666 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:47:04.666 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:47:04.666 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:47:04.667 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:47:04.667 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:47:04.667 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:47:04.668 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:47:04.668 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:47:04.668 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:47:04.669 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:47:04.669 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:47:04.670 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:47:04.670 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:47:04.670 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:47:04.671 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:47:04.671 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:47:04.671 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:47:04.672 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:47:04.672 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:47:04.672 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:47:04.673 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:47:04.673 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:47:04.673 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:47:04.674 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:47:04.674 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:47:04.674 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:47:04.674 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:47:04.675 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:47:04.675 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:47:04.675 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:47:04.676 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:47:04.676 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:47:04.676 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:47:04.677 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:47:04.677 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:47:04.677 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:47:04.678 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:47:04.678 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:47:04.678 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:47:04.679 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:47:04.679 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:47:04.680 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:47:04.680 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:47:04.680 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763618.858372_97.1", 2022-09-09T22:47:04.681 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:47:04.681 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:47:04.681 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:47:04.682 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:46:58.858372Z", 2022-09-09T22:47:04.682 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:47:04.682 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:47:04.683 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:47:04.683 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:47:04.683 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:47:04.684 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:47:04.684 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:47:04.684 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:47:04.684 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:47:04.685 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:47:04Z" 2022-09-09T22:47:04.685 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:47:04.687 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763618.858372_97.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:46:58.858372Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:47:04Z" } 2022-09-09T22:47:04.688 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763618.858372_97.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:46:58.858372Z', 'realm_epoch': 2}}] 2022-09-09T22:47:04.688 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763618.858372_97.1'} 2022-09-09T22:47:04.689 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763618.858372_97.1'} 2022-09-09T22:47:04.689 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-09T22:47:04.690 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:47:04.691 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:47:04.692 DEBUG:boto:Method: HEAD 2022-09-09T22:47:04.692 DEBUG:boto:Path: /jkcgbj-20/ 2022-09-09T22:47:04.693 DEBUG:boto:Data: 2022-09-09T22:47:04.693 DEBUG:boto:Headers: {} 2022-09-09T22:47:04.694 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:47:04.695 DEBUG:boto:Port: 8000 2022-09-09T22:47:04.695 DEBUG:boto:Params: {} 2022-09-09T22:47:04.696 DEBUG:boto:Token: None 2022-09-09T22:47:04.697 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:47:04 GMT /jkcgbj-20/ 2022-09-09T22:47:04.697 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:S8DllK6Awz22jbXYHaNK4runIOI= 2022-09-09T22:47:04.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:47:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:S8DllK6Awz22jbXYHaNK4runIOI=', 'Content-Length': '0'} 2022-09-09T22:47:04.704 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca3618e0226130a7-00631bc268-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:47:04.706 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:47:04.707 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:47:04.708 DEBUG:boto:Method: PUT 2022-09-09T22:47:04.709 DEBUG:boto:Path: /jkcgbj-20/obj 2022-09-09T22:47:04.710 DEBUG:boto:Data: 2022-09-09T22:47:04.711 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-09T22:47:04.711 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:47:04.712 DEBUG:boto:Port: 8000 2022-09-09T22:47:04.713 DEBUG:boto:Params: {} 2022-09-09T22:47:04.714 DEBUG:boto:Token: None 2022-09-09T22:47:04.714 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 09 Sep 2022 22:47:04 GMT /jkcgbj-20/obj 2022-09-09T22:47:04.715 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:mVsHiCl8th80zBDN/PDUyev/eSg= 2022-09-09T22:47:04.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:47:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:mVsHiCl8th80zBDN/PDUyev/eSg='} 2022-09-09T22:47:04.733 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c8b3e36a91629aa4-00631bc268-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:47:04.734 DEBUG:rgw_multi.tests:created initial version id=None 2022-09-09T22:47:04.735 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jkcgbj-20 2022-09-09T22:47:04.736 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:47:04.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:47:04.737 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:47:04.792 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:47:04.792 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:47:04.856 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:47:04.851+0000 7f9ad5982dc0 1 waiting to reach incremental sync.. 2022-09-09T22:47:09.858 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:47:09.851+0000 7f9ad5982dc0 1 waiting to reach incremental sync.. 2022-09-09T22:47:14.859 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:47:14.855+0000 7f9ad5982dc0 1 waiting to reach incremental sync.. 2022-09-09T22:47:19.860 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:47:19.855+0000 7f9ad5982dc0 1 waiting to reach incremental sync.. 2022-09-09T22:47:24.863 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:47:24.859+0000 7f9ad5982dc0 1 bucket sync caught up with source: 2022-09-09T22:47:24.864 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000001.23.5, , ] 2022-09-09T22:47:24.864 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000001.23.5, , ] 2022-09-09T22:47:24.864 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:47:24.859+0000 7f9ad5982dc0 0 bucket checkpoint complete 2022-09-09T22:47:24.890 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jkcgbj-20 2022-09-09T22:47:24.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:47:24.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:47:24.892 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:47:24.953 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:47:24.954 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:47:25.017 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:47:25.010+0000 7fb2f9d1adc0 1 waiting to reach incremental sync.. 2022-09-09T22:47:30.018 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:47:30.010+0000 7fb2f9d1adc0 1 waiting to reach incremental sync.. 2022-09-09T22:47:35.019 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:47:35.014+0000 7fb2f9d1adc0 1 waiting to reach incremental sync.. 2022-09-09T22:47:40.020 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:47:40.014+0000 7fb2f9d1adc0 1 waiting to reach incremental sync.. 2022-09-09T22:47:45.021 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:47:45.018+0000 7fb2f9d1adc0 1 waiting to reach incremental sync.. 2022-09-09T22:47:50.022 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:47:50.013+0000 7fb2f9d1adc0 1 waiting to reach incremental sync.. 2022-09-09T22:47:55.023 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:47:55.017+0000 7fb2f9d1adc0 1 waiting to reach incremental sync.. 2022-09-09T22:48:00.024 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:48:00.017+0000 7fb2f9d1adc0 1 waiting to reach incremental sync.. 2022-09-09T22:48:05.028 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:48:05.021+0000 7fb2f9d1adc0 1 bucket sync caught up with source: 2022-09-09T22:48:05.029 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000001.23.5, , ] 2022-09-09T22:48:05.030 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000001.23.5, , ] 2022-09-09T22:48:05.030 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:48:05.021+0000 7fb2f9d1adc0 0 bucket checkpoint complete 2022-09-09T22:48:05.057 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jkcgbj-20 2022-09-09T22:48:05.059 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:48:05.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:48:05.060 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:48:05.115 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:48:05.116 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:48:05.182 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:48:05.178+0000 7fb13f985dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:48:05.183 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , , , ] 2022-09-09T22:48:05.183 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000001.23.5, , ] 2022-09-09T22:48:10.186 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:48:10.182+0000 7fb13f985dc0 1 bucket sync caught up with source: 2022-09-09T22:48:10.186 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000001.23.5, , ] 2022-09-09T22:48:10.187 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000001.23.5, , ] 2022-09-09T22:48:10.187 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:48:10.182+0000 7fb13f985dc0 0 bucket checkpoint complete 2022-09-09T22:48:10.213 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jkcgbj-20 2022-09-09T22:48:10.213 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:48:10.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:48:10.215 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:48:10.269 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:48:10.270 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:48:10.337 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:48:10.333+0000 7f52cc990dc0 1 bucket sync caught up with source: 2022-09-09T22:48:10.338 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000001.31.5, , ] 2022-09-09T22:48:10.338 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000001.31.5, , ] 2022-09-09T22:48:10.338 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:48:10.333+0000 7f52cc990dc0 0 bucket checkpoint complete 2022-09-09T22:48:10.366 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jkcgbj-20 2022-09-09T22:48:10.367 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:48:10.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:48:10.368 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:48:10.421 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:48:10.421 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:48:10.488 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:48:10.485+0000 7f0141f0ddc0 1 bucket sync caught up with source: 2022-09-09T22:48:10.489 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000001.31.5, , ] 2022-09-09T22:48:10.489 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000001.31.5, , ] 2022-09-09T22:48:10.489 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:48:10.485+0000 7f0141f0ddc0 0 bucket checkpoint complete 2022-09-09T22:48:10.517 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jkcgbj-20 2022-09-09T22:48:10.518 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jkcgbj-20 2022-09-09T22:48:10.518 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:48:10.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:48:10.519 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:48:10.571 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:48:10.571 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:48:10.637 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:48:10.633+0000 7f5d509e2dc0 1 bucket sync caught up with source: 2022-09-09T22:48:10.637 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000001.46.5, , ] 2022-09-09T22:48:10.637 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000001.46.5, , ] 2022-09-09T22:48:10.638 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:48:10.633+0000 7f5d509e2dc0 0 bucket checkpoint complete 2022-09-09T22:48:10.661 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jkcgbj-20 2022-09-09T22:48:10.662 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:48:10.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:48:10.663 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:48:10.718 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:48:10.718 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:48:10.798 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:48:10.794+0000 7f744e5aadc0 1 bucket sync caught up with source: 2022-09-09T22:48:10.798 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000001.46.5, , ] 2022-09-09T22:48:10.799 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000001.46.5, , ] 2022-09-09T22:48:10.799 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:48:10.794+0000 7f744e5aadc0 0 bucket checkpoint complete 2022-09-09T22:48:10.829 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jkcgbj-20 2022-09-09T22:48:10.830 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-20 zones={test-zone2, test-zone1} 2022-09-09T22:48:10.831 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:10.831 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:10.832 DEBUG:boto:Method: HEAD 2022-09-09T22:48:10.833 DEBUG:boto:Path: /jkcgbj-20/ 2022-09-09T22:48:10.833 DEBUG:boto:Data: 2022-09-09T22:48:10.834 DEBUG:boto:Headers: {} 2022-09-09T22:48:10.834 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:10.835 DEBUG:boto:Port: 8000 2022-09-09T22:48:10.835 DEBUG:boto:Params: {} 2022-09-09T22:48:10.836 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T22:48:10.837 DEBUG:boto:Token: None 2022-09-09T22:48:10.837 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:10 GMT /jkcgbj-20/ 2022-09-09T22:48:10.838 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:GXMA3qS0y0yTWTaH8uFB0576I5E= 2022-09-09T22:48:10.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:GXMA3qS0y0yTWTaH8uFB0576I5E=', 'Content-Length': '0'} 2022-09-09T22:48:10.848 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0638e3cce92c4b1-00631bc2aa-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:10.849 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:10.850 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:10.850 DEBUG:boto:Method: HEAD 2022-09-09T22:48:10.851 DEBUG:boto:Path: /jkcgbj-20/ 2022-09-09T22:48:10.851 DEBUG:boto:Data: 2022-09-09T22:48:10.852 DEBUG:boto:Headers: {} 2022-09-09T22:48:10.852 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:10.853 DEBUG:boto:Port: 8000 2022-09-09T22:48:10.853 DEBUG:boto:Params: {} 2022-09-09T22:48:10.854 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T22:48:10.855 DEBUG:boto:Token: None 2022-09-09T22:48:10.855 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:10 GMT /jkcgbj-20/ 2022-09-09T22:48:10.856 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:GXMA3qS0y0yTWTaH8uFB0576I5E= 2022-09-09T22:48:10.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:GXMA3qS0y0yTWTaH8uFB0576I5E=', 'Content-Length': '0'} 2022-09-09T22:48:10.862 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a91e741df83ba555-00631bc2aa-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:10.863 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:48:10.864 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:10.864 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:10.865 DEBUG:boto:path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:10.865 DEBUG:boto:auth_path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:10.866 DEBUG:boto:Method: GET 2022-09-09T22:48:10.867 DEBUG:boto:Path: /jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:10.867 DEBUG:boto:Data: 2022-09-09T22:48:10.868 DEBUG:boto:Headers: {} 2022-09-09T22:48:10.868 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:10.869 DEBUG:boto:Port: 8000 2022-09-09T22:48:10.869 DEBUG:boto:Params: {} 2022-09-09T22:48:10.870 DEBUG:boto:Token: None 2022-09-09T22:48:10.870 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:10 GMT /jkcgbj-20/?versions 2022-09-09T22:48:10.871 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wdZav6PCpmhMWIeEIUgLtOrEC0k= 2022-09-09T22:48:10.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wdZav6PCpmhMWIeEIUgLtOrEC0k=', 'Content-Length': '0'} 2022-09-09T22:48:10.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018b3a10f8f57d7f7-00631bc2aa-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:10.877 DEBUG:boto:b'jkcgbj-20999falseobjnulltrue2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:10.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:48:10.880 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:48:10.880 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:10.881 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:10.881 DEBUG:boto:path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:10.882 DEBUG:boto:auth_path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:10.882 DEBUG:boto:Method: GET 2022-09-09T22:48:10.883 DEBUG:boto:Path: /jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:10.883 DEBUG:boto:Data: 2022-09-09T22:48:10.884 DEBUG:boto:Headers: {} 2022-09-09T22:48:10.884 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:10.885 DEBUG:boto:Port: 8000 2022-09-09T22:48:10.885 DEBUG:boto:Params: {} 2022-09-09T22:48:10.886 DEBUG:boto:Token: None 2022-09-09T22:48:10.886 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:10 GMT /jkcgbj-20/?versions 2022-09-09T22:48:10.887 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wdZav6PCpmhMWIeEIUgLtOrEC0k= 2022-09-09T22:48:10.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wdZav6PCpmhMWIeEIUgLtOrEC0k=', 'Content-Length': '0'} 2022-09-09T22:48:10.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d75cd792c34ac9e-00631bc2aa-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:10.894 DEBUG:boto:b'jkcgbj-20999falseobjnulltrue2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:10.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:48:10.896 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:10.896 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:10.897 DEBUG:boto:path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:10.897 DEBUG:boto:auth_path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:10.898 DEBUG:boto:Method: GET 2022-09-09T22:48:10.898 DEBUG:boto:Path: /jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:10.899 DEBUG:boto:Data: 2022-09-09T22:48:10.900 DEBUG:boto:Headers: {} 2022-09-09T22:48:10.900 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:10.901 DEBUG:boto:Port: 8000 2022-09-09T22:48:10.901 DEBUG:boto:Params: {} 2022-09-09T22:48:10.902 DEBUG:boto:Token: None 2022-09-09T22:48:10.903 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:10 GMT /jkcgbj-20/?versions 2022-09-09T22:48:10.903 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wdZav6PCpmhMWIeEIUgLtOrEC0k= 2022-09-09T22:48:10.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wdZav6PCpmhMWIeEIUgLtOrEC0k=', 'Content-Length': '0'} 2022-09-09T22:48:10.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ee34a321cab595f-00631bc2aa-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:10.910 DEBUG:boto:b'jkcgbj-20999falseobjnulltrue2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:10.911 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:10.911 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:10.912 DEBUG:boto:path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:10.913 DEBUG:boto:auth_path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:10.913 DEBUG:boto:Method: GET 2022-09-09T22:48:10.914 DEBUG:boto:Path: /jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:10.915 DEBUG:boto:Data: 2022-09-09T22:48:10.915 DEBUG:boto:Headers: {} 2022-09-09T22:48:10.916 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:10.917 DEBUG:boto:Port: 8000 2022-09-09T22:48:10.917 DEBUG:boto:Params: {} 2022-09-09T22:48:10.918 DEBUG:boto:Token: None 2022-09-09T22:48:10.919 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:10 GMT /jkcgbj-20/?versions 2022-09-09T22:48:10.920 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wdZav6PCpmhMWIeEIUgLtOrEC0k= 2022-09-09T22:48:10.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wdZav6PCpmhMWIeEIUgLtOrEC0k=', 'Content-Length': '0'} 2022-09-09T22:48:10.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019a02129d9427013-00631bc2aa-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:10.926 DEBUG:boto:b'jkcgbj-20999falseobjnulltrue2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:10.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:48:10.928 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:10.928 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:10.929 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:10.930 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:10.930 DEBUG:boto:Method: GET 2022-09-09T22:48:10.931 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:10.931 DEBUG:boto:Data: 2022-09-09T22:48:10.932 DEBUG:boto:Headers: {} 2022-09-09T22:48:10.933 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:10.933 DEBUG:boto:Port: 8000 2022-09-09T22:48:10.934 DEBUG:boto:Params: {} 2022-09-09T22:48:10.935 DEBUG:boto:Token: None 2022-09-09T22:48:10.935 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:10 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:10.936 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R0Fd+RQMLdlVui+6pZiMO9J4bMo= 2022-09-09T22:48:10.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R0Fd+RQMLdlVui+6pZiMO9J4bMo=', 'Content-Length': '0'} 2022-09-09T22:48:10.940 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000049f37636f5dfa359-00631bc2aa-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:10.941 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:10.942 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:10.942 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:10.943 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:10.943 DEBUG:boto:Method: GET 2022-09-09T22:48:10.944 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:10.944 DEBUG:boto:Data: 2022-09-09T22:48:10.945 DEBUG:boto:Headers: {} 2022-09-09T22:48:10.946 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:10.946 DEBUG:boto:Port: 8000 2022-09-09T22:48:10.947 DEBUG:boto:Params: {} 2022-09-09T22:48:10.948 DEBUG:boto:Token: None 2022-09-09T22:48:10.948 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:10 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:10.949 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R0Fd+RQMLdlVui+6pZiMO9J4bMo= 2022-09-09T22:48:10.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R0Fd+RQMLdlVui+6pZiMO9J4bMo=', 'Content-Length': '0'} 2022-09-09T22:48:10.953 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000de4a3422ba246f9e-00631bc2aa-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:10.954 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:10.954 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:10.955 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:10.956 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:10.956 DEBUG:boto:Method: HEAD 2022-09-09T22:48:10.957 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:10.957 DEBUG:boto:Data: 2022-09-09T22:48:10.958 DEBUG:boto:Headers: {} 2022-09-09T22:48:10.958 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:10.959 DEBUG:boto:Port: 8000 2022-09-09T22:48:10.959 DEBUG:boto:Params: {} 2022-09-09T22:48:10.960 DEBUG:boto:Token: None 2022-09-09T22:48:10.961 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:10 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:10.962 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:HJdUqYRjEhoTuMcxqx3RT+yvO5Y= 2022-09-09T22:48:10.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:HJdUqYRjEhoTuMcxqx3RT+yvO5Y=', 'Content-Length': '0'} 2022-09-09T22:48:10.965 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000061525a68c61347c6-00631bc2aa-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:10.966 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:10.967 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:10.968 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:10.968 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:10.969 DEBUG:boto:Method: HEAD 2022-09-09T22:48:10.970 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:10.970 DEBUG:boto:Data: 2022-09-09T22:48:10.971 DEBUG:boto:Headers: {} 2022-09-09T22:48:10.971 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:10.972 DEBUG:boto:Port: 8000 2022-09-09T22:48:10.973 DEBUG:boto:Params: {} 2022-09-09T22:48:10.973 DEBUG:boto:Token: None 2022-09-09T22:48:10.974 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:10 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:10.974 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:HJdUqYRjEhoTuMcxqx3RT+yvO5Y= 2022-09-09T22:48:10.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:HJdUqYRjEhoTuMcxqx3RT+yvO5Y=', 'Content-Length': '0'} 2022-09-09T22:48:10.978 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002849adb0d92e3323-00631bc2aa-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:10.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:48:10.980 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:10.980 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:10.981 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:10.981 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:10.982 DEBUG:boto:Method: GET 2022-09-09T22:48:10.983 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:10.983 DEBUG:boto:Data: 2022-09-09T22:48:10.984 DEBUG:boto:Headers: {} 2022-09-09T22:48:10.984 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:10.985 DEBUG:boto:Port: 8000 2022-09-09T22:48:10.985 DEBUG:boto:Params: {} 2022-09-09T22:48:10.986 DEBUG:boto:Token: None 2022-09-09T22:48:10.986 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:10 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:10.987 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R0Fd+RQMLdlVui+6pZiMO9J4bMo= 2022-09-09T22:48:10.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R0Fd+RQMLdlVui+6pZiMO9J4bMo=', 'Content-Length': '0'} 2022-09-09T22:48:10.991 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005d4d0fda6764caf5-00631bc2aa-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:10.991 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:10.992 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:10.993 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:10.993 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:10.994 DEBUG:boto:Method: GET 2022-09-09T22:48:10.994 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:10.995 DEBUG:boto:Data: 2022-09-09T22:48:10.995 DEBUG:boto:Headers: {} 2022-09-09T22:48:10.996 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:10.997 DEBUG:boto:Port: 8000 2022-09-09T22:48:10.997 DEBUG:boto:Params: {} 2022-09-09T22:48:10.998 DEBUG:boto:Token: None 2022-09-09T22:48:10.998 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:10 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:10.999 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R0Fd+RQMLdlVui+6pZiMO9J4bMo= 2022-09-09T22:48:11.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R0Fd+RQMLdlVui+6pZiMO9J4bMo=', 'Content-Length': '0'} 2022-09-09T22:48:11.003 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000077a372c8b8fdcc10-00631bc2aa-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.004 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:11.004 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:11.005 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.005 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.006 DEBUG:boto:Method: GET 2022-09-09T22:48:11.007 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.007 DEBUG:boto:Data: 2022-09-09T22:48:11.008 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.008 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:11.009 DEBUG:boto:Port: 8000 2022-09-09T22:48:11.009 DEBUG:boto:Params: {} 2022-09-09T22:48:11.010 DEBUG:boto:Token: None 2022-09-09T22:48:11.011 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/ 2022-09-09T22:48:11.011 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7wAJjRcIN30P68+RrLEZ6Ed9fvI= 2022-09-09T22:48:11.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7wAJjRcIN30P68+RrLEZ6Ed9fvI=', 'Content-Length': '0'} 2022-09-09T22:48:11.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002adc9c45c7bd6bd9-00631bc2ab-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.062 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:11.063 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:11.064 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:11.065 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.065 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.066 DEBUG:boto:Method: GET 2022-09-09T22:48:11.066 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.067 DEBUG:boto:Data: 2022-09-09T22:48:11.069 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.070 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:11.071 DEBUG:boto:Port: 8000 2022-09-09T22:48:11.072 DEBUG:boto:Params: {} 2022-09-09T22:48:11.073 DEBUG:boto:Token: None 2022-09-09T22:48:11.074 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/ 2022-09-09T22:48:11.075 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7wAJjRcIN30P68+RrLEZ6Ed9fvI= 2022-09-09T22:48:11.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7wAJjRcIN30P68+RrLEZ6Ed9fvI=', 'Content-Length': '0'} 2022-09-09T22:48:11.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000798a78a4fdc5139b-00631bc2ab-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.083 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:11.083 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:11.084 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:11.084 DEBUG:boto:Method: HEAD 2022-09-09T22:48:11.085 DEBUG:boto:Path: /jkcgbj-20/obj 2022-09-09T22:48:11.086 DEBUG:boto:Data: 2022-09-09T22:48:11.086 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.087 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:11.087 DEBUG:boto:Port: 8000 2022-09-09T22:48:11.088 DEBUG:boto:Params: {} 2022-09-09T22:48:11.089 DEBUG:boto:Token: None 2022-09-09T22:48:11.089 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/obj 2022-09-09T22:48:11.090 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:F/Ul3UsaCXDU2eZ3N9d5r138Syw= 2022-09-09T22:48:11.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:F/Ul3UsaCXDU2eZ3N9d5r138Syw=', 'Content-Length': '0'} 2022-09-09T22:48:11.094 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000482dce694ddeb5fb-00631bc2ab-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.095 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:11.096 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:11.097 DEBUG:boto:Method: HEAD 2022-09-09T22:48:11.097 DEBUG:boto:Path: /jkcgbj-20/obj 2022-09-09T22:48:11.098 DEBUG:boto:Data: 2022-09-09T22:48:11.098 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.099 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:11.099 DEBUG:boto:Port: 8000 2022-09-09T22:48:11.100 DEBUG:boto:Params: {} 2022-09-09T22:48:11.100 DEBUG:boto:Token: None 2022-09-09T22:48:11.101 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/obj 2022-09-09T22:48:11.102 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:F/Ul3UsaCXDU2eZ3N9d5r138Syw= 2022-09-09T22:48:11.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:F/Ul3UsaCXDU2eZ3N9d5r138Syw=', 'Content-Length': '0'} 2022-09-09T22:48:11.106 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006b647768f426f5c0-00631bc2ab-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:48:11.107 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:11.108 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:11.108 DEBUG:boto:Method: GET 2022-09-09T22:48:11.109 DEBUG:boto:Path: /jkcgbj-20/obj 2022-09-09T22:48:11.110 DEBUG:boto:Data: 2022-09-09T22:48:11.110 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.111 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:11.111 DEBUG:boto:Port: 8000 2022-09-09T22:48:11.112 DEBUG:boto:Params: {} 2022-09-09T22:48:11.113 DEBUG:boto:Token: None 2022-09-09T22:48:11.113 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/obj 2022-09-09T22:48:11.114 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:QEDcT8revkSEX8zhFz7VVQ0XYGk= 2022-09-09T22:48:11.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:QEDcT8revkSEX8zhFz7VVQ0XYGk=', 'Content-Length': '0'} 2022-09-09T22:48:11.118 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000845cdf1b4905a21d-00631bc2ab-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.119 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:11.119 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:11.120 DEBUG:boto:Method: GET 2022-09-09T22:48:11.121 DEBUG:boto:Path: /jkcgbj-20/obj 2022-09-09T22:48:11.121 DEBUG:boto:Data: 2022-09-09T22:48:11.122 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.122 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:11.123 DEBUG:boto:Port: 8000 2022-09-09T22:48:11.123 DEBUG:boto:Params: {} 2022-09-09T22:48:11.124 DEBUG:boto:Token: None 2022-09-09T22:48:11.125 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/obj 2022-09-09T22:48:11.125 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:QEDcT8revkSEX8zhFz7VVQ0XYGk= 2022-09-09T22:48:11.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:QEDcT8revkSEX8zhFz7VVQ0XYGk=', 'Content-Length': '0'} 2022-09-09T22:48:11.129 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c6ecef0dbed45eef-00631bc2ab-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.130 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:11.131 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:11.131 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.132 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.132 DEBUG:boto:Method: GET 2022-09-09T22:48:11.133 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.134 DEBUG:boto:Data: 2022-09-09T22:48:11.134 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.135 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:11.136 DEBUG:boto:Port: 8000 2022-09-09T22:48:11.136 DEBUG:boto:Params: {} 2022-09-09T22:48:11.137 DEBUG:boto:Token: None 2022-09-09T22:48:11.138 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/ 2022-09-09T22:48:11.139 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7wAJjRcIN30P68+RrLEZ6Ed9fvI= 2022-09-09T22:48:11.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7wAJjRcIN30P68+RrLEZ6Ed9fvI=', 'Content-Length': '0'} 2022-09-09T22:48:11.144 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c84eb755ae951d2a-00631bc2ab-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.145 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:11.146 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:11.147 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:11.147 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.148 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.148 DEBUG:boto:Method: GET 2022-09-09T22:48:11.149 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.150 DEBUG:boto:Data: 2022-09-09T22:48:11.151 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.151 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:11.152 DEBUG:boto:Port: 8000 2022-09-09T22:48:11.153 DEBUG:boto:Params: {} 2022-09-09T22:48:11.153 DEBUG:boto:Token: None 2022-09-09T22:48:11.154 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/ 2022-09-09T22:48:11.156 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7wAJjRcIN30P68+RrLEZ6Ed9fvI= 2022-09-09T22:48:11.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7wAJjRcIN30P68+RrLEZ6Ed9fvI=', 'Content-Length': '0'} 2022-09-09T22:48:11.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032b57ba106b9d6d9-00631bc2ab-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.163 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:11.164 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-20 zones={test-zone2, test-zone1} 2022-09-09T22:48:11.165 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-20 zones={test-zone3, test-zone1} 2022-09-09T22:48:11.165 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:11.166 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:11.166 DEBUG:boto:Method: HEAD 2022-09-09T22:48:11.167 DEBUG:boto:Path: /jkcgbj-20/ 2022-09-09T22:48:11.167 DEBUG:boto:Data: 2022-09-09T22:48:11.168 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.168 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:11.169 DEBUG:boto:Port: 8001 2022-09-09T22:48:11.169 DEBUG:boto:Params: {} 2022-09-09T22:48:11.170 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-09T22:48:11.171 DEBUG:boto:Token: None 2022-09-09T22:48:11.171 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/ 2022-09-09T22:48:11.172 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6QLV8PHGoZfLHVnAar9zmWGS80Q= 2022-09-09T22:48:11.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6QLV8PHGoZfLHVnAar9zmWGS80Q=', 'Content-Length': '0'} 2022-09-09T22:48:11.182 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a404e38fea0313d-00631bc2ab-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.183 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:11.183 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:11.184 DEBUG:boto:Method: HEAD 2022-09-09T22:48:11.184 DEBUG:boto:Path: /jkcgbj-20/ 2022-09-09T22:48:11.185 DEBUG:boto:Data: 2022-09-09T22:48:11.185 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.186 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:11.187 DEBUG:boto:Port: 8000 2022-09-09T22:48:11.187 DEBUG:boto:Params: {} 2022-09-09T22:48:11.188 DEBUG:boto:Token: None 2022-09-09T22:48:11.188 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/ 2022-09-09T22:48:11.189 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6QLV8PHGoZfLHVnAar9zmWGS80Q= 2022-09-09T22:48:11.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6QLV8PHGoZfLHVnAar9zmWGS80Q=', 'Content-Length': '0'} 2022-09-09T22:48:11.196 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2c5b743d9f290d0-00631bc2ab-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.196 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:48:11.197 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:11.198 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:11.198 DEBUG:boto:path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:11.199 DEBUG:boto:auth_path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:11.199 DEBUG:boto:Method: GET 2022-09-09T22:48:11.200 DEBUG:boto:Path: /jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:11.200 DEBUG:boto:Data: 2022-09-09T22:48:11.201 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.201 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:11.202 DEBUG:boto:Port: 8001 2022-09-09T22:48:11.202 DEBUG:boto:Params: {} 2022-09-09T22:48:11.203 DEBUG:boto:Token: None 2022-09-09T22:48:11.204 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/?versions 2022-09-09T22:48:11.204 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wjf+nEtX+jDvC2QEirP9O16Jrxc= 2022-09-09T22:48:11.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wjf+nEtX+jDvC2QEirP9O16Jrxc=', 'Content-Length': '0'} 2022-09-09T22:48:11.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003a6f3547cc6d952-00631bc2ab-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.211 DEBUG:boto:b'jkcgbj-20999falseobjnulltrue2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:11.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:48:11.212 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:48:11.213 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:11.214 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:11.214 DEBUG:boto:path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:11.215 DEBUG:boto:auth_path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:11.215 DEBUG:boto:Method: GET 2022-09-09T22:48:11.216 DEBUG:boto:Path: /jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:11.216 DEBUG:boto:Data: 2022-09-09T22:48:11.217 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.217 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:11.218 DEBUG:boto:Port: 8000 2022-09-09T22:48:11.218 DEBUG:boto:Params: {} 2022-09-09T22:48:11.219 DEBUG:boto:Token: None 2022-09-09T22:48:11.220 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/?versions 2022-09-09T22:48:11.220 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wjf+nEtX+jDvC2QEirP9O16Jrxc= 2022-09-09T22:48:11.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wjf+nEtX+jDvC2QEirP9O16Jrxc=', 'Content-Length': '0'} 2022-09-09T22:48:11.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4831e8f868dae73-00631bc2ab-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.274 DEBUG:boto:b'jkcgbj-20999falseobjnulltrue2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:11.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:48:11.276 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:11.276 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:11.277 DEBUG:boto:path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:11.277 DEBUG:boto:auth_path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:11.278 DEBUG:boto:Method: GET 2022-09-09T22:48:11.278 DEBUG:boto:Path: /jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:11.279 DEBUG:boto:Data: 2022-09-09T22:48:11.279 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.280 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:11.280 DEBUG:boto:Port: 8001 2022-09-09T22:48:11.281 DEBUG:boto:Params: {} 2022-09-09T22:48:11.281 DEBUG:boto:Token: None 2022-09-09T22:48:11.282 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/?versions 2022-09-09T22:48:11.283 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wjf+nEtX+jDvC2QEirP9O16Jrxc= 2022-09-09T22:48:11.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wjf+nEtX+jDvC2QEirP9O16Jrxc=', 'Content-Length': '0'} 2022-09-09T22:48:11.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0d1c76ef5b435a5-00631bc2ab-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.289 DEBUG:boto:b'jkcgbj-20999falseobjnulltrue2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:11.290 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:11.291 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:11.291 DEBUG:boto:path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:11.292 DEBUG:boto:auth_path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:11.292 DEBUG:boto:Method: GET 2022-09-09T22:48:11.293 DEBUG:boto:Path: /jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:11.293 DEBUG:boto:Data: 2022-09-09T22:48:11.294 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.294 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:11.295 DEBUG:boto:Port: 8000 2022-09-09T22:48:11.295 DEBUG:boto:Params: {} 2022-09-09T22:48:11.296 DEBUG:boto:Token: None 2022-09-09T22:48:11.296 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/?versions 2022-09-09T22:48:11.297 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wjf+nEtX+jDvC2QEirP9O16Jrxc= 2022-09-09T22:48:11.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wjf+nEtX+jDvC2QEirP9O16Jrxc=', 'Content-Length': '0'} 2022-09-09T22:48:11.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a372aa036c2762b5-00631bc2ab-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.304 DEBUG:boto:b'jkcgbj-20999falseobjnulltrue2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:11.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:48:11.305 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:11.305 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:11.306 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.306 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.307 DEBUG:boto:Method: GET 2022-09-09T22:48:11.308 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.308 DEBUG:boto:Data: 2022-09-09T22:48:11.309 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.309 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:11.310 DEBUG:boto:Port: 8001 2022-09-09T22:48:11.310 DEBUG:boto:Params: {} 2022-09-09T22:48:11.311 DEBUG:boto:Token: None 2022-09-09T22:48:11.311 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.312 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:hWQKUXFPylGKQvn/TXO0tCaLnS4= 2022-09-09T22:48:11.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:hWQKUXFPylGKQvn/TXO0tCaLnS4=', 'Content-Length': '0'} 2022-09-09T22:48:11.317 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ba9206ce27b46fc1-00631bc2ab-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.317 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:11.318 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:11.318 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.319 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.319 DEBUG:boto:Method: GET 2022-09-09T22:48:11.320 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.321 DEBUG:boto:Data: 2022-09-09T22:48:11.321 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.322 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:11.322 DEBUG:boto:Port: 8000 2022-09-09T22:48:11.323 DEBUG:boto:Params: {} 2022-09-09T22:48:11.323 DEBUG:boto:Token: None 2022-09-09T22:48:11.324 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.324 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:hWQKUXFPylGKQvn/TXO0tCaLnS4= 2022-09-09T22:48:11.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:hWQKUXFPylGKQvn/TXO0tCaLnS4=', 'Content-Length': '0'} 2022-09-09T22:48:11.328 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000086ab4d2315ff8acb-00631bc2ab-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.329 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:11.330 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:11.330 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.331 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.331 DEBUG:boto:Method: HEAD 2022-09-09T22:48:11.332 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.332 DEBUG:boto:Data: 2022-09-09T22:48:11.333 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.333 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:11.334 DEBUG:boto:Port: 8001 2022-09-09T22:48:11.334 DEBUG:boto:Params: {} 2022-09-09T22:48:11.335 DEBUG:boto:Token: None 2022-09-09T22:48:11.335 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.336 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Nxo/+dbzE8lfKQ10WVKOcEFQYbI= 2022-09-09T22:48:11.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Nxo/+dbzE8lfKQ10WVKOcEFQYbI=', 'Content-Length': '0'} 2022-09-09T22:48:11.339 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000ff408677cd7b66e-00631bc2ab-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.340 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:11.341 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:11.341 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.341 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.342 DEBUG:boto:Method: HEAD 2022-09-09T22:48:11.343 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.343 DEBUG:boto:Data: 2022-09-09T22:48:11.343 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.344 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:11.344 DEBUG:boto:Port: 8000 2022-09-09T22:48:11.345 DEBUG:boto:Params: {} 2022-09-09T22:48:11.345 DEBUG:boto:Token: None 2022-09-09T22:48:11.346 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.347 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Nxo/+dbzE8lfKQ10WVKOcEFQYbI= 2022-09-09T22:48:11.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Nxo/+dbzE8lfKQ10WVKOcEFQYbI=', 'Content-Length': '0'} 2022-09-09T22:48:11.350 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000036ba2a7215895662-00631bc2ab-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:48:11.351 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:11.352 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:11.352 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.353 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.353 DEBUG:boto:Method: GET 2022-09-09T22:48:11.354 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.354 DEBUG:boto:Data: 2022-09-09T22:48:11.355 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.355 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:11.356 DEBUG:boto:Port: 8001 2022-09-09T22:48:11.356 DEBUG:boto:Params: {} 2022-09-09T22:48:11.357 DEBUG:boto:Token: None 2022-09-09T22:48:11.358 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.358 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:hWQKUXFPylGKQvn/TXO0tCaLnS4= 2022-09-09T22:48:11.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:hWQKUXFPylGKQvn/TXO0tCaLnS4=', 'Content-Length': '0'} 2022-09-09T22:48:11.362 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b3c7d0ca379cbfa2-00631bc2ab-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.363 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:11.363 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:11.364 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.364 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.365 DEBUG:boto:Method: GET 2022-09-09T22:48:11.366 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.366 DEBUG:boto:Data: 2022-09-09T22:48:11.367 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.367 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:11.368 DEBUG:boto:Port: 8000 2022-09-09T22:48:11.368 DEBUG:boto:Params: {} 2022-09-09T22:48:11.369 DEBUG:boto:Token: None 2022-09-09T22:48:11.369 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.370 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:hWQKUXFPylGKQvn/TXO0tCaLnS4= 2022-09-09T22:48:11.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:hWQKUXFPylGKQvn/TXO0tCaLnS4=', 'Content-Length': '0'} 2022-09-09T22:48:11.373 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005213dbe2e3441367-00631bc2ab-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.374 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:11.375 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:11.375 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.376 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.376 DEBUG:boto:Method: GET 2022-09-09T22:48:11.377 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.377 DEBUG:boto:Data: 2022-09-09T22:48:11.378 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.378 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:11.379 DEBUG:boto:Port: 8001 2022-09-09T22:48:11.379 DEBUG:boto:Params: {} 2022-09-09T22:48:11.380 DEBUG:boto:Token: None 2022-09-09T22:48:11.381 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/ 2022-09-09T22:48:11.381 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7wAJjRcIN30P68+RrLEZ6Ed9fvI= 2022-09-09T22:48:11.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7wAJjRcIN30P68+RrLEZ6Ed9fvI=', 'Content-Length': '0'} 2022-09-09T22:48:11.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083bd528af5f94fda-00631bc2ab-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.430 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:11.431 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:11.432 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:11.432 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.433 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.433 DEBUG:boto:Method: GET 2022-09-09T22:48:11.434 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.435 DEBUG:boto:Data: 2022-09-09T22:48:11.435 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.436 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:11.436 DEBUG:boto:Port: 8000 2022-09-09T22:48:11.437 DEBUG:boto:Params: {} 2022-09-09T22:48:11.437 DEBUG:boto:Token: None 2022-09-09T22:48:11.438 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/ 2022-09-09T22:48:11.439 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7wAJjRcIN30P68+RrLEZ6Ed9fvI= 2022-09-09T22:48:11.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7wAJjRcIN30P68+RrLEZ6Ed9fvI=', 'Content-Length': '0'} 2022-09-09T22:48:11.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008232778975c3f9b0-00631bc2ab-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.447 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:11.448 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:11.449 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:11.449 DEBUG:boto:Method: HEAD 2022-09-09T22:48:11.450 DEBUG:boto:Path: /jkcgbj-20/obj 2022-09-09T22:48:11.450 DEBUG:boto:Data: 2022-09-09T22:48:11.451 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.452 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:11.452 DEBUG:boto:Port: 8001 2022-09-09T22:48:11.453 DEBUG:boto:Params: {} 2022-09-09T22:48:11.454 DEBUG:boto:Token: None 2022-09-09T22:48:11.454 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/obj 2022-09-09T22:48:11.455 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:F/Ul3UsaCXDU2eZ3N9d5r138Syw= 2022-09-09T22:48:11.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:F/Ul3UsaCXDU2eZ3N9d5r138Syw=', 'Content-Length': '0'} 2022-09-09T22:48:11.459 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008ae02b815b1231ed-00631bc2ab-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.460 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:11.461 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:11.462 DEBUG:boto:Method: HEAD 2022-09-09T22:48:11.463 DEBUG:boto:Path: /jkcgbj-20/obj 2022-09-09T22:48:11.464 DEBUG:boto:Data: 2022-09-09T22:48:11.465 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.466 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:11.467 DEBUG:boto:Port: 8000 2022-09-09T22:48:11.468 DEBUG:boto:Params: {} 2022-09-09T22:48:11.468 DEBUG:boto:Token: None 2022-09-09T22:48:11.469 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/obj 2022-09-09T22:48:11.470 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:F/Ul3UsaCXDU2eZ3N9d5r138Syw= 2022-09-09T22:48:11.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:F/Ul3UsaCXDU2eZ3N9d5r138Syw=', 'Content-Length': '0'} 2022-09-09T22:48:11.474 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f2c860ba376f3094-00631bc2ab-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:48:11.475 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:11.476 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:11.476 DEBUG:boto:Method: GET 2022-09-09T22:48:11.477 DEBUG:boto:Path: /jkcgbj-20/obj 2022-09-09T22:48:11.478 DEBUG:boto:Data: 2022-09-09T22:48:11.478 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.479 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:11.479 DEBUG:boto:Port: 8001 2022-09-09T22:48:11.480 DEBUG:boto:Params: {} 2022-09-09T22:48:11.480 DEBUG:boto:Token: None 2022-09-09T22:48:11.481 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/obj 2022-09-09T22:48:11.482 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:QEDcT8revkSEX8zhFz7VVQ0XYGk= 2022-09-09T22:48:11.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:QEDcT8revkSEX8zhFz7VVQ0XYGk=', 'Content-Length': '0'} 2022-09-09T22:48:11.485 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000023bd9fe7dda4575f-00631bc2ab-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.486 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:11.487 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:11.487 DEBUG:boto:Method: GET 2022-09-09T22:48:11.488 DEBUG:boto:Path: /jkcgbj-20/obj 2022-09-09T22:48:11.489 DEBUG:boto:Data: 2022-09-09T22:48:11.490 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.490 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:11.491 DEBUG:boto:Port: 8000 2022-09-09T22:48:11.491 DEBUG:boto:Params: {} 2022-09-09T22:48:11.492 DEBUG:boto:Token: None 2022-09-09T22:48:11.492 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/obj 2022-09-09T22:48:11.493 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:QEDcT8revkSEX8zhFz7VVQ0XYGk= 2022-09-09T22:48:11.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:QEDcT8revkSEX8zhFz7VVQ0XYGk=', 'Content-Length': '0'} 2022-09-09T22:48:11.497 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004d6a00959ef201b7-00631bc2ab-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.498 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:11.498 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:11.499 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.500 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.500 DEBUG:boto:Method: GET 2022-09-09T22:48:11.501 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.501 DEBUG:boto:Data: 2022-09-09T22:48:11.502 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.503 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:11.503 DEBUG:boto:Port: 8001 2022-09-09T22:48:11.504 DEBUG:boto:Params: {} 2022-09-09T22:48:11.505 DEBUG:boto:Token: None 2022-09-09T22:48:11.505 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/ 2022-09-09T22:48:11.506 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7wAJjRcIN30P68+RrLEZ6Ed9fvI= 2022-09-09T22:48:11.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7wAJjRcIN30P68+RrLEZ6Ed9fvI=', 'Content-Length': '0'} 2022-09-09T22:48:11.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d4f154450c08fca-00631bc2ab-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.513 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:11.514 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:11.515 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:11.515 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.516 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.516 DEBUG:boto:Method: GET 2022-09-09T22:48:11.517 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.517 DEBUG:boto:Data: 2022-09-09T22:48:11.518 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.519 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:11.519 DEBUG:boto:Port: 8000 2022-09-09T22:48:11.520 DEBUG:boto:Params: {} 2022-09-09T22:48:11.520 DEBUG:boto:Token: None 2022-09-09T22:48:11.521 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/ 2022-09-09T22:48:11.521 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7wAJjRcIN30P68+RrLEZ6Ed9fvI= 2022-09-09T22:48:11.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7wAJjRcIN30P68+RrLEZ6Ed9fvI=', 'Content-Length': '0'} 2022-09-09T22:48:11.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b939a40e09b2de8-00631bc2ab-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.529 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:11.530 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-20 zones={test-zone3, test-zone1} 2022-09-09T22:48:11.531 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-20 zones={test-zone3, test-zone2} 2022-09-09T22:48:11.531 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:11.532 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:11.532 DEBUG:boto:Method: HEAD 2022-09-09T22:48:11.533 DEBUG:boto:Path: /jkcgbj-20/ 2022-09-09T22:48:11.533 DEBUG:boto:Data: 2022-09-09T22:48:11.534 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.534 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:11.535 DEBUG:boto:Port: 8001 2022-09-09T22:48:11.535 DEBUG:boto:Params: {} 2022-09-09T22:48:11.536 DEBUG:boto:Token: None 2022-09-09T22:48:11.537 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/ 2022-09-09T22:48:11.537 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6QLV8PHGoZfLHVnAar9zmWGS80Q= 2022-09-09T22:48:11.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6QLV8PHGoZfLHVnAar9zmWGS80Q=', 'Content-Length': '0'} 2022-09-09T22:48:11.543 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005864782f768c6d7e-00631bc2ab-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.543 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:11.544 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:11.544 DEBUG:boto:Method: HEAD 2022-09-09T22:48:11.545 DEBUG:boto:Path: /jkcgbj-20/ 2022-09-09T22:48:11.545 DEBUG:boto:Data: 2022-09-09T22:48:11.546 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.546 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:11.547 DEBUG:boto:Port: 8000 2022-09-09T22:48:11.547 DEBUG:boto:Params: {} 2022-09-09T22:48:11.548 DEBUG:boto:Token: None 2022-09-09T22:48:11.549 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/ 2022-09-09T22:48:11.549 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6QLV8PHGoZfLHVnAar9zmWGS80Q= 2022-09-09T22:48:11.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6QLV8PHGoZfLHVnAar9zmWGS80Q=', 'Content-Length': '0'} 2022-09-09T22:48:11.554 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0ff5fea37e929f8-00631bc2ab-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.555 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:48:11.556 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:11.556 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:11.557 DEBUG:boto:path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:11.557 DEBUG:boto:auth_path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:11.558 DEBUG:boto:Method: GET 2022-09-09T22:48:11.558 DEBUG:boto:Path: /jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:11.559 DEBUG:boto:Data: 2022-09-09T22:48:11.560 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.560 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:11.561 DEBUG:boto:Port: 8001 2022-09-09T22:48:11.561 DEBUG:boto:Params: {} 2022-09-09T22:48:11.562 DEBUG:boto:Token: None 2022-09-09T22:48:11.562 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/?versions 2022-09-09T22:48:11.563 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wjf+nEtX+jDvC2QEirP9O16Jrxc= 2022-09-09T22:48:11.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wjf+nEtX+jDvC2QEirP9O16Jrxc=', 'Content-Length': '0'} 2022-09-09T22:48:11.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8c4869f59b607da-00631bc2ab-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.610 DEBUG:boto:b'jkcgbj-20999falseobjnulltrue2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:11.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:48:11.612 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:48:11.612 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:11.613 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:11.613 DEBUG:boto:path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:11.614 DEBUG:boto:auth_path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:11.614 DEBUG:boto:Method: GET 2022-09-09T22:48:11.615 DEBUG:boto:Path: /jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:11.615 DEBUG:boto:Data: 2022-09-09T22:48:11.616 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.616 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:11.617 DEBUG:boto:Port: 8000 2022-09-09T22:48:11.617 DEBUG:boto:Params: {} 2022-09-09T22:48:11.618 DEBUG:boto:Token: None 2022-09-09T22:48:11.618 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/?versions 2022-09-09T22:48:11.619 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wjf+nEtX+jDvC2QEirP9O16Jrxc= 2022-09-09T22:48:11.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wjf+nEtX+jDvC2QEirP9O16Jrxc=', 'Content-Length': '0'} 2022-09-09T22:48:11.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ff064527ed3fc3a-00631bc2ab-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.626 DEBUG:boto:b'jkcgbj-20999falseobjnulltrue2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:11.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:48:11.627 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:11.628 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:11.628 DEBUG:boto:path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:11.629 DEBUG:boto:auth_path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:11.629 DEBUG:boto:Method: GET 2022-09-09T22:48:11.630 DEBUG:boto:Path: /jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:11.630 DEBUG:boto:Data: 2022-09-09T22:48:11.631 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.631 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:11.632 DEBUG:boto:Port: 8001 2022-09-09T22:48:11.632 DEBUG:boto:Params: {} 2022-09-09T22:48:11.633 DEBUG:boto:Token: None 2022-09-09T22:48:11.633 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/?versions 2022-09-09T22:48:11.634 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wjf+nEtX+jDvC2QEirP9O16Jrxc= 2022-09-09T22:48:11.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wjf+nEtX+jDvC2QEirP9O16Jrxc=', 'Content-Length': '0'} 2022-09-09T22:48:11.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000814bb9421fb2a33e-00631bc2ab-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.641 DEBUG:boto:b'jkcgbj-20999falseobjnulltrue2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:11.642 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:11.643 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:11.643 DEBUG:boto:path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:11.644 DEBUG:boto:auth_path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:11.644 DEBUG:boto:Method: GET 2022-09-09T22:48:11.645 DEBUG:boto:Path: /jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:11.645 DEBUG:boto:Data: 2022-09-09T22:48:11.646 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.646 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:11.647 DEBUG:boto:Port: 8000 2022-09-09T22:48:11.647 DEBUG:boto:Params: {} 2022-09-09T22:48:11.648 DEBUG:boto:Token: None 2022-09-09T22:48:11.649 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/?versions 2022-09-09T22:48:11.649 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wjf+nEtX+jDvC2QEirP9O16Jrxc= 2022-09-09T22:48:11.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wjf+nEtX+jDvC2QEirP9O16Jrxc=', 'Content-Length': '0'} 2022-09-09T22:48:11.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068f51bda441c4eea-00631bc2ab-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.698 DEBUG:boto:b'jkcgbj-20999falseobjnulltrue2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:11.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:48:11.700 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:11.700 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:11.701 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.701 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.702 DEBUG:boto:Method: GET 2022-09-09T22:48:11.702 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.703 DEBUG:boto:Data: 2022-09-09T22:48:11.703 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.704 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:11.704 DEBUG:boto:Port: 8001 2022-09-09T22:48:11.705 DEBUG:boto:Params: {} 2022-09-09T22:48:11.706 DEBUG:boto:Token: None 2022-09-09T22:48:11.706 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.707 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:hWQKUXFPylGKQvn/TXO0tCaLnS4= 2022-09-09T22:48:11.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:hWQKUXFPylGKQvn/TXO0tCaLnS4=', 'Content-Length': '0'} 2022-09-09T22:48:11.710 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d6bbb08e9d9d6170-00631bc2ab-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.711 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:11.711 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:11.712 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.713 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.713 DEBUG:boto:Method: GET 2022-09-09T22:48:11.714 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.714 DEBUG:boto:Data: 2022-09-09T22:48:11.719 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.720 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:11.720 DEBUG:boto:Port: 8000 2022-09-09T22:48:11.721 DEBUG:boto:Params: {} 2022-09-09T22:48:11.721 DEBUG:boto:Token: None 2022-09-09T22:48:11.722 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.723 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:hWQKUXFPylGKQvn/TXO0tCaLnS4= 2022-09-09T22:48:11.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:hWQKUXFPylGKQvn/TXO0tCaLnS4=', 'Content-Length': '0'} 2022-09-09T22:48:11.727 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003d2925f43c51d835-00631bc2ab-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.728 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:11.729 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:11.730 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.731 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.731 DEBUG:boto:Method: HEAD 2022-09-09T22:48:11.732 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.733 DEBUG:boto:Data: 2022-09-09T22:48:11.734 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.735 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:11.736 DEBUG:boto:Port: 8001 2022-09-09T22:48:11.737 DEBUG:boto:Params: {} 2022-09-09T22:48:11.738 DEBUG:boto:Token: None 2022-09-09T22:48:11.739 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.740 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Nxo/+dbzE8lfKQ10WVKOcEFQYbI= 2022-09-09T22:48:11.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Nxo/+dbzE8lfKQ10WVKOcEFQYbI=', 'Content-Length': '0'} 2022-09-09T22:48:11.744 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000002edd51c0ccd32bf-00631bc2ab-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.744 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:11.745 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:11.746 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.746 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.747 DEBUG:boto:Method: HEAD 2022-09-09T22:48:11.747 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.748 DEBUG:boto:Data: 2022-09-09T22:48:11.749 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.749 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:11.750 DEBUG:boto:Port: 8000 2022-09-09T22:48:11.751 DEBUG:boto:Params: {} 2022-09-09T22:48:11.752 DEBUG:boto:Token: None 2022-09-09T22:48:11.753 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.753 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Nxo/+dbzE8lfKQ10WVKOcEFQYbI= 2022-09-09T22:48:11.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Nxo/+dbzE8lfKQ10WVKOcEFQYbI=', 'Content-Length': '0'} 2022-09-09T22:48:11.756 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000022cca9fbca257c30-00631bc2ab-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:48:11.758 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:11.758 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:11.759 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.759 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.760 DEBUG:boto:Method: GET 2022-09-09T22:48:11.760 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.761 DEBUG:boto:Data: 2022-09-09T22:48:11.762 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.763 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:11.764 DEBUG:boto:Port: 8001 2022-09-09T22:48:11.765 DEBUG:boto:Params: {} 2022-09-09T22:48:11.766 DEBUG:boto:Token: None 2022-09-09T22:48:11.767 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.768 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:hWQKUXFPylGKQvn/TXO0tCaLnS4= 2022-09-09T22:48:11.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:hWQKUXFPylGKQvn/TXO0tCaLnS4=', 'Content-Length': '0'} 2022-09-09T22:48:11.773 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fe203827f2108a5d-00631bc2ab-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.774 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:11.775 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:11.775 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.776 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.776 DEBUG:boto:Method: GET 2022-09-09T22:48:11.777 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.778 DEBUG:boto:Data: 2022-09-09T22:48:11.779 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.780 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:11.781 DEBUG:boto:Port: 8000 2022-09-09T22:48:11.782 DEBUG:boto:Params: {} 2022-09-09T22:48:11.783 DEBUG:boto:Token: None 2022-09-09T22:48:11.784 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:11.785 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:hWQKUXFPylGKQvn/TXO0tCaLnS4= 2022-09-09T22:48:11.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:hWQKUXFPylGKQvn/TXO0tCaLnS4=', 'Content-Length': '0'} 2022-09-09T22:48:11.789 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fc3a3f56b743a901-00631bc2ab-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.791 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:11.792 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:11.792 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.793 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.794 DEBUG:boto:Method: GET 2022-09-09T22:48:11.795 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.796 DEBUG:boto:Data: 2022-09-09T22:48:11.798 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.799 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:11.800 DEBUG:boto:Port: 8001 2022-09-09T22:48:11.801 DEBUG:boto:Params: {} 2022-09-09T22:48:11.801 DEBUG:boto:Token: None 2022-09-09T22:48:11.802 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/ 2022-09-09T22:48:11.803 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7wAJjRcIN30P68+RrLEZ6Ed9fvI= 2022-09-09T22:48:11.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7wAJjRcIN30P68+RrLEZ6Ed9fvI=', 'Content-Length': '0'} 2022-09-09T22:48:11.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c4d759231900da8-00631bc2ab-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.854 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:11.855 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:11.856 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:11.856 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.857 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.858 DEBUG:boto:Method: GET 2022-09-09T22:48:11.859 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.860 DEBUG:boto:Data: 2022-09-09T22:48:11.861 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.863 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:11.864 DEBUG:boto:Port: 8000 2022-09-09T22:48:11.864 DEBUG:boto:Params: {} 2022-09-09T22:48:11.865 DEBUG:boto:Token: None 2022-09-09T22:48:11.866 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/ 2022-09-09T22:48:11.872 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7wAJjRcIN30P68+RrLEZ6Ed9fvI= 2022-09-09T22:48:11.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7wAJjRcIN30P68+RrLEZ6Ed9fvI=', 'Content-Length': '0'} 2022-09-09T22:48:11.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac3c9dcb4e473b89-00631bc2ab-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.879 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:11.880 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:11.881 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:11.882 DEBUG:boto:Method: HEAD 2022-09-09T22:48:11.882 DEBUG:boto:Path: /jkcgbj-20/obj 2022-09-09T22:48:11.883 DEBUG:boto:Data: 2022-09-09T22:48:11.884 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.885 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:11.885 DEBUG:boto:Port: 8001 2022-09-09T22:48:11.887 DEBUG:boto:Params: {} 2022-09-09T22:48:11.888 DEBUG:boto:Token: None 2022-09-09T22:48:11.889 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/obj 2022-09-09T22:48:11.890 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:F/Ul3UsaCXDU2eZ3N9d5r138Syw= 2022-09-09T22:48:11.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:F/Ul3UsaCXDU2eZ3N9d5r138Syw=', 'Content-Length': '0'} 2022-09-09T22:48:11.894 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f9e7291006d8e118-00631bc2ab-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.895 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:11.896 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:11.897 DEBUG:boto:Method: HEAD 2022-09-09T22:48:11.898 DEBUG:boto:Path: /jkcgbj-20/obj 2022-09-09T22:48:11.899 DEBUG:boto:Data: 2022-09-09T22:48:11.900 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.900 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:11.901 DEBUG:boto:Port: 8000 2022-09-09T22:48:11.902 DEBUG:boto:Params: {} 2022-09-09T22:48:11.903 DEBUG:boto:Token: None 2022-09-09T22:48:11.904 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/obj 2022-09-09T22:48:11.906 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:F/Ul3UsaCXDU2eZ3N9d5r138Syw= 2022-09-09T22:48:11.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:F/Ul3UsaCXDU2eZ3N9d5r138Syw=', 'Content-Length': '0'} 2022-09-09T22:48:11.910 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000067d6e69dfc23366d-00631bc2ab-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:48:11.912 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:11.913 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:11.914 DEBUG:boto:Method: GET 2022-09-09T22:48:11.915 DEBUG:boto:Path: /jkcgbj-20/obj 2022-09-09T22:48:11.916 DEBUG:boto:Data: 2022-09-09T22:48:11.917 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.918 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:11.919 DEBUG:boto:Port: 8001 2022-09-09T22:48:11.920 DEBUG:boto:Params: {} 2022-09-09T22:48:11.921 DEBUG:boto:Token: None 2022-09-09T22:48:11.922 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/obj 2022-09-09T22:48:11.923 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:QEDcT8revkSEX8zhFz7VVQ0XYGk= 2022-09-09T22:48:11.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:QEDcT8revkSEX8zhFz7VVQ0XYGk=', 'Content-Length': '0'} 2022-09-09T22:48:11.928 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004147e244a791bedb-00631bc2ab-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.930 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:11.931 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:11.932 DEBUG:boto:Method: GET 2022-09-09T22:48:11.933 DEBUG:boto:Path: /jkcgbj-20/obj 2022-09-09T22:48:11.934 DEBUG:boto:Data: 2022-09-09T22:48:11.935 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.936 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:11.937 DEBUG:boto:Port: 8000 2022-09-09T22:48:11.938 DEBUG:boto:Params: {} 2022-09-09T22:48:11.939 DEBUG:boto:Token: None 2022-09-09T22:48:11.940 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/obj 2022-09-09T22:48:11.941 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:QEDcT8revkSEX8zhFz7VVQ0XYGk= 2022-09-09T22:48:11.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:QEDcT8revkSEX8zhFz7VVQ0XYGk=', 'Content-Length': '0'} 2022-09-09T22:48:11.944 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b81faf99700718ec-00631bc2ab-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.945 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:11.946 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:11.946 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.947 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.947 DEBUG:boto:Method: GET 2022-09-09T22:48:11.948 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.948 DEBUG:boto:Data: 2022-09-09T22:48:11.949 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.949 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:11.950 DEBUG:boto:Port: 8001 2022-09-09T22:48:11.950 DEBUG:boto:Params: {} 2022-09-09T22:48:11.951 DEBUG:boto:Token: None 2022-09-09T22:48:11.952 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/ 2022-09-09T22:48:11.952 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7wAJjRcIN30P68+RrLEZ6Ed9fvI= 2022-09-09T22:48:11.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7wAJjRcIN30P68+RrLEZ6Ed9fvI=', 'Content-Length': '0'} 2022-09-09T22:48:11.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000220634c202cb2c46-00631bc2ab-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:11.960 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:11.961 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:11.961 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:11.962 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.963 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.963 DEBUG:boto:Method: GET 2022-09-09T22:48:11.964 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:11.964 DEBUG:boto:Data: 2022-09-09T22:48:11.965 DEBUG:boto:Headers: {} 2022-09-09T22:48:11.965 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:11.966 DEBUG:boto:Port: 8000 2022-09-09T22:48:11.966 DEBUG:boto:Params: {} 2022-09-09T22:48:11.967 DEBUG:boto:Token: None 2022-09-09T22:48:11.968 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:11 GMT /jkcgbj-20/ 2022-09-09T22:48:11.968 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7wAJjRcIN30P68+RrLEZ6Ed9fvI= 2022-09-09T22:48:11.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:11 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7wAJjRcIN30P68+RrLEZ6Ed9fvI=', 'Content-Length': '0'} 2022-09-09T22:48:11.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029eec31eccddb09f-00631bc2ab-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:12.018 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:12.019 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-20 zones={test-zone3, test-zone2} 2022-09-09T22:48:12.020 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:12.020 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:12.021 DEBUG:boto:path=/jkcgbj-20/?versioning 2022-09-09T22:48:12.021 DEBUG:boto:auth_path=/jkcgbj-20/?versioning 2022-09-09T22:48:12.022 DEBUG:boto:Method: PUT 2022-09-09T22:48:12.023 DEBUG:boto:Path: /jkcgbj-20/?versioning 2022-09-09T22:48:12.023 DEBUG:boto:Data: Enabled Disabled 2022-09-09T22:48:12.024 DEBUG:boto:Headers: {} 2022-09-09T22:48:12.024 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:12.025 DEBUG:boto:Port: 8000 2022-09-09T22:48:12.025 DEBUG:boto:Params: {} 2022-09-09T22:48:12.026 DEBUG:boto:Token: None 2022-09-09T22:48:12.027 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:48:12 GMT /jkcgbj-20/?versioning 2022-09-09T22:48:12.028 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:PGqptlT6+jsaeb1h1U7gW/tA9oo= 2022-09-09T22:48:12.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:12 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:PGqptlT6+jsaeb1h1U7gW/tA9oo=', 'Content-Length': '228'} 2022-09-09T22:48:12.045 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b6fdde79ce3d8628-00631bc2ac-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:12.046 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:48:12.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:48:12.048 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:48:12.094 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:48:12.095 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:48:12.150 INFO:teuthology.orchestra.run.smithi005.stderr:No --period given, using current period=503b1d61-4513-48c8-a308-17b76219610c 2022-09-09T22:48:12.177 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T22:48:12.178 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:48:12.178 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:48:12.178 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:28:06.146147Z" 2022-09-09T22:48:12.179 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:48:12.179 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:48:12.179 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:48:12.179 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:48:12.180 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:48:12.180 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:48:12.180 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:48:12.181 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:30:37.757014Z" 2022-09-09T22:48:12.181 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:48:12.181 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:48:12.181 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662763692.038589_111.1", 2022-09-09T22:48:12.182 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:48:12.038589Z" 2022-09-09T22:48:12.182 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:48:12.182 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T22:48:12.184 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763692.038589_111.1'} 2022-09-09T22:48:12.184 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-09T22:48:12.185 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:48:12.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:48:12.186 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:48:12.232 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:48:12.233 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:48:12.278 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:48:12.274+0000 7f91646bedc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:48:12.279 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:48:12.274+0000 7f91646bedc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:48:12.304 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:48:12.304 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:48:12.304 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:48:12.305 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:48:12.305 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:48:12.305 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:48:12.306 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:48:12.306 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:48:12.306 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:48:12.307 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:48:12.307 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:48:12.307 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:48:12.308 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:48:12.309 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:48:12.309 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:48:12.309 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:48:12.310 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:48:12.310 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:48:12.310 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:48:12.311 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:48:12.311 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:48:12.311 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:48:12.311 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:48:12.312 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:48:12.312 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:48:12.312 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:48:12.313 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:48:12.313 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:48:12.313 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:48:12.314 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:48:12.314 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:48:12.314 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:48:12.314 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:48:12.315 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:48:12.315 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:48:12.315 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:48:12.315 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:48:12.316 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:48:12.316 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:48:12.316 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:48:12.317 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:48:12.317 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:48:12.318 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:48:12.319 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:48:12.319 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:48:12.320 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:48:12.320 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:48:12.320 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:48:12.320 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:48:12.321 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763692.038589_111.1", 2022-09-09T22:48:12.321 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:48:12.321 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:48:12.322 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:48:12.322 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:48:12.038589Z", 2022-09-09T22:48:12.322 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:48:12.323 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:48:12.323 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:48:12.323 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:48:12.323 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:48:12.324 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:48:12.324 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:48:12.324 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:48:12.324 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:48:12.325 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:48:12Z" 2022-09-09T22:48:12.325 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:48:12.326 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763692.038589_111.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:48:12.038589Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:48:12Z" } 2022-09-09T22:48:12.327 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763692.038589_111.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:48:12.038589Z', 'realm_epoch': 2}}] 2022-09-09T22:48:12.327 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763692.038589_111.1'} 2022-09-09T22:48:12.328 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763692.038589_111.1'} 2022-09-09T22:48:12.329 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-09T22:48:12.330 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-09T22:48:12.331 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:48:12.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:48:12.333 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:48:12.382 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:48:12.382 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:48:12.436 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:48:12.433+0000 7fe711a14dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:48:12.438 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:48:12.433+0000 7fe711a14dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:48:12.464 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:48:12.464 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T22:48:12.465 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T22:48:12.465 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T22:48:12.466 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T22:48:12.466 INFO:teuthology.orchestra.run.smithi005.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:48:12.467 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:48:12.467 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:48:12.468 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T22:48:12.468 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:48:12.469 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T22:48:12.469 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:48:12.470 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:48:12.470 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:48:12.471 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:48:12.471 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 1, 2022-09-09T22:48:12.472 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:48:12.472 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:48:12.472 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:48:12.473 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:48:12.473 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:48:12.474 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:48:12.474 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T22:48:12.475 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:48:12.475 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:48:12.476 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:48:12.476 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:48:12.477 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:48:12.477 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:48:12.478 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:48:12.478 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:48:12.479 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:48:12.479 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:48:12.480 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:48:12.480 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T22:48:12.481 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:48:12.481 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:48:12.481 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:48:12.482 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:48:12.483 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:48:12.483 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:48:12.483 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:48:12.484 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:48:12.484 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:48:12.485 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:48:12.485 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:48:12.486 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T22:48:12.486 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:48:12.487 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:48:12.487 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662763618.858372_97.1", 2022-09-09T22:48:12.488 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:48:12.488 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:48:12.489 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:48:12.489 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:46:58.858372Z", 2022-09-09T22:48:12.490 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:48:12.490 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:48:12.490 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:48:12.491 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:48:12.491 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:48:12.491 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T22:48:12.492 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 1, 2022-09-09T22:48:12.492 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 1 2022-09-09T22:48:12.492 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:48:12.492 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T22:48:12Z" 2022-09-09T22:48:12.493 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:48:12.494 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763618.858372_97.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:46:58.858372Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:48:12Z" } 2022-09-09T22:48:12.494 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763618.858372_97.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:46:58.858372Z', 'realm_epoch': 2}}] 2022-09-09T22:48:12.495 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763692.038589_111.1'} 2022-09-09T22:48:12.496 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763618.858372_97.1'} 2022-09-09T22:48:12.496 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1662763692.038589_111.1 target=1_1662763618.858372_97.1 2022-09-09T22:48:17.497 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:48:17.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:48:17.499 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:48:17.547 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:48:17.548 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:48:17.600 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:48:17.597+0000 7f23e0de5dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:48:17.602 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:48:17.597+0000 7f23e0de5dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:48:17.628 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:48:17.629 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T22:48:17.629 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T22:48:17.630 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T22:48:17.630 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T22:48:17.631 INFO:teuthology.orchestra.run.smithi005.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:48:17.631 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:48:17.632 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:48:17.632 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T22:48:17.633 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:48:17.633 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T22:48:17.634 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:48:17.634 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:48:17.635 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:48:17.635 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:48:17.636 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 1, 2022-09-09T22:48:17.636 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:48:17.637 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:48:17.637 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:48:17.638 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:48:17.638 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:48:17.639 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:48:17.639 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T22:48:17.640 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:48:17.640 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:48:17.641 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:48:17.641 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:48:17.641 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:48:17.642 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:48:17.642 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:48:17.643 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:48:17.643 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:48:17.643 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:48:17.644 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:48:17.644 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T22:48:17.644 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:48:17.644 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:48:17.645 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:48:17.645 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:48:17.645 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:48:17.646 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:48:17.646 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:48:17.646 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:48:17.646 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:48:17.647 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:48:17.647 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:48:17.647 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T22:48:17.647 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:48:17.648 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:48:17.648 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662763692.038589_111.1", 2022-09-09T22:48:17.648 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:48:17.648 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:48:17.649 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:48:17.649 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:48:12.038589Z", 2022-09-09T22:48:17.649 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:48:17.649 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:48:17.650 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:48:17.650 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:48:17.650 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:48:17.651 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T22:48:17.651 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 1, 2022-09-09T22:48:17.652 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 1 2022-09-09T22:48:17.652 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:48:17.653 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T22:48:17Z" 2022-09-09T22:48:17.653 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:48:17.655 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763692.038589_111.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:48:12.038589Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:48:17Z" } 2022-09-09T22:48:17.656 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763692.038589_111.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:48:12.038589Z', 'realm_epoch': 2}}] 2022-09-09T22:48:17.657 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763692.038589_111.1'} 2022-09-09T22:48:17.659 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763692.038589_111.1'} 2022-09-09T22:48:17.660 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-09T22:48:17.661 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-09T22:48:17.662 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:48:17.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:48:17.664 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:48:17.712 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:48:17.712 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:48:17.777 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:48:17.774+0000 7fe51a0cddc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:48:17.780 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:48:17.778+0000 7fe51a0cddc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:48:17.806 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:48:17.807 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:48:17.807 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:48:17.807 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:48:17.807 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:48:17.808 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:48:17.808 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:48:17.808 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:48:17.809 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:48:17.809 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:48:17.809 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:48:17.809 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:48:17.810 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:48:17.810 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:48:17.810 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:48:17.811 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:48:17.811 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:48:17.811 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:48:17.812 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:48:17.812 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:48:17.813 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:48:17.813 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:48:17.813 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:48:17.813 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:48:17.814 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:48:17.814 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:48:17.814 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:48:17.815 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:48:17.815 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:48:17.815 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:48:17.815 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:48:17.816 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:48:17.816 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:48:17.816 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:48:17.816 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:48:17.817 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:48:17.817 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:48:17.817 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:48:17.818 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:48:17.818 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:48:17.818 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:48:17.819 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:48:17.819 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:48:17.819 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:48:17.819 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:48:17.820 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:48:17.820 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:48:17.820 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:48:17.821 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:48:17.821 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763692.038589_111.1", 2022-09-09T22:48:17.821 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:48:17.821 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:48:17.822 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:48:17.822 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:48:12.038589Z", 2022-09-09T22:48:17.822 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:48:17.823 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:48:17.824 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:48:17.824 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:48:17.824 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:48:17.824 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:48:17.825 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:48:17.825 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:48:17.825 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:48:17.825 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:48:17Z" 2022-09-09T22:48:17.826 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:48:17.827 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763692.038589_111.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:48:12.038589Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:48:17Z" } 2022-09-09T22:48:17.828 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763692.038589_111.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:48:12.038589Z', 'realm_epoch': 2}}] 2022-09-09T22:48:17.829 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763692.038589_111.1'} 2022-09-09T22:48:17.829 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763692.038589_111.1'} 2022-09-09T22:48:17.830 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-09T22:48:17.830 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:17.831 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:17.831 DEBUG:boto:Method: HEAD 2022-09-09T22:48:17.832 DEBUG:boto:Path: /jkcgbj-20/ 2022-09-09T22:48:17.832 DEBUG:boto:Data: 2022-09-09T22:48:17.833 DEBUG:boto:Headers: {} 2022-09-09T22:48:17.834 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:17.835 DEBUG:boto:Port: 8000 2022-09-09T22:48:17.835 DEBUG:boto:Params: {} 2022-09-09T22:48:17.836 DEBUG:boto:Token: None 2022-09-09T22:48:17.836 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:17 GMT /jkcgbj-20/ 2022-09-09T22:48:17.837 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:3dCV5I++s2LIjqFf8qIQvxjK8dk= 2022-09-09T22:48:17.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:3dCV5I++s2LIjqFf8qIQvxjK8dk=', 'Content-Length': '0'} 2022-09-09T22:48:17.843 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000096b769bf6c23f5fe-00631bc2b1-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:17.844 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:17.845 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:17.845 DEBUG:boto:Method: PUT 2022-09-09T22:48:17.846 DEBUG:boto:Path: /jkcgbj-20/obj 2022-09-09T22:48:17.846 DEBUG:boto:Data: 2022-09-09T22:48:17.847 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-09T22:48:17.847 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:17.848 DEBUG:boto:Port: 8000 2022-09-09T22:48:17.848 DEBUG:boto:Params: {} 2022-09-09T22:48:17.849 DEBUG:boto:Token: None 2022-09-09T22:48:17.850 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 09 Sep 2022 22:48:17 GMT /jkcgbj-20/obj 2022-09-09T22:48:17.850 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:huFNUDEUVQ7WP8iq12ZP8rtaHps= 2022-09-09T22:48:17.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:48:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:huFNUDEUVQ7WP8iq12ZP8rtaHps='} 2022-09-09T22:48:17.878 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-amz-request-id', 'tx000006c0a2b356f0e9507-00631bc2b1-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:17.879 DEBUG:rgw_multi.tests:created new version id=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:17.880 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jkcgbj-20 2022-09-09T22:48:17.881 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:48:17.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:48:17.883 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:48:17.938 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:48:17.938 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:48:18.025 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:48:18.022+0000 7f4cb5f32dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:48:18.026 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000001.23.5, , ] 2022-09-09T22:48:18.026 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000003.26.12, , ] 2022-09-09T22:48:23.028 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:48:23.025+0000 7f4cb5f32dc0 1 bucket sync caught up with source: 2022-09-09T22:48:23.029 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000003.26.12, , ] 2022-09-09T22:48:23.029 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000003.26.12, , ] 2022-09-09T22:48:23.030 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:48:23.025+0000 7f4cb5f32dc0 0 bucket checkpoint complete 2022-09-09T22:48:23.058 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jkcgbj-20 2022-09-09T22:48:23.059 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:48:23.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:48:23.062 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:48:23.117 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:48:23.118 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:48:23.184 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:48:23.181+0000 7f5968d9adc0 1 bucket sync caught up with source: 2022-09-09T22:48:23.185 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000003.26.12, , ] 2022-09-09T22:48:23.185 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000003.26.12, , ] 2022-09-09T22:48:23.186 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:48:23.181+0000 7f5968d9adc0 0 bucket checkpoint complete 2022-09-09T22:48:23.211 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jkcgbj-20 2022-09-09T22:48:23.211 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:48:23.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:48:23.213 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:48:23.267 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:48:23.267 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:48:23.341 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:48:23.329+0000 7f03dc129dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:48:23.342 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000001.23.5, , ] 2022-09-09T22:48:23.342 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000003.26.12, , ] 2022-09-09T22:48:28.337 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:48:28.333+0000 7f03dc129dc0 1 bucket sync caught up with source: 2022-09-09T22:48:28.338 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000003.26.12, , ] 2022-09-09T22:48:28.338 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000003.26.12, , ] 2022-09-09T22:48:28.339 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:48:28.333+0000 7f03dc129dc0 0 bucket checkpoint complete 2022-09-09T22:48:28.367 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jkcgbj-20 2022-09-09T22:48:28.368 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:48:28.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:48:28.370 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:48:28.425 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:48:28.426 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:48:28.495 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:48:28.493+0000 7f015ec14dc0 1 bucket sync caught up with source: 2022-09-09T22:48:28.496 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000003.35.12, , ] 2022-09-09T22:48:28.497 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000003.35.12, , ] 2022-09-09T22:48:28.497 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:48:28.493+0000 7f015ec14dc0 0 bucket checkpoint complete 2022-09-09T22:48:28.524 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jkcgbj-20 2022-09-09T22:48:28.525 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:48:28.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:48:28.526 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:48:28.579 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:48:28.579 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:48:28.649 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:48:28.645+0000 7fb57eb21dc0 1 bucket sync caught up with source: 2022-09-09T22:48:28.650 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000003.35.12, , ] 2022-09-09T22:48:28.650 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000003.35.12, , ] 2022-09-09T22:48:28.650 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:48:28.645+0000 7fb57eb21dc0 0 bucket checkpoint complete 2022-09-09T22:48:28.677 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jkcgbj-20 2022-09-09T22:48:28.678 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jkcgbj-20 2022-09-09T22:48:28.678 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:48:28.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:48:28.679 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:48:28.731 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:48:28.731 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:48:28.798 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:48:28.793+0000 7f2916110dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:48:28.798 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000001.46.5, , ] 2022-09-09T22:48:28.799 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000004.53.9, , ] 2022-09-09T22:48:33.801 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:48:33.797+0000 7f2916110dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:48:33.802 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000001.46.5, , ] 2022-09-09T22:48:33.802 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000004.53.9, , ] 2022-09-09T22:48:38.804 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:48:38.801+0000 7f2916110dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:48:38.804 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000001.46.5, , ] 2022-09-09T22:48:38.804 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000004.53.9, , ] 2022-09-09T22:48:43.807 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:48:43.805+0000 7f2916110dc0 1 bucket sync caught up with source: 2022-09-09T22:48:43.807 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000004.53.9, , ] 2022-09-09T22:48:43.807 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000004.53.9, , ] 2022-09-09T22:48:43.808 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:48:43.805+0000 7f2916110dc0 0 bucket checkpoint complete 2022-09-09T22:48:43.836 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jkcgbj-20 2022-09-09T22:48:43.837 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:48:43.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:48:43.839 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:48:43.894 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:48:43.894 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:48:43.961 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:48:43.957+0000 7fe125a66dc0 1 bucket sync caught up with source: 2022-09-09T22:48:43.962 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000004.53.9, , ] 2022-09-09T22:48:43.962 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000004.53.9, , ] 2022-09-09T22:48:43.963 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:48:43.957+0000 7fe125a66dc0 0 bucket checkpoint complete 2022-09-09T22:48:43.990 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jkcgbj-20 2022-09-09T22:48:43.991 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-20 zones={test-zone2, test-zone1} 2022-09-09T22:48:43.992 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:43.992 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:43.993 DEBUG:boto:Method: HEAD 2022-09-09T22:48:43.993 DEBUG:boto:Path: /jkcgbj-20/ 2022-09-09T22:48:43.994 DEBUG:boto:Data: 2022-09-09T22:48:43.994 DEBUG:boto:Headers: {} 2022-09-09T22:48:43.995 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:43.995 DEBUG:boto:Port: 8000 2022-09-09T22:48:43.996 DEBUG:boto:Params: {} 2022-09-09T22:48:43.996 DEBUG:boto:Token: None 2022-09-09T22:48:43.997 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:43 GMT /jkcgbj-20/ 2022-09-09T22:48:43.998 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sCDgycBwT2bxlp77H+nO6NB0eQI= 2022-09-09T22:48:43.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sCDgycBwT2bxlp77H+nO6NB0eQI=', 'Content-Length': '0'} 2022-09-09T22:48:44.003 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b957d2738416bb30-00631bc2cb-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:48:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.004 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:44.004 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:44.005 DEBUG:boto:Method: HEAD 2022-09-09T22:48:44.005 DEBUG:boto:Path: /jkcgbj-20/ 2022-09-09T22:48:44.006 DEBUG:boto:Data: 2022-09-09T22:48:44.006 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.007 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:44.007 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.008 DEBUG:boto:Params: {} 2022-09-09T22:48:44.008 DEBUG:boto:Token: None 2022-09-09T22:48:44.009 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/ 2022-09-09T22:48:44.010 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CH+x3bdpsrYfWedTXM0lKxxmFoE= 2022-09-09T22:48:44.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CH+x3bdpsrYfWedTXM0lKxxmFoE=', 'Content-Length': '0'} 2022-09-09T22:48:44.016 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1629b00bbd61b06-00631bc2cc-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.017 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:48:44.017 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:44.018 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:44.018 DEBUG:boto:path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:44.019 DEBUG:boto:auth_path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:44.019 DEBUG:boto:Method: GET 2022-09-09T22:48:44.020 DEBUG:boto:Path: /jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:44.021 DEBUG:boto:Data: 2022-09-09T22:48:44.021 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.022 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:44.022 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.023 DEBUG:boto:Params: {} 2022-09-09T22:48:44.023 DEBUG:boto:Token: None 2022-09-09T22:48:44.024 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/?versions 2022-09-09T22:48:44.025 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:lz/sY8I73KVszCtbhvcEdPsHWMQ= 2022-09-09T22:48:44.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:lz/sY8I73KVszCtbhvcEdPsHWMQ=', 'Content-Length': '0'} 2022-09-09T22:48:44.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084cd8861103562cd-00631bc2cc-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.032 DEBUG:boto:b'jkcgbj-20999falseobji9PSHBUuQ8xfOhtiVzZhEqzCuln.uXjtrue2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:44.033 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:48:44.033 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:48:44.034 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:48:44.034 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:44.035 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:44.035 DEBUG:boto:path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:44.036 DEBUG:boto:auth_path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:44.036 DEBUG:boto:Method: GET 2022-09-09T22:48:44.037 DEBUG:boto:Path: /jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:44.037 DEBUG:boto:Data: 2022-09-09T22:48:44.038 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.038 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:44.039 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.039 DEBUG:boto:Params: {} 2022-09-09T22:48:44.040 DEBUG:boto:Token: None 2022-09-09T22:48:44.040 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/?versions 2022-09-09T22:48:44.041 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:lz/sY8I73KVszCtbhvcEdPsHWMQ= 2022-09-09T22:48:44.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:lz/sY8I73KVszCtbhvcEdPsHWMQ=', 'Content-Length': '0'} 2022-09-09T22:48:44.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c26514a9eadf3bd-00631bc2cc-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.094 DEBUG:boto:b'jkcgbj-20999falseobji9PSHBUuQ8xfOhtiVzZhEqzCuln.uXjtrue2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:44.095 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:48:44.096 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:48:44.097 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:44.098 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:44.099 DEBUG:boto:path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:44.100 DEBUG:boto:auth_path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:44.100 DEBUG:boto:Method: GET 2022-09-09T22:48:44.101 DEBUG:boto:Path: /jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:44.102 DEBUG:boto:Data: 2022-09-09T22:48:44.103 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.104 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:44.105 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.106 DEBUG:boto:Params: {} 2022-09-09T22:48:44.107 DEBUG:boto:Token: None 2022-09-09T22:48:44.108 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/?versions 2022-09-09T22:48:44.109 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:lz/sY8I73KVszCtbhvcEdPsHWMQ= 2022-09-09T22:48:44.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:lz/sY8I73KVszCtbhvcEdPsHWMQ=', 'Content-Length': '0'} 2022-09-09T22:48:44.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071e6843ee2503449-00631bc2cc-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.116 DEBUG:boto:b'jkcgbj-20999falseobji9PSHBUuQ8xfOhtiVzZhEqzCuln.uXjtrue2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:44.117 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:44.118 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:44.119 DEBUG:boto:path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:44.120 DEBUG:boto:auth_path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:44.120 DEBUG:boto:Method: GET 2022-09-09T22:48:44.121 DEBUG:boto:Path: /jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:44.121 DEBUG:boto:Data: 2022-09-09T22:48:44.132 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.133 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:44.134 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.134 DEBUG:boto:Params: {} 2022-09-09T22:48:44.135 DEBUG:boto:Token: None 2022-09-09T22:48:44.136 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/?versions 2022-09-09T22:48:44.136 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:lz/sY8I73KVszCtbhvcEdPsHWMQ= 2022-09-09T22:48:44.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:lz/sY8I73KVszCtbhvcEdPsHWMQ=', 'Content-Length': '0'} 2022-09-09T22:48:44.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0ad3547f458a559-00631bc2cc-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.143 DEBUG:boto:b'jkcgbj-20999falseobji9PSHBUuQ8xfOhtiVzZhEqzCuln.uXjtrue2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:44.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:48:44.145 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:44.146 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:44.147 DEBUG:boto:path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.148 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.149 DEBUG:boto:Method: GET 2022-09-09T22:48:44.150 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.151 DEBUG:boto:Data: 2022-09-09T22:48:44.152 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.153 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:44.154 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.155 DEBUG:boto:Params: {} 2022-09-09T22:48:44.156 DEBUG:boto:Token: None 2022-09-09T22:48:44.157 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.158 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YixcSJxgf3p7LQDrCZucPATBEp8= 2022-09-09T22:48:44.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YixcSJxgf3p7LQDrCZucPATBEp8=', 'Content-Length': '0'} 2022-09-09T22:48:44.162 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f86a73ccc80b3134-00631bc2cc-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.163 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:44.163 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:44.164 DEBUG:boto:path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.164 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.165 DEBUG:boto:Method: GET 2022-09-09T22:48:44.165 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.166 DEBUG:boto:Data: 2022-09-09T22:48:44.167 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.167 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:44.168 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.168 DEBUG:boto:Params: {} 2022-09-09T22:48:44.169 DEBUG:boto:Token: None 2022-09-09T22:48:44.169 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.170 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YixcSJxgf3p7LQDrCZucPATBEp8= 2022-09-09T22:48:44.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YixcSJxgf3p7LQDrCZucPATBEp8=', 'Content-Length': '0'} 2022-09-09T22:48:44.174 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005df67501c71f3f8e-00631bc2cc-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.175 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:44.175 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:44.176 DEBUG:boto:path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.177 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.177 DEBUG:boto:Method: HEAD 2022-09-09T22:48:44.178 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.179 DEBUG:boto:Data: 2022-09-09T22:48:44.180 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.181 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:44.182 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.183 DEBUG:boto:Params: {} 2022-09-09T22:48:44.184 DEBUG:boto:Token: None 2022-09-09T22:48:44.185 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.186 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8ZVUAGRbplObJjqy1WNbHbPnRgQ= 2022-09-09T22:48:44.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8ZVUAGRbplObJjqy1WNbHbPnRgQ=', 'Content-Length': '0'} 2022-09-09T22:48:44.190 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000007d0191abf001ad2-00631bc2cc-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.191 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:44.192 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:44.193 DEBUG:boto:path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.194 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.195 DEBUG:boto:Method: HEAD 2022-09-09T22:48:44.195 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.196 DEBUG:boto:Data: 2022-09-09T22:48:44.197 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.198 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:44.198 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.199 DEBUG:boto:Params: {} 2022-09-09T22:48:44.199 DEBUG:boto:Token: None 2022-09-09T22:48:44.200 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.200 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8ZVUAGRbplObJjqy1WNbHbPnRgQ= 2022-09-09T22:48:44.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8ZVUAGRbplObJjqy1WNbHbPnRgQ=', 'Content-Length': '0'} 2022-09-09T22:48:44.204 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000063e005f181cdfaff-00631bc2cc-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:48:44.205 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:44.206 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:44.206 DEBUG:boto:path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.207 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.208 DEBUG:boto:Method: GET 2022-09-09T22:48:44.208 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.209 DEBUG:boto:Data: 2022-09-09T22:48:44.209 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.210 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:44.210 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.211 DEBUG:boto:Params: {} 2022-09-09T22:48:44.211 DEBUG:boto:Token: None 2022-09-09T22:48:44.212 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.213 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YixcSJxgf3p7LQDrCZucPATBEp8= 2022-09-09T22:48:44.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YixcSJxgf3p7LQDrCZucPATBEp8=', 'Content-Length': '0'} 2022-09-09T22:48:44.216 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000043148b5e6742ae92-00631bc2cc-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.218 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:44.219 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:44.220 DEBUG:boto:path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.221 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.222 DEBUG:boto:Method: GET 2022-09-09T22:48:44.223 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.224 DEBUG:boto:Data: 2022-09-09T22:48:44.225 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.225 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:44.226 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.227 DEBUG:boto:Params: {} 2022-09-09T22:48:44.229 DEBUG:boto:Token: None 2022-09-09T22:48:44.230 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.231 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YixcSJxgf3p7LQDrCZucPATBEp8= 2022-09-09T22:48:44.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YixcSJxgf3p7LQDrCZucPATBEp8=', 'Content-Length': '0'} 2022-09-09T22:48:44.236 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d9a6c191bf4519f2-00631bc2cc-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.237 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:44.238 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:44.240 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:44.240 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:44.241 DEBUG:boto:Method: GET 2022-09-09T22:48:44.243 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:44.244 DEBUG:boto:Data: 2022-09-09T22:48:44.245 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.246 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:44.247 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.248 DEBUG:boto:Params: {} 2022-09-09T22:48:44.249 DEBUG:boto:Token: None 2022-09-09T22:48:44.250 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/ 2022-09-09T22:48:44.251 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sqHBwFPZu4J48MNEyd8CiHK7PYw= 2022-09-09T22:48:44.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sqHBwFPZu4J48MNEyd8CiHK7PYw=', 'Content-Length': '0'} 2022-09-09T22:48:44.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000582a155185143f16-00631bc2cc-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.260 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:44.261 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:44.262 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:44.263 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:44.264 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:44.265 DEBUG:boto:Method: GET 2022-09-09T22:48:44.266 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:44.267 DEBUG:boto:Data: 2022-09-09T22:48:44.268 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.269 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:44.270 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.271 DEBUG:boto:Params: {} 2022-09-09T22:48:44.273 DEBUG:boto:Token: None 2022-09-09T22:48:44.274 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/ 2022-09-09T22:48:44.275 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sqHBwFPZu4J48MNEyd8CiHK7PYw= 2022-09-09T22:48:44.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sqHBwFPZu4J48MNEyd8CiHK7PYw=', 'Content-Length': '0'} 2022-09-09T22:48:44.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020b55e76c3c49699-00631bc2cc-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.283 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:44.285 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:44.285 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:44.286 DEBUG:boto:Method: HEAD 2022-09-09T22:48:44.287 DEBUG:boto:Path: /jkcgbj-20/obj 2022-09-09T22:48:44.287 DEBUG:boto:Data: 2022-09-09T22:48:44.288 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.288 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:44.289 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.289 DEBUG:boto:Params: {} 2022-09-09T22:48:44.290 DEBUG:boto:Token: None 2022-09-09T22:48:44.291 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/obj 2022-09-09T22:48:44.292 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:WJLp4PreGioScYb7cgIzna/avPM= 2022-09-09T22:48:44.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:WJLp4PreGioScYb7cgIzna/avPM=', 'Content-Length': '0'} 2022-09-09T22:48:44.296 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000008f172092325dc6c-00631bc2cc-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.298 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:44.299 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:44.300 DEBUG:boto:Method: HEAD 2022-09-09T22:48:44.300 DEBUG:boto:Path: /jkcgbj-20/obj 2022-09-09T22:48:44.301 DEBUG:boto:Data: 2022-09-09T22:48:44.302 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.303 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:44.303 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.304 DEBUG:boto:Params: {} 2022-09-09T22:48:44.305 DEBUG:boto:Token: None 2022-09-09T22:48:44.305 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/obj 2022-09-09T22:48:44.306 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:WJLp4PreGioScYb7cgIzna/avPM= 2022-09-09T22:48:44.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:WJLp4PreGioScYb7cgIzna/avPM=', 'Content-Length': '0'} 2022-09-09T22:48:44.310 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000054e0ece5a23f487f-00631bc2cc-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:48:44.312 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:44.313 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:44.314 DEBUG:boto:path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.315 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.316 DEBUG:boto:Method: GET 2022-09-09T22:48:44.316 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.317 DEBUG:boto:Data: 2022-09-09T22:48:44.318 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.318 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:44.319 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.319 DEBUG:boto:Params: {} 2022-09-09T22:48:44.320 DEBUG:boto:Token: None 2022-09-09T22:48:44.320 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.321 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YixcSJxgf3p7LQDrCZucPATBEp8= 2022-09-09T22:48:44.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YixcSJxgf3p7LQDrCZucPATBEp8=', 'Content-Length': '0'} 2022-09-09T22:48:44.325 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002c5327165d0a80c4-00631bc2cc-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.325 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:44.326 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:44.326 DEBUG:boto:path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.327 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.327 DEBUG:boto:Method: GET 2022-09-09T22:48:44.328 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.329 DEBUG:boto:Data: 2022-09-09T22:48:44.329 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.330 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:44.330 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.331 DEBUG:boto:Params: {} 2022-09-09T22:48:44.331 DEBUG:boto:Token: None 2022-09-09T22:48:44.332 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.332 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YixcSJxgf3p7LQDrCZucPATBEp8= 2022-09-09T22:48:44.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YixcSJxgf3p7LQDrCZucPATBEp8=', 'Content-Length': '0'} 2022-09-09T22:48:44.336 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000060a6cfb14c81478b-00631bc2cc-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.337 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:44.337 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:44.338 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:44.338 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:44.339 DEBUG:boto:Method: GET 2022-09-09T22:48:44.340 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:44.340 DEBUG:boto:Data: 2022-09-09T22:48:44.341 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.341 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:44.342 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.342 DEBUG:boto:Params: {} 2022-09-09T22:48:44.343 DEBUG:boto:Token: None 2022-09-09T22:48:44.343 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/ 2022-09-09T22:48:44.344 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sqHBwFPZu4J48MNEyd8CiHK7PYw= 2022-09-09T22:48:44.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sqHBwFPZu4J48MNEyd8CiHK7PYw=', 'Content-Length': '0'} 2022-09-09T22:48:44.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b63e7983a64032a2-00631bc2cc-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.394 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:44.395 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:44.396 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:44.397 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:44.397 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:44.398 DEBUG:boto:Method: GET 2022-09-09T22:48:44.398 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:44.399 DEBUG:boto:Data: 2022-09-09T22:48:44.399 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.400 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:44.400 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.401 DEBUG:boto:Params: {} 2022-09-09T22:48:44.401 DEBUG:boto:Token: None 2022-09-09T22:48:44.402 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/ 2022-09-09T22:48:44.403 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sqHBwFPZu4J48MNEyd8CiHK7PYw= 2022-09-09T22:48:44.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sqHBwFPZu4J48MNEyd8CiHK7PYw=', 'Content-Length': '0'} 2022-09-09T22:48:44.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066be62b79f8e212c-00631bc2cc-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.410 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:44.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:48:44.413 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:44.414 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:44.414 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:44.415 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:44.415 DEBUG:boto:Method: GET 2022-09-09T22:48:44.416 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:44.417 DEBUG:boto:Data: 2022-09-09T22:48:44.417 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.418 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:44.418 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.419 DEBUG:boto:Params: {} 2022-09-09T22:48:44.419 DEBUG:boto:Token: None 2022-09-09T22:48:44.420 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:44.421 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:DiNDesKbFaKErBbeYFtJKq5dIt0= 2022-09-09T22:48:44.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:DiNDesKbFaKErBbeYFtJKq5dIt0=', 'Content-Length': '0'} 2022-09-09T22:48:44.425 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b70597aa443cdfb5-00631bc2cc-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.425 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:44.426 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:44.427 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:44.427 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:44.428 DEBUG:boto:Method: GET 2022-09-09T22:48:44.428 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:44.429 DEBUG:boto:Data: 2022-09-09T22:48:44.429 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.430 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:44.430 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.431 DEBUG:boto:Params: {} 2022-09-09T22:48:44.432 DEBUG:boto:Token: None 2022-09-09T22:48:44.433 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:44.434 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:DiNDesKbFaKErBbeYFtJKq5dIt0= 2022-09-09T22:48:44.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:DiNDesKbFaKErBbeYFtJKq5dIt0=', 'Content-Length': '0'} 2022-09-09T22:48:44.438 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000099b747f9b8d2257b-00631bc2cc-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.439 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:44.440 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:44.441 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:44.442 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:44.442 DEBUG:boto:Method: HEAD 2022-09-09T22:48:44.443 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:44.444 DEBUG:boto:Data: 2022-09-09T22:48:44.445 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.446 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:44.446 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.447 DEBUG:boto:Params: {} 2022-09-09T22:48:44.448 DEBUG:boto:Token: None 2022-09-09T22:48:44.448 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:44.449 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:mJt6QYIyiMqIH/twQ/zJgcIdMv4= 2022-09-09T22:48:44.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:mJt6QYIyiMqIH/twQ/zJgcIdMv4=', 'Content-Length': '0'} 2022-09-09T22:48:44.453 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000100c17fb53c4f34f-00631bc2cc-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.454 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:44.455 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:44.456 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:44.457 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:44.458 DEBUG:boto:Method: HEAD 2022-09-09T22:48:44.459 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:44.460 DEBUG:boto:Data: 2022-09-09T22:48:44.461 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.462 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:44.463 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.463 DEBUG:boto:Params: {} 2022-09-09T22:48:44.464 DEBUG:boto:Token: None 2022-09-09T22:48:44.465 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:44.465 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:mJt6QYIyiMqIH/twQ/zJgcIdMv4= 2022-09-09T22:48:44.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:mJt6QYIyiMqIH/twQ/zJgcIdMv4=', 'Content-Length': '0'} 2022-09-09T22:48:44.469 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000044579fd99821ce11-00631bc2cc-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:48:44.470 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:44.471 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:44.471 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:44.472 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:44.472 DEBUG:boto:Method: GET 2022-09-09T22:48:44.473 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:44.473 DEBUG:boto:Data: 2022-09-09T22:48:44.474 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.475 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:44.475 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.476 DEBUG:boto:Params: {} 2022-09-09T22:48:44.476 DEBUG:boto:Token: None 2022-09-09T22:48:44.477 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:44.477 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:DiNDesKbFaKErBbeYFtJKq5dIt0= 2022-09-09T22:48:44.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:DiNDesKbFaKErBbeYFtJKq5dIt0=', 'Content-Length': '0'} 2022-09-09T22:48:44.481 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009eb8d9e678f25e7a-00631bc2cc-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.482 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:44.482 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:44.483 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:44.483 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:44.484 DEBUG:boto:Method: GET 2022-09-09T22:48:44.485 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:44.485 DEBUG:boto:Data: 2022-09-09T22:48:44.486 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.486 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:44.487 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.487 DEBUG:boto:Params: {} 2022-09-09T22:48:44.488 DEBUG:boto:Token: None 2022-09-09T22:48:44.488 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:44.489 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:DiNDesKbFaKErBbeYFtJKq5dIt0= 2022-09-09T22:48:44.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:DiNDesKbFaKErBbeYFtJKq5dIt0=', 'Content-Length': '0'} 2022-09-09T22:48:44.494 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a6f307c31f3bbfcd-00631bc2cc-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.495 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:44.495 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:44.496 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:44.496 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:44.497 DEBUG:boto:Method: GET 2022-09-09T22:48:44.498 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:44.498 DEBUG:boto:Data: 2022-09-09T22:48:44.499 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.499 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:44.500 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.501 DEBUG:boto:Params: {} 2022-09-09T22:48:44.502 DEBUG:boto:Token: None 2022-09-09T22:48:44.503 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/ 2022-09-09T22:48:44.503 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sqHBwFPZu4J48MNEyd8CiHK7PYw= 2022-09-09T22:48:44.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sqHBwFPZu4J48MNEyd8CiHK7PYw=', 'Content-Length': '0'} 2022-09-09T22:48:44.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb7d4e48d14ac99b-00631bc2cc-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.511 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:44.512 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:44.512 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:44.513 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:44.513 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:44.514 DEBUG:boto:Method: GET 2022-09-09T22:48:44.514 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:44.515 DEBUG:boto:Data: 2022-09-09T22:48:44.515 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.516 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:44.516 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.517 DEBUG:boto:Params: {} 2022-09-09T22:48:44.517 DEBUG:boto:Token: None 2022-09-09T22:48:44.518 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/ 2022-09-09T22:48:44.519 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sqHBwFPZu4J48MNEyd8CiHK7PYw= 2022-09-09T22:48:44.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sqHBwFPZu4J48MNEyd8CiHK7PYw=', 'Content-Length': '0'} 2022-09-09T22:48:44.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014b2629ec40835d7-00631bc2cc-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.525 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:44.526 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:44.527 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:44.527 DEBUG:boto:Method: HEAD 2022-09-09T22:48:44.528 DEBUG:boto:Path: /jkcgbj-20/obj 2022-09-09T22:48:44.528 DEBUG:boto:Data: 2022-09-09T22:48:44.529 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.529 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:44.530 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.530 DEBUG:boto:Params: {} 2022-09-09T22:48:44.531 DEBUG:boto:Token: None 2022-09-09T22:48:44.531 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/obj 2022-09-09T22:48:44.532 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:WJLp4PreGioScYb7cgIzna/avPM= 2022-09-09T22:48:44.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:WJLp4PreGioScYb7cgIzna/avPM=', 'Content-Length': '0'} 2022-09-09T22:48:44.536 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b3667abe4c71bbc6-00631bc2cc-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.537 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:44.538 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:44.538 DEBUG:boto:Method: HEAD 2022-09-09T22:48:44.539 DEBUG:boto:Path: /jkcgbj-20/obj 2022-09-09T22:48:44.539 DEBUG:boto:Data: 2022-09-09T22:48:44.540 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.540 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:44.541 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.541 DEBUG:boto:Params: {} 2022-09-09T22:48:44.542 DEBUG:boto:Token: None 2022-09-09T22:48:44.542 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/obj 2022-09-09T22:48:44.543 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:WJLp4PreGioScYb7cgIzna/avPM= 2022-09-09T22:48:44.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:WJLp4PreGioScYb7cgIzna/avPM=', 'Content-Length': '0'} 2022-09-09T22:48:44.547 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003d0965e673dd2a08-00631bc2cc-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:48:44.549 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:44.549 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:44.550 DEBUG:boto:path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.550 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.551 DEBUG:boto:Method: GET 2022-09-09T22:48:44.551 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.552 DEBUG:boto:Data: 2022-09-09T22:48:44.552 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.553 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:44.553 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.554 DEBUG:boto:Params: {} 2022-09-09T22:48:44.554 DEBUG:boto:Token: None 2022-09-09T22:48:44.555 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.556 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YixcSJxgf3p7LQDrCZucPATBEp8= 2022-09-09T22:48:44.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YixcSJxgf3p7LQDrCZucPATBEp8=', 'Content-Length': '0'} 2022-09-09T22:48:44.559 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000015e174eeb202722e-00631bc2cc-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.560 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:44.561 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:44.561 DEBUG:boto:path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.562 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.562 DEBUG:boto:Method: GET 2022-09-09T22:48:44.563 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.563 DEBUG:boto:Data: 2022-09-09T22:48:44.564 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.564 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:44.565 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.566 DEBUG:boto:Params: {} 2022-09-09T22:48:44.566 DEBUG:boto:Token: None 2022-09-09T22:48:44.567 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.568 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YixcSJxgf3p7LQDrCZucPATBEp8= 2022-09-09T22:48:44.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YixcSJxgf3p7LQDrCZucPATBEp8=', 'Content-Length': '0'} 2022-09-09T22:48:44.572 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008c3b05b4bcfd5f0e-00631bc2cc-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.573 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:44.573 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:44.574 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:44.574 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:44.575 DEBUG:boto:Method: GET 2022-09-09T22:48:44.575 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:44.576 DEBUG:boto:Data: 2022-09-09T22:48:44.576 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.577 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:44.577 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.578 DEBUG:boto:Params: {} 2022-09-09T22:48:44.579 DEBUG:boto:Token: None 2022-09-09T22:48:44.579 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/ 2022-09-09T22:48:44.580 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sqHBwFPZu4J48MNEyd8CiHK7PYw= 2022-09-09T22:48:44.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sqHBwFPZu4J48MNEyd8CiHK7PYw=', 'Content-Length': '0'} 2022-09-09T22:48:44.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5ba9180e06587d3-00631bc2cc-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.630 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:44.631 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:44.632 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:44.632 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:44.633 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:44.633 DEBUG:boto:Method: GET 2022-09-09T22:48:44.634 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:44.635 DEBUG:boto:Data: 2022-09-09T22:48:44.635 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.636 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:44.636 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.637 DEBUG:boto:Params: {} 2022-09-09T22:48:44.638 DEBUG:boto:Token: None 2022-09-09T22:48:44.638 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/ 2022-09-09T22:48:44.639 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sqHBwFPZu4J48MNEyd8CiHK7PYw= 2022-09-09T22:48:44.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sqHBwFPZu4J48MNEyd8CiHK7PYw=', 'Content-Length': '0'} 2022-09-09T22:48:44.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c542e0593e25bf8d-00631bc2cc-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.646 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:44.646 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-20 zones={test-zone2, test-zone1} 2022-09-09T22:48:44.647 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-20 zones={test-zone3, test-zone1} 2022-09-09T22:48:44.647 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:44.648 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:44.649 DEBUG:boto:Method: HEAD 2022-09-09T22:48:44.649 DEBUG:boto:Path: /jkcgbj-20/ 2022-09-09T22:48:44.650 DEBUG:boto:Data: 2022-09-09T22:48:44.651 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.652 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:44.653 DEBUG:boto:Port: 8001 2022-09-09T22:48:44.654 DEBUG:boto:Params: {} 2022-09-09T22:48:44.655 DEBUG:boto:Token: None 2022-09-09T22:48:44.656 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/ 2022-09-09T22:48:44.656 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CH+x3bdpsrYfWedTXM0lKxxmFoE= 2022-09-09T22:48:44.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CH+x3bdpsrYfWedTXM0lKxxmFoE=', 'Content-Length': '0'} 2022-09-09T22:48:44.662 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007196ed8b9130c994-00631bc2cc-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.662 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:44.663 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:44.663 DEBUG:boto:Method: HEAD 2022-09-09T22:48:44.664 DEBUG:boto:Path: /jkcgbj-20/ 2022-09-09T22:48:44.664 DEBUG:boto:Data: 2022-09-09T22:48:44.665 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.665 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:44.666 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.666 DEBUG:boto:Params: {} 2022-09-09T22:48:44.667 DEBUG:boto:Token: None 2022-09-09T22:48:44.668 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/ 2022-09-09T22:48:44.668 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CH+x3bdpsrYfWedTXM0lKxxmFoE= 2022-09-09T22:48:44.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CH+x3bdpsrYfWedTXM0lKxxmFoE=', 'Content-Length': '0'} 2022-09-09T22:48:44.673 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3cbdb292a160a0d-00631bc2cc-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.674 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:48:44.675 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:44.675 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:44.676 DEBUG:boto:path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:44.676 DEBUG:boto:auth_path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:44.677 DEBUG:boto:Method: GET 2022-09-09T22:48:44.677 DEBUG:boto:Path: /jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:44.678 DEBUG:boto:Data: 2022-09-09T22:48:44.678 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.679 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:44.679 DEBUG:boto:Port: 8001 2022-09-09T22:48:44.680 DEBUG:boto:Params: {} 2022-09-09T22:48:44.680 DEBUG:boto:Token: None 2022-09-09T22:48:44.681 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/?versions 2022-09-09T22:48:44.681 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:lz/sY8I73KVszCtbhvcEdPsHWMQ= 2022-09-09T22:48:44.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:lz/sY8I73KVszCtbhvcEdPsHWMQ=', 'Content-Length': '0'} 2022-09-09T22:48:44.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001424603defc3e23b-00631bc2cc-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.688 DEBUG:boto:b'jkcgbj-20999falseobji9PSHBUuQ8xfOhtiVzZhEqzCuln.uXjtrue2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:44.689 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:48:44.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:48:44.690 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:48:44.691 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:44.691 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:44.692 DEBUG:boto:path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:44.692 DEBUG:boto:auth_path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:44.693 DEBUG:boto:Method: GET 2022-09-09T22:48:44.693 DEBUG:boto:Path: /jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:44.694 DEBUG:boto:Data: 2022-09-09T22:48:44.694 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.695 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:44.695 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.696 DEBUG:boto:Params: {} 2022-09-09T22:48:44.696 DEBUG:boto:Token: None 2022-09-09T22:48:44.697 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/?versions 2022-09-09T22:48:44.697 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:lz/sY8I73KVszCtbhvcEdPsHWMQ= 2022-09-09T22:48:44.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:lz/sY8I73KVszCtbhvcEdPsHWMQ=', 'Content-Length': '0'} 2022-09-09T22:48:44.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004730e9449b82caeb-00631bc2cc-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.704 DEBUG:boto:b'jkcgbj-20999falseobji9PSHBUuQ8xfOhtiVzZhEqzCuln.uXjtrue2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:44.705 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:48:44.705 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:48:44.706 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:44.706 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:44.707 DEBUG:boto:path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:44.707 DEBUG:boto:auth_path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:44.708 DEBUG:boto:Method: GET 2022-09-09T22:48:44.708 DEBUG:boto:Path: /jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:44.709 DEBUG:boto:Data: 2022-09-09T22:48:44.709 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.710 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:44.710 DEBUG:boto:Port: 8001 2022-09-09T22:48:44.711 DEBUG:boto:Params: {} 2022-09-09T22:48:44.711 DEBUG:boto:Token: None 2022-09-09T22:48:44.712 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/?versions 2022-09-09T22:48:44.712 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:lz/sY8I73KVszCtbhvcEdPsHWMQ= 2022-09-09T22:48:44.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:lz/sY8I73KVszCtbhvcEdPsHWMQ=', 'Content-Length': '0'} 2022-09-09T22:48:44.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048688fdc6fd8bc1b-00631bc2cc-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.766 DEBUG:boto:b'jkcgbj-20999falseobji9PSHBUuQ8xfOhtiVzZhEqzCuln.uXjtrue2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:44.767 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:44.768 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:44.768 DEBUG:boto:path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:44.769 DEBUG:boto:auth_path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:44.769 DEBUG:boto:Method: GET 2022-09-09T22:48:44.770 DEBUG:boto:Path: /jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:44.771 DEBUG:boto:Data: 2022-09-09T22:48:44.771 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.772 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:44.773 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.774 DEBUG:boto:Params: {} 2022-09-09T22:48:44.775 DEBUG:boto:Token: None 2022-09-09T22:48:44.776 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/?versions 2022-09-09T22:48:44.777 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:lz/sY8I73KVszCtbhvcEdPsHWMQ= 2022-09-09T22:48:44.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:lz/sY8I73KVszCtbhvcEdPsHWMQ=', 'Content-Length': '0'} 2022-09-09T22:48:44.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8f54f7ade97a50d-00631bc2cc-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.830 DEBUG:boto:b'jkcgbj-20999falseobji9PSHBUuQ8xfOhtiVzZhEqzCuln.uXjtrue2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:44.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:48:44.832 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:44.832 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:44.833 DEBUG:boto:path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.833 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.834 DEBUG:boto:Method: GET 2022-09-09T22:48:44.834 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.835 DEBUG:boto:Data: 2022-09-09T22:48:44.836 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.836 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:44.837 DEBUG:boto:Port: 8001 2022-09-09T22:48:44.837 DEBUG:boto:Params: {} 2022-09-09T22:48:44.838 DEBUG:boto:Token: None 2022-09-09T22:48:44.838 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.839 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YixcSJxgf3p7LQDrCZucPATBEp8= 2022-09-09T22:48:44.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YixcSJxgf3p7LQDrCZucPATBEp8=', 'Content-Length': '0'} 2022-09-09T22:48:44.843 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004894fcfdb8882ad2-00631bc2cc-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.844 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:44.844 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:44.845 DEBUG:boto:path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.845 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.846 DEBUG:boto:Method: GET 2022-09-09T22:48:44.846 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.847 DEBUG:boto:Data: 2022-09-09T22:48:44.848 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.848 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:44.849 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.849 DEBUG:boto:Params: {} 2022-09-09T22:48:44.850 DEBUG:boto:Token: None 2022-09-09T22:48:44.850 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.851 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YixcSJxgf3p7LQDrCZucPATBEp8= 2022-09-09T22:48:44.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YixcSJxgf3p7LQDrCZucPATBEp8=', 'Content-Length': '0'} 2022-09-09T22:48:44.855 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001cfda513accb532b-00631bc2cc-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.855 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:44.856 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:44.856 DEBUG:boto:path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.857 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.857 DEBUG:boto:Method: HEAD 2022-09-09T22:48:44.858 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.858 DEBUG:boto:Data: 2022-09-09T22:48:44.859 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.859 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:44.860 DEBUG:boto:Port: 8001 2022-09-09T22:48:44.860 DEBUG:boto:Params: {} 2022-09-09T22:48:44.861 DEBUG:boto:Token: None 2022-09-09T22:48:44.861 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.862 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8ZVUAGRbplObJjqy1WNbHbPnRgQ= 2022-09-09T22:48:44.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8ZVUAGRbplObJjqy1WNbHbPnRgQ=', 'Content-Length': '0'} 2022-09-09T22:48:44.865 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bda5041d42d118be-00631bc2cc-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.866 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:44.867 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:44.867 DEBUG:boto:path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.868 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.868 DEBUG:boto:Method: HEAD 2022-09-09T22:48:44.869 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.869 DEBUG:boto:Data: 2022-09-09T22:48:44.870 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.870 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:44.871 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.871 DEBUG:boto:Params: {} 2022-09-09T22:48:44.872 DEBUG:boto:Token: None 2022-09-09T22:48:44.872 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.873 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8ZVUAGRbplObJjqy1WNbHbPnRgQ= 2022-09-09T22:48:44.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8ZVUAGRbplObJjqy1WNbHbPnRgQ=', 'Content-Length': '0'} 2022-09-09T22:48:44.876 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b20584f6c8a59831-00631bc2cc-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:48:44.878 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:44.878 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:44.879 DEBUG:boto:path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.879 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.880 DEBUG:boto:Method: GET 2022-09-09T22:48:44.880 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.881 DEBUG:boto:Data: 2022-09-09T22:48:44.881 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.882 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:44.882 DEBUG:boto:Port: 8001 2022-09-09T22:48:44.883 DEBUG:boto:Params: {} 2022-09-09T22:48:44.883 DEBUG:boto:Token: None 2022-09-09T22:48:44.884 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.884 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YixcSJxgf3p7LQDrCZucPATBEp8= 2022-09-09T22:48:44.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YixcSJxgf3p7LQDrCZucPATBEp8=', 'Content-Length': '0'} 2022-09-09T22:48:44.888 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a3ef3e6b501b8145-00631bc2cc-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.889 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:44.889 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:44.890 DEBUG:boto:path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.890 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.891 DEBUG:boto:Method: GET 2022-09-09T22:48:44.891 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.892 DEBUG:boto:Data: 2022-09-09T22:48:44.892 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.893 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:44.893 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.894 DEBUG:boto:Params: {} 2022-09-09T22:48:44.895 DEBUG:boto:Token: None 2022-09-09T22:48:44.895 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:44.896 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YixcSJxgf3p7LQDrCZucPATBEp8= 2022-09-09T22:48:44.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YixcSJxgf3p7LQDrCZucPATBEp8=', 'Content-Length': '0'} 2022-09-09T22:48:44.899 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f158812f8db46fec-00631bc2cc-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.900 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:44.900 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:44.901 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:44.901 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:44.902 DEBUG:boto:Method: GET 2022-09-09T22:48:44.903 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:44.903 DEBUG:boto:Data: 2022-09-09T22:48:44.904 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.904 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:44.905 DEBUG:boto:Port: 8001 2022-09-09T22:48:44.905 DEBUG:boto:Params: {} 2022-09-09T22:48:44.906 DEBUG:boto:Token: None 2022-09-09T22:48:44.906 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/ 2022-09-09T22:48:44.907 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sqHBwFPZu4J48MNEyd8CiHK7PYw= 2022-09-09T22:48:44.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sqHBwFPZu4J48MNEyd8CiHK7PYw=', 'Content-Length': '0'} 2022-09-09T22:48:44.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f193e7eb047214dc-00631bc2cc-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.915 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:44.915 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:44.916 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:44.916 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:44.917 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:44.917 DEBUG:boto:Method: GET 2022-09-09T22:48:44.918 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:44.919 DEBUG:boto:Data: 2022-09-09T22:48:44.919 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.920 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:44.920 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.921 DEBUG:boto:Params: {} 2022-09-09T22:48:44.921 DEBUG:boto:Token: None 2022-09-09T22:48:44.922 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/ 2022-09-09T22:48:44.922 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sqHBwFPZu4J48MNEyd8CiHK7PYw= 2022-09-09T22:48:44.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sqHBwFPZu4J48MNEyd8CiHK7PYw=', 'Content-Length': '0'} 2022-09-09T22:48:44.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b376d731b09d695-00631bc2cc-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.970 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:44.971 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:44.972 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:44.973 DEBUG:boto:Method: HEAD 2022-09-09T22:48:44.975 DEBUG:boto:Path: /jkcgbj-20/obj 2022-09-09T22:48:44.976 DEBUG:boto:Data: 2022-09-09T22:48:44.977 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.978 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:44.979 DEBUG:boto:Port: 8001 2022-09-09T22:48:44.980 DEBUG:boto:Params: {} 2022-09-09T22:48:44.981 DEBUG:boto:Token: None 2022-09-09T22:48:44.982 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/obj 2022-09-09T22:48:44.983 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:WJLp4PreGioScYb7cgIzna/avPM= 2022-09-09T22:48:44.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:WJLp4PreGioScYb7cgIzna/avPM=', 'Content-Length': '0'} 2022-09-09T22:48:44.988 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000794afcd7ef116318-00631bc2cc-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:44.989 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:44.989 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:44.990 DEBUG:boto:Method: HEAD 2022-09-09T22:48:44.990 DEBUG:boto:Path: /jkcgbj-20/obj 2022-09-09T22:48:44.991 DEBUG:boto:Data: 2022-09-09T22:48:44.992 DEBUG:boto:Headers: {} 2022-09-09T22:48:44.992 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:44.993 DEBUG:boto:Port: 8000 2022-09-09T22:48:44.994 DEBUG:boto:Params: {} 2022-09-09T22:48:44.995 DEBUG:boto:Token: None 2022-09-09T22:48:44.996 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:44 GMT /jkcgbj-20/obj 2022-09-09T22:48:44.997 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:WJLp4PreGioScYb7cgIzna/avPM= 2022-09-09T22:48:44.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:WJLp4PreGioScYb7cgIzna/avPM=', 'Content-Length': '0'} 2022-09-09T22:48:45.002 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000006c4e965b85ab06e-00631bc2cc-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:48:45.004 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.005 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:45.006 DEBUG:boto:path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.007 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.007 DEBUG:boto:Method: GET 2022-09-09T22:48:45.008 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.008 DEBUG:boto:Data: 2022-09-09T22:48:45.009 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.009 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:45.010 DEBUG:boto:Port: 8001 2022-09-09T22:48:45.011 DEBUG:boto:Params: {} 2022-09-09T22:48:45.012 DEBUG:boto:Token: None 2022-09-09T22:48:45.013 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.014 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jjvC7vRq+x9IjTsn4yCRXY2ScKQ= 2022-09-09T22:48:45.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jjvC7vRq+x9IjTsn4yCRXY2ScKQ=', 'Content-Length': '0'} 2022-09-09T22:48:45.017 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000081671d5c2bd6e0de-00631bc2cd-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.018 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.019 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:45.019 DEBUG:boto:path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.020 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.021 DEBUG:boto:Method: GET 2022-09-09T22:48:45.021 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.022 DEBUG:boto:Data: 2022-09-09T22:48:45.023 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.023 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:45.024 DEBUG:boto:Port: 8000 2022-09-09T22:48:45.025 DEBUG:boto:Params: {} 2022-09-09T22:48:45.025 DEBUG:boto:Token: None 2022-09-09T22:48:45.026 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.027 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jjvC7vRq+x9IjTsn4yCRXY2ScKQ= 2022-09-09T22:48:45.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jjvC7vRq+x9IjTsn4yCRXY2ScKQ=', 'Content-Length': '0'} 2022-09-09T22:48:45.030 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001ae2b008c59bdad4-00631bc2cd-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.031 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:45.032 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:45.032 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.033 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.033 DEBUG:boto:Method: GET 2022-09-09T22:48:45.034 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.034 DEBUG:boto:Data: 2022-09-09T22:48:45.038 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.039 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:45.040 DEBUG:boto:Port: 8001 2022-09-09T22:48:45.041 DEBUG:boto:Params: {} 2022-09-09T22:48:45.042 DEBUG:boto:Token: None 2022-09-09T22:48:45.044 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/ 2022-09-09T22:48:45.045 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8AWrIz//TRK6g10JcgdbeHwZHgU= 2022-09-09T22:48:45.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8AWrIz//TRK6g10JcgdbeHwZHgU=', 'Content-Length': '0'} 2022-09-09T22:48:45.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fbfaf8da43e7934-00631bc2cd-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.053 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:45.055 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:45.056 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:45.057 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.058 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.059 DEBUG:boto:Method: GET 2022-09-09T22:48:45.061 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.061 DEBUG:boto:Data: 2022-09-09T22:48:45.062 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.062 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:45.063 DEBUG:boto:Port: 8000 2022-09-09T22:48:45.063 DEBUG:boto:Params: {} 2022-09-09T22:48:45.064 DEBUG:boto:Token: None 2022-09-09T22:48:45.065 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/ 2022-09-09T22:48:45.065 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8AWrIz//TRK6g10JcgdbeHwZHgU= 2022-09-09T22:48:45.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8AWrIz//TRK6g10JcgdbeHwZHgU=', 'Content-Length': '0'} 2022-09-09T22:48:45.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000eaed3dd34f3a597-00631bc2cd-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.118 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:45.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:48:45.120 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.121 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:45.122 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.123 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.124 DEBUG:boto:Method: GET 2022-09-09T22:48:45.125 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.125 DEBUG:boto:Data: 2022-09-09T22:48:45.126 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.127 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:45.127 DEBUG:boto:Port: 8001 2022-09-09T22:48:45.128 DEBUG:boto:Params: {} 2022-09-09T22:48:45.128 DEBUG:boto:Token: None 2022-09-09T22:48:45.129 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.130 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Yh7/M4KdyCu35u4dcN/UU3lnhBQ= 2022-09-09T22:48:45.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Yh7/M4KdyCu35u4dcN/UU3lnhBQ=', 'Content-Length': '0'} 2022-09-09T22:48:45.134 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fa7991891ca1414e-00631bc2cd-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.135 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.135 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:45.136 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.136 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.137 DEBUG:boto:Method: GET 2022-09-09T22:48:45.138 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.139 DEBUG:boto:Data: 2022-09-09T22:48:45.140 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.140 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:45.142 DEBUG:boto:Port: 8000 2022-09-09T22:48:45.143 DEBUG:boto:Params: {} 2022-09-09T22:48:45.144 DEBUG:boto:Token: None 2022-09-09T22:48:45.145 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.146 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Yh7/M4KdyCu35u4dcN/UU3lnhBQ= 2022-09-09T22:48:45.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Yh7/M4KdyCu35u4dcN/UU3lnhBQ=', 'Content-Length': '0'} 2022-09-09T22:48:45.151 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c7114c93e5333b76-00631bc2cd-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.151 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.152 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:45.152 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.153 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.153 DEBUG:boto:Method: HEAD 2022-09-09T22:48:45.154 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.155 DEBUG:boto:Data: 2022-09-09T22:48:45.156 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.157 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:45.158 DEBUG:boto:Port: 8001 2022-09-09T22:48:45.159 DEBUG:boto:Params: {} 2022-09-09T22:48:45.160 DEBUG:boto:Token: None 2022-09-09T22:48:45.161 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.162 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YU30SimEFIuLW9Qn9Ahx/ONDK8I= 2022-09-09T22:48:45.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YU30SimEFIuLW9Qn9Ahx/ONDK8I=', 'Content-Length': '0'} 2022-09-09T22:48:45.166 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000049e6eae286f31f5f-00631bc2cd-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.168 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.169 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:45.170 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.171 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.171 DEBUG:boto:Method: HEAD 2022-09-09T22:48:45.172 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.173 DEBUG:boto:Data: 2022-09-09T22:48:45.174 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.175 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:45.176 DEBUG:boto:Port: 8000 2022-09-09T22:48:45.177 DEBUG:boto:Params: {} 2022-09-09T22:48:45.178 DEBUG:boto:Token: None 2022-09-09T22:48:45.179 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.180 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YU30SimEFIuLW9Qn9Ahx/ONDK8I= 2022-09-09T22:48:45.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YU30SimEFIuLW9Qn9Ahx/ONDK8I=', 'Content-Length': '0'} 2022-09-09T22:48:45.185 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000098a13ab91faa4e39-00631bc2cd-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:48:45.187 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.188 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:45.189 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.190 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.191 DEBUG:boto:Method: GET 2022-09-09T22:48:45.191 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.192 DEBUG:boto:Data: 2022-09-09T22:48:45.193 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.194 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:45.195 DEBUG:boto:Port: 8001 2022-09-09T22:48:45.196 DEBUG:boto:Params: {} 2022-09-09T22:48:45.197 DEBUG:boto:Token: None 2022-09-09T22:48:45.198 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.199 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Yh7/M4KdyCu35u4dcN/UU3lnhBQ= 2022-09-09T22:48:45.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Yh7/M4KdyCu35u4dcN/UU3lnhBQ=', 'Content-Length': '0'} 2022-09-09T22:48:45.204 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a90eff070ad9079f-00631bc2cd-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.206 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.207 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:45.207 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.208 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.209 DEBUG:boto:Method: GET 2022-09-09T22:48:45.210 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.211 DEBUG:boto:Data: 2022-09-09T22:48:45.212 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.213 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:45.214 DEBUG:boto:Port: 8000 2022-09-09T22:48:45.215 DEBUG:boto:Params: {} 2022-09-09T22:48:45.216 DEBUG:boto:Token: None 2022-09-09T22:48:45.217 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.219 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Yh7/M4KdyCu35u4dcN/UU3lnhBQ= 2022-09-09T22:48:45.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Yh7/M4KdyCu35u4dcN/UU3lnhBQ=', 'Content-Length': '0'} 2022-09-09T22:48:45.224 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000890eaa251f5075d2-00631bc2cd-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.225 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:45.226 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:45.227 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.228 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.228 DEBUG:boto:Method: GET 2022-09-09T22:48:45.229 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.230 DEBUG:boto:Data: 2022-09-09T22:48:45.231 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.231 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:45.232 DEBUG:boto:Port: 8001 2022-09-09T22:48:45.232 DEBUG:boto:Params: {} 2022-09-09T22:48:45.233 DEBUG:boto:Token: None 2022-09-09T22:48:45.233 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/ 2022-09-09T22:48:45.234 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8AWrIz//TRK6g10JcgdbeHwZHgU= 2022-09-09T22:48:45.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8AWrIz//TRK6g10JcgdbeHwZHgU=', 'Content-Length': '0'} 2022-09-09T22:48:45.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060201415338e37e9-00631bc2cd-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.282 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:45.283 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:45.284 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:45.285 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.285 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.286 DEBUG:boto:Method: GET 2022-09-09T22:48:45.287 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.287 DEBUG:boto:Data: 2022-09-09T22:48:45.288 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.288 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:45.289 DEBUG:boto:Port: 8000 2022-09-09T22:48:45.290 DEBUG:boto:Params: {} 2022-09-09T22:48:45.290 DEBUG:boto:Token: None 2022-09-09T22:48:45.291 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/ 2022-09-09T22:48:45.292 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8AWrIz//TRK6g10JcgdbeHwZHgU= 2022-09-09T22:48:45.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8AWrIz//TRK6g10JcgdbeHwZHgU=', 'Content-Length': '0'} 2022-09-09T22:48:45.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d180af9b69386f2-00631bc2cd-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.298 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:45.299 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.300 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:45.300 DEBUG:boto:Method: HEAD 2022-09-09T22:48:45.301 DEBUG:boto:Path: /jkcgbj-20/obj 2022-09-09T22:48:45.301 DEBUG:boto:Data: 2022-09-09T22:48:45.302 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.302 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:45.303 DEBUG:boto:Port: 8001 2022-09-09T22:48:45.303 DEBUG:boto:Params: {} 2022-09-09T22:48:45.304 DEBUG:boto:Token: None 2022-09-09T22:48:45.304 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/obj 2022-09-09T22:48:45.305 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:GzrFfzxbzr/waDn4gpFRLcBZtPk= 2022-09-09T22:48:45.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:GzrFfzxbzr/waDn4gpFRLcBZtPk=', 'Content-Length': '0'} 2022-09-09T22:48:45.310 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003c3137ed15d97431-00631bc2cd-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.310 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.311 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:45.312 DEBUG:boto:Method: HEAD 2022-09-09T22:48:45.312 DEBUG:boto:Path: /jkcgbj-20/obj 2022-09-09T22:48:45.313 DEBUG:boto:Data: 2022-09-09T22:48:45.313 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.314 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:45.314 DEBUG:boto:Port: 8000 2022-09-09T22:48:45.315 DEBUG:boto:Params: {} 2022-09-09T22:48:45.315 DEBUG:boto:Token: None 2022-09-09T22:48:45.316 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/obj 2022-09-09T22:48:45.316 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:GzrFfzxbzr/waDn4gpFRLcBZtPk= 2022-09-09T22:48:45.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:GzrFfzxbzr/waDn4gpFRLcBZtPk=', 'Content-Length': '0'} 2022-09-09T22:48:45.321 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ef6e80aa41cbf6de-00631bc2cd-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:48:45.322 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.323 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:45.323 DEBUG:boto:path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.324 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.324 DEBUG:boto:Method: GET 2022-09-09T22:48:45.325 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.325 DEBUG:boto:Data: 2022-09-09T22:48:45.326 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.326 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:45.327 DEBUG:boto:Port: 8001 2022-09-09T22:48:45.327 DEBUG:boto:Params: {} 2022-09-09T22:48:45.328 DEBUG:boto:Token: None 2022-09-09T22:48:45.328 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.329 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jjvC7vRq+x9IjTsn4yCRXY2ScKQ= 2022-09-09T22:48:45.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jjvC7vRq+x9IjTsn4yCRXY2ScKQ=', 'Content-Length': '0'} 2022-09-09T22:48:45.333 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003a44a866fa0621d6-00631bc2cd-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.333 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.334 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:45.335 DEBUG:boto:path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.335 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.336 DEBUG:boto:Method: GET 2022-09-09T22:48:45.336 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.337 DEBUG:boto:Data: 2022-09-09T22:48:45.337 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.338 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:45.338 DEBUG:boto:Port: 8000 2022-09-09T22:48:45.339 DEBUG:boto:Params: {} 2022-09-09T22:48:45.339 DEBUG:boto:Token: None 2022-09-09T22:48:45.340 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.340 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jjvC7vRq+x9IjTsn4yCRXY2ScKQ= 2022-09-09T22:48:45.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jjvC7vRq+x9IjTsn4yCRXY2ScKQ=', 'Content-Length': '0'} 2022-09-09T22:48:45.344 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b0b76328ecc3674e-00631bc2cd-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.344 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:45.345 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:45.346 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.346 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.347 DEBUG:boto:Method: GET 2022-09-09T22:48:45.347 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.348 DEBUG:boto:Data: 2022-09-09T22:48:45.348 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.349 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:45.349 DEBUG:boto:Port: 8001 2022-09-09T22:48:45.350 DEBUG:boto:Params: {} 2022-09-09T22:48:45.350 DEBUG:boto:Token: None 2022-09-09T22:48:45.351 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/ 2022-09-09T22:48:45.352 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8AWrIz//TRK6g10JcgdbeHwZHgU= 2022-09-09T22:48:45.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8AWrIz//TRK6g10JcgdbeHwZHgU=', 'Content-Length': '0'} 2022-09-09T22:48:45.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ff23ec452266d5f-00631bc2cd-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.360 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:45.361 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:45.362 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:45.363 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.364 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.365 DEBUG:boto:Method: GET 2022-09-09T22:48:45.366 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.367 DEBUG:boto:Data: 2022-09-09T22:48:45.368 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.369 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:45.370 DEBUG:boto:Port: 8000 2022-09-09T22:48:45.371 DEBUG:boto:Params: {} 2022-09-09T22:48:45.372 DEBUG:boto:Token: None 2022-09-09T22:48:45.373 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/ 2022-09-09T22:48:45.375 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8AWrIz//TRK6g10JcgdbeHwZHgU= 2022-09-09T22:48:45.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8AWrIz//TRK6g10JcgdbeHwZHgU=', 'Content-Length': '0'} 2022-09-09T22:48:45.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb4f6014e31fe61a-00631bc2cd-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.422 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:45.423 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-20 zones={test-zone3, test-zone1} 2022-09-09T22:48:45.424 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-20 zones={test-zone3, test-zone2} 2022-09-09T22:48:45.424 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:45.425 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:45.425 DEBUG:boto:Method: HEAD 2022-09-09T22:48:45.426 DEBUG:boto:Path: /jkcgbj-20/ 2022-09-09T22:48:45.426 DEBUG:boto:Data: 2022-09-09T22:48:45.427 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.428 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:45.428 DEBUG:boto:Port: 8001 2022-09-09T22:48:45.429 DEBUG:boto:Params: {} 2022-09-09T22:48:45.430 DEBUG:boto:Token: None 2022-09-09T22:48:45.430 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/ 2022-09-09T22:48:45.431 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:HjoR36X3Bd02I43axQKaX11vsFQ= 2022-09-09T22:48:45.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:HjoR36X3Bd02I43axQKaX11vsFQ=', 'Content-Length': '0'} 2022-09-09T22:48:45.437 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c17d51870f0d9b3-00631bc2cd-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.439 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:45.439 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:45.440 DEBUG:boto:Method: HEAD 2022-09-09T22:48:45.440 DEBUG:boto:Path: /jkcgbj-20/ 2022-09-09T22:48:45.441 DEBUG:boto:Data: 2022-09-09T22:48:45.441 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.442 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:45.443 DEBUG:boto:Port: 8000 2022-09-09T22:48:45.443 DEBUG:boto:Params: {} 2022-09-09T22:48:45.444 DEBUG:boto:Token: None 2022-09-09T22:48:45.445 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/ 2022-09-09T22:48:45.447 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:HjoR36X3Bd02I43axQKaX11vsFQ= 2022-09-09T22:48:45.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:HjoR36X3Bd02I43axQKaX11vsFQ=', 'Content-Length': '0'} 2022-09-09T22:48:45.453 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d56e6d55b3e20846-00631bc2cd-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.454 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:48:45.455 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:45.456 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:45.457 DEBUG:boto:path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:45.458 DEBUG:boto:auth_path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:45.459 DEBUG:boto:Method: GET 2022-09-09T22:48:45.460 DEBUG:boto:Path: /jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:45.461 DEBUG:boto:Data: 2022-09-09T22:48:45.461 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.462 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:45.463 DEBUG:boto:Port: 8001 2022-09-09T22:48:45.464 DEBUG:boto:Params: {} 2022-09-09T22:48:45.466 DEBUG:boto:Token: None 2022-09-09T22:48:45.467 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/?versions 2022-09-09T22:48:45.468 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:l7hMDUbcra+S+Dd+TkcHj5chuQ8= 2022-09-09T22:48:45.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:l7hMDUbcra+S+Dd+TkcHj5chuQ8=', 'Content-Length': '0'} 2022-09-09T22:48:45.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008be43f6e3c75504-00631bc2cd-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.476 DEBUG:boto:b'jkcgbj-20999falseobji9PSHBUuQ8xfOhtiVzZhEqzCuln.uXjtrue2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:45.477 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:48:45.478 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:48:45.479 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:48:45.480 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:45.481 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:45.482 DEBUG:boto:path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:45.483 DEBUG:boto:auth_path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:45.484 DEBUG:boto:Method: GET 2022-09-09T22:48:45.485 DEBUG:boto:Path: /jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:45.486 DEBUG:boto:Data: 2022-09-09T22:48:45.487 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.488 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:45.489 DEBUG:boto:Port: 8000 2022-09-09T22:48:45.490 DEBUG:boto:Params: {} 2022-09-09T22:48:45.491 DEBUG:boto:Token: None 2022-09-09T22:48:45.491 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/?versions 2022-09-09T22:48:45.492 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:l7hMDUbcra+S+Dd+TkcHj5chuQ8= 2022-09-09T22:48:45.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:l7hMDUbcra+S+Dd+TkcHj5chuQ8=', 'Content-Length': '0'} 2022-09-09T22:48:45.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000814e540b75a83c5a-00631bc2cd-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.498 DEBUG:boto:b'jkcgbj-20999falseobji9PSHBUuQ8xfOhtiVzZhEqzCuln.uXjtrue2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:45.499 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:48:45.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:48:45.500 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:45.501 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:45.501 DEBUG:boto:path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:45.502 DEBUG:boto:auth_path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:45.502 DEBUG:boto:Method: GET 2022-09-09T22:48:45.503 DEBUG:boto:Path: /jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:45.503 DEBUG:boto:Data: 2022-09-09T22:48:45.504 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.504 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:45.505 DEBUG:boto:Port: 8001 2022-09-09T22:48:45.505 DEBUG:boto:Params: {} 2022-09-09T22:48:45.506 DEBUG:boto:Token: None 2022-09-09T22:48:45.506 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/?versions 2022-09-09T22:48:45.507 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:l7hMDUbcra+S+Dd+TkcHj5chuQ8= 2022-09-09T22:48:45.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:l7hMDUbcra+S+Dd+TkcHj5chuQ8=', 'Content-Length': '0'} 2022-09-09T22:48:45.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8858fb2af26ed34-00631bc2cd-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.554 DEBUG:boto:b'jkcgbj-20999falseobji9PSHBUuQ8xfOhtiVzZhEqzCuln.uXjtrue2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:45.555 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:45.556 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:45.556 DEBUG:boto:path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:45.557 DEBUG:boto:auth_path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:45.557 DEBUG:boto:Method: GET 2022-09-09T22:48:45.558 DEBUG:boto:Path: /jkcgbj-20/?versions&max-keys=999 2022-09-09T22:48:45.558 DEBUG:boto:Data: 2022-09-09T22:48:45.559 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.559 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:45.560 DEBUG:boto:Port: 8000 2022-09-09T22:48:45.560 DEBUG:boto:Params: {} 2022-09-09T22:48:45.561 DEBUG:boto:Token: None 2022-09-09T22:48:45.561 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/?versions 2022-09-09T22:48:45.563 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:l7hMDUbcra+S+Dd+TkcHj5chuQ8= 2022-09-09T22:48:45.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:l7hMDUbcra+S+Dd+TkcHj5chuQ8=', 'Content-Length': '0'} 2022-09-09T22:48:45.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f7cac1e461dadda-00631bc2cd-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.570 DEBUG:boto:b'jkcgbj-20999falseobji9PSHBUuQ8xfOhtiVzZhEqzCuln.uXjtrue2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-09T22:47:04.719Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:45.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:48:45.572 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.572 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:45.573 DEBUG:boto:path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.573 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.574 DEBUG:boto:Method: GET 2022-09-09T22:48:45.575 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.575 DEBUG:boto:Data: 2022-09-09T22:48:45.576 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.576 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:45.577 DEBUG:boto:Port: 8001 2022-09-09T22:48:45.577 DEBUG:boto:Params: {} 2022-09-09T22:48:45.578 DEBUG:boto:Token: None 2022-09-09T22:48:45.579 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.579 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jjvC7vRq+x9IjTsn4yCRXY2ScKQ= 2022-09-09T22:48:45.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jjvC7vRq+x9IjTsn4yCRXY2ScKQ=', 'Content-Length': '0'} 2022-09-09T22:48:45.583 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000047908254a27839ed-00631bc2cd-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.584 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.584 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:45.585 DEBUG:boto:path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.586 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.586 DEBUG:boto:Method: GET 2022-09-09T22:48:45.587 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.587 DEBUG:boto:Data: 2022-09-09T22:48:45.588 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.588 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:45.589 DEBUG:boto:Port: 8000 2022-09-09T22:48:45.589 DEBUG:boto:Params: {} 2022-09-09T22:48:45.590 DEBUG:boto:Token: None 2022-09-09T22:48:45.591 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.591 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jjvC7vRq+x9IjTsn4yCRXY2ScKQ= 2022-09-09T22:48:45.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jjvC7vRq+x9IjTsn4yCRXY2ScKQ=', 'Content-Length': '0'} 2022-09-09T22:48:45.595 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000421a932ab4fe942f-00631bc2cd-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.596 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.596 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:45.597 DEBUG:boto:path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.597 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.598 DEBUG:boto:Method: HEAD 2022-09-09T22:48:45.599 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.599 DEBUG:boto:Data: 2022-09-09T22:48:45.600 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.600 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:45.601 DEBUG:boto:Port: 8001 2022-09-09T22:48:45.601 DEBUG:boto:Params: {} 2022-09-09T22:48:45.602 DEBUG:boto:Token: None 2022-09-09T22:48:45.602 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.603 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:/Su5fIETW/ieBpxWVATG0+sMUhA= 2022-09-09T22:48:45.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:/Su5fIETW/ieBpxWVATG0+sMUhA=', 'Content-Length': '0'} 2022-09-09T22:48:45.606 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ef67cb36d101aa9a-00631bc2cd-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.607 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.608 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:45.608 DEBUG:boto:path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.609 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.609 DEBUG:boto:Method: HEAD 2022-09-09T22:48:45.610 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.610 DEBUG:boto:Data: 2022-09-09T22:48:45.611 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.611 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:45.612 DEBUG:boto:Port: 8000 2022-09-09T22:48:45.612 DEBUG:boto:Params: {} 2022-09-09T22:48:45.613 DEBUG:boto:Token: None 2022-09-09T22:48:45.613 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.614 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:/Su5fIETW/ieBpxWVATG0+sMUhA= 2022-09-09T22:48:45.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:/Su5fIETW/ieBpxWVATG0+sMUhA=', 'Content-Length': '0'} 2022-09-09T22:48:45.618 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000056791b90f8211ab-00631bc2cd-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:48:45.620 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.620 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:45.621 DEBUG:boto:path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.621 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.622 DEBUG:boto:Method: GET 2022-09-09T22:48:45.622 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.623 DEBUG:boto:Data: 2022-09-09T22:48:45.623 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.624 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:45.624 DEBUG:boto:Port: 8001 2022-09-09T22:48:45.625 DEBUG:boto:Params: {} 2022-09-09T22:48:45.626 DEBUG:boto:Token: None 2022-09-09T22:48:45.626 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.627 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jjvC7vRq+x9IjTsn4yCRXY2ScKQ= 2022-09-09T22:48:45.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jjvC7vRq+x9IjTsn4yCRXY2ScKQ=', 'Content-Length': '0'} 2022-09-09T22:48:45.630 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009904ac0e1f679031-00631bc2cd-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.631 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.632 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:45.632 DEBUG:boto:path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.633 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.633 DEBUG:boto:Method: GET 2022-09-09T22:48:45.634 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.634 DEBUG:boto:Data: 2022-09-09T22:48:45.635 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.636 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:45.637 DEBUG:boto:Port: 8000 2022-09-09T22:48:45.638 DEBUG:boto:Params: {} 2022-09-09T22:48:45.639 DEBUG:boto:Token: None 2022-09-09T22:48:45.639 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.640 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jjvC7vRq+x9IjTsn4yCRXY2ScKQ= 2022-09-09T22:48:45.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jjvC7vRq+x9IjTsn4yCRXY2ScKQ=', 'Content-Length': '0'} 2022-09-09T22:48:45.644 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009fdec42234c4c900-00631bc2cd-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.645 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:45.646 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:45.647 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.648 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.649 DEBUG:boto:Method: GET 2022-09-09T22:48:45.650 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.651 DEBUG:boto:Data: 2022-09-09T22:48:45.651 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.652 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:45.652 DEBUG:boto:Port: 8001 2022-09-09T22:48:45.653 DEBUG:boto:Params: {} 2022-09-09T22:48:45.654 DEBUG:boto:Token: None 2022-09-09T22:48:45.654 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/ 2022-09-09T22:48:45.655 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8AWrIz//TRK6g10JcgdbeHwZHgU= 2022-09-09T22:48:45.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8AWrIz//TRK6g10JcgdbeHwZHgU=', 'Content-Length': '0'} 2022-09-09T22:48:45.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcac41772f4de1c3-00631bc2cd-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.702 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:45.703 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:45.704 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:45.704 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.705 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.705 DEBUG:boto:Method: GET 2022-09-09T22:48:45.706 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.706 DEBUG:boto:Data: 2022-09-09T22:48:45.707 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.707 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:45.708 DEBUG:boto:Port: 8000 2022-09-09T22:48:45.709 DEBUG:boto:Params: {} 2022-09-09T22:48:45.710 DEBUG:boto:Token: None 2022-09-09T22:48:45.711 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/ 2022-09-09T22:48:45.711 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8AWrIz//TRK6g10JcgdbeHwZHgU= 2022-09-09T22:48:45.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8AWrIz//TRK6g10JcgdbeHwZHgU=', 'Content-Length': '0'} 2022-09-09T22:48:45.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbda6bfd1269c440-00631bc2cd-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.718 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:45.719 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.720 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:45.720 DEBUG:boto:Method: HEAD 2022-09-09T22:48:45.721 DEBUG:boto:Path: /jkcgbj-20/obj 2022-09-09T22:48:45.721 DEBUG:boto:Data: 2022-09-09T22:48:45.722 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.723 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:45.723 DEBUG:boto:Port: 8001 2022-09-09T22:48:45.724 DEBUG:boto:Params: {} 2022-09-09T22:48:45.724 DEBUG:boto:Token: None 2022-09-09T22:48:45.725 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/obj 2022-09-09T22:48:45.725 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:GzrFfzxbzr/waDn4gpFRLcBZtPk= 2022-09-09T22:48:45.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:GzrFfzxbzr/waDn4gpFRLcBZtPk=', 'Content-Length': '0'} 2022-09-09T22:48:45.730 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cf2c66c01b6d587e-00631bc2cd-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.731 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.732 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:45.733 DEBUG:boto:Method: HEAD 2022-09-09T22:48:45.733 DEBUG:boto:Path: /jkcgbj-20/obj 2022-09-09T22:48:45.734 DEBUG:boto:Data: 2022-09-09T22:48:45.734 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.735 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:45.735 DEBUG:boto:Port: 8000 2022-09-09T22:48:45.736 DEBUG:boto:Params: {} 2022-09-09T22:48:45.736 DEBUG:boto:Token: None 2022-09-09T22:48:45.737 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/obj 2022-09-09T22:48:45.738 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:GzrFfzxbzr/waDn4gpFRLcBZtPk= 2022-09-09T22:48:45.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:GzrFfzxbzr/waDn4gpFRLcBZtPk=', 'Content-Length': '0'} 2022-09-09T22:48:45.742 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000123c7b16b878ae01-00631bc2cd-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:48:45.744 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.744 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:45.745 DEBUG:boto:path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.745 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.746 DEBUG:boto:Method: GET 2022-09-09T22:48:45.746 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.747 DEBUG:boto:Data: 2022-09-09T22:48:45.747 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.748 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:45.748 DEBUG:boto:Port: 8001 2022-09-09T22:48:45.749 DEBUG:boto:Params: {} 2022-09-09T22:48:45.749 DEBUG:boto:Token: None 2022-09-09T22:48:45.750 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.751 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jjvC7vRq+x9IjTsn4yCRXY2ScKQ= 2022-09-09T22:48:45.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jjvC7vRq+x9IjTsn4yCRXY2ScKQ=', 'Content-Length': '0'} 2022-09-09T22:48:45.753 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007893109d9c0529f2-00631bc2cd-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.754 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.755 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:45.755 DEBUG:boto:path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.756 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.757 DEBUG:boto:Method: GET 2022-09-09T22:48:45.757 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.758 DEBUG:boto:Data: 2022-09-09T22:48:45.759 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.759 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:45.760 DEBUG:boto:Port: 8000 2022-09-09T22:48:45.760 DEBUG:boto:Params: {} 2022-09-09T22:48:45.761 DEBUG:boto:Token: None 2022-09-09T22:48:45.763 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.764 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jjvC7vRq+x9IjTsn4yCRXY2ScKQ= 2022-09-09T22:48:45.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jjvC7vRq+x9IjTsn4yCRXY2ScKQ=', 'Content-Length': '0'} 2022-09-09T22:48:45.769 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000060b205588c338082-00631bc2cd-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.771 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:45.772 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:45.773 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.774 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.775 DEBUG:boto:Method: GET 2022-09-09T22:48:45.775 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.776 DEBUG:boto:Data: 2022-09-09T22:48:45.776 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.777 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:45.777 DEBUG:boto:Port: 8001 2022-09-09T22:48:45.778 DEBUG:boto:Params: {} 2022-09-09T22:48:45.779 DEBUG:boto:Token: None 2022-09-09T22:48:45.779 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/ 2022-09-09T22:48:45.780 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8AWrIz//TRK6g10JcgdbeHwZHgU= 2022-09-09T22:48:45.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8AWrIz//TRK6g10JcgdbeHwZHgU=', 'Content-Length': '0'} 2022-09-09T22:48:45.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1edf8ef152ef55f-00631bc2cd-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.787 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:45.787 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:45.788 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:45.789 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.789 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.790 DEBUG:boto:Method: GET 2022-09-09T22:48:45.790 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.791 DEBUG:boto:Data: 2022-09-09T22:48:45.791 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.792 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:45.792 DEBUG:boto:Port: 8000 2022-09-09T22:48:45.793 DEBUG:boto:Params: {} 2022-09-09T22:48:45.794 DEBUG:boto:Token: None 2022-09-09T22:48:45.794 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/ 2022-09-09T22:48:45.795 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8AWrIz//TRK6g10JcgdbeHwZHgU= 2022-09-09T22:48:45.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8AWrIz//TRK6g10JcgdbeHwZHgU=', 'Content-Length': '0'} 2022-09-09T22:48:45.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4ecbd24f13c690f-00631bc2cd-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.801 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:45.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:48:45.803 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.803 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:45.804 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.804 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.805 DEBUG:boto:Method: GET 2022-09-09T22:48:45.805 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.806 DEBUG:boto:Data: 2022-09-09T22:48:45.806 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.807 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:45.808 DEBUG:boto:Port: 8001 2022-09-09T22:48:45.808 DEBUG:boto:Params: {} 2022-09-09T22:48:45.809 DEBUG:boto:Token: None 2022-09-09T22:48:45.809 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.810 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Yh7/M4KdyCu35u4dcN/UU3lnhBQ= 2022-09-09T22:48:45.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Yh7/M4KdyCu35u4dcN/UU3lnhBQ=', 'Content-Length': '0'} 2022-09-09T22:48:45.814 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000023ad0c00dd2a6188-00631bc2cd-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.814 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.815 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:45.815 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.816 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.816 DEBUG:boto:Method: GET 2022-09-09T22:48:45.817 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.818 DEBUG:boto:Data: 2022-09-09T22:48:45.818 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.819 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:45.819 DEBUG:boto:Port: 8000 2022-09-09T22:48:45.820 DEBUG:boto:Params: {} 2022-09-09T22:48:45.820 DEBUG:boto:Token: None 2022-09-09T22:48:45.821 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.822 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Yh7/M4KdyCu35u4dcN/UU3lnhBQ= 2022-09-09T22:48:45.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Yh7/M4KdyCu35u4dcN/UU3lnhBQ=', 'Content-Length': '0'} 2022-09-09T22:48:45.825 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c6eb1fa7d12d3971-00631bc2cd-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.826 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.827 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:45.827 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.828 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.828 DEBUG:boto:Method: HEAD 2022-09-09T22:48:45.829 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.829 DEBUG:boto:Data: 2022-09-09T22:48:45.830 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.830 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:45.831 DEBUG:boto:Port: 8001 2022-09-09T22:48:45.831 DEBUG:boto:Params: {} 2022-09-09T22:48:45.832 DEBUG:boto:Token: None 2022-09-09T22:48:45.832 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.833 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YU30SimEFIuLW9Qn9Ahx/ONDK8I= 2022-09-09T22:48:45.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YU30SimEFIuLW9Qn9Ahx/ONDK8I=', 'Content-Length': '0'} 2022-09-09T22:48:45.837 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d7795e8564771099-00631bc2cd-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.837 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.838 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:45.839 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.839 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.840 DEBUG:boto:Method: HEAD 2022-09-09T22:48:45.840 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.841 DEBUG:boto:Data: 2022-09-09T22:48:45.842 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.842 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:45.843 DEBUG:boto:Port: 8000 2022-09-09T22:48:45.843 DEBUG:boto:Params: {} 2022-09-09T22:48:45.844 DEBUG:boto:Token: None 2022-09-09T22:48:45.844 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.845 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YU30SimEFIuLW9Qn9Ahx/ONDK8I= 2022-09-09T22:48:45.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YU30SimEFIuLW9Qn9Ahx/ONDK8I=', 'Content-Length': '0'} 2022-09-09T22:48:45.849 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000016b763a469005400-00631bc2cd-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:48:45.850 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.850 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:45.851 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.852 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.852 DEBUG:boto:Method: GET 2022-09-09T22:48:45.853 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.853 DEBUG:boto:Data: 2022-09-09T22:48:45.854 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.854 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:45.855 DEBUG:boto:Port: 8001 2022-09-09T22:48:45.855 DEBUG:boto:Params: {} 2022-09-09T22:48:45.856 DEBUG:boto:Token: None 2022-09-09T22:48:45.856 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.857 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Yh7/M4KdyCu35u4dcN/UU3lnhBQ= 2022-09-09T22:48:45.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Yh7/M4KdyCu35u4dcN/UU3lnhBQ=', 'Content-Length': '0'} 2022-09-09T22:48:45.862 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000089876f700f2edefc-00631bc2cd-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.863 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.863 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:45.864 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.864 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.865 DEBUG:boto:Method: GET 2022-09-09T22:48:45.865 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.866 DEBUG:boto:Data: 2022-09-09T22:48:45.866 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.867 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:45.867 DEBUG:boto:Port: 8000 2022-09-09T22:48:45.868 DEBUG:boto:Params: {} 2022-09-09T22:48:45.869 DEBUG:boto:Token: None 2022-09-09T22:48:45.869 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:48:45.870 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Yh7/M4KdyCu35u4dcN/UU3lnhBQ= 2022-09-09T22:48:45.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Yh7/M4KdyCu35u4dcN/UU3lnhBQ=', 'Content-Length': '0'} 2022-09-09T22:48:45.874 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fb60a8a5a04185f8-00631bc2cd-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.874 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:45.875 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:45.875 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.876 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.877 DEBUG:boto:Method: GET 2022-09-09T22:48:45.877 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.878 DEBUG:boto:Data: 2022-09-09T22:48:45.878 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.879 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:45.880 DEBUG:boto:Port: 8001 2022-09-09T22:48:45.880 DEBUG:boto:Params: {} 2022-09-09T22:48:45.881 DEBUG:boto:Token: None 2022-09-09T22:48:45.881 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/ 2022-09-09T22:48:45.882 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8AWrIz//TRK6g10JcgdbeHwZHgU= 2022-09-09T22:48:45.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8AWrIz//TRK6g10JcgdbeHwZHgU=', 'Content-Length': '0'} 2022-09-09T22:48:45.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092dff39e0e68a7de-00631bc2cd-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.930 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:45.931 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:45.932 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:45.932 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.933 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.933 DEBUG:boto:Method: GET 2022-09-09T22:48:45.934 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:45.934 DEBUG:boto:Data: 2022-09-09T22:48:45.935 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.935 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:45.936 DEBUG:boto:Port: 8000 2022-09-09T22:48:45.936 DEBUG:boto:Params: {} 2022-09-09T22:48:45.938 DEBUG:boto:Token: None 2022-09-09T22:48:45.939 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/ 2022-09-09T22:48:45.939 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8AWrIz//TRK6g10JcgdbeHwZHgU= 2022-09-09T22:48:45.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8AWrIz//TRK6g10JcgdbeHwZHgU=', 'Content-Length': '0'} 2022-09-09T22:48:45.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000852f10d9953c7f86-00631bc2cd-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.946 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:45.947 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.948 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:45.948 DEBUG:boto:Method: HEAD 2022-09-09T22:48:45.949 DEBUG:boto:Path: /jkcgbj-20/obj 2022-09-09T22:48:45.949 DEBUG:boto:Data: 2022-09-09T22:48:45.950 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.951 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:45.952 DEBUG:boto:Port: 8001 2022-09-09T22:48:45.954 DEBUG:boto:Params: {} 2022-09-09T22:48:45.955 DEBUG:boto:Token: None 2022-09-09T22:48:45.955 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/obj 2022-09-09T22:48:45.956 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:GzrFfzxbzr/waDn4gpFRLcBZtPk= 2022-09-09T22:48:45.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:GzrFfzxbzr/waDn4gpFRLcBZtPk=', 'Content-Length': '0'} 2022-09-09T22:48:45.960 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000def2dc9039152872-00631bc2cd-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.961 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.962 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:45.963 DEBUG:boto:Method: HEAD 2022-09-09T22:48:45.963 DEBUG:boto:Path: /jkcgbj-20/obj 2022-09-09T22:48:45.964 DEBUG:boto:Data: 2022-09-09T22:48:45.964 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.965 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:45.966 DEBUG:boto:Port: 8000 2022-09-09T22:48:45.967 DEBUG:boto:Params: {} 2022-09-09T22:48:45.968 DEBUG:boto:Token: None 2022-09-09T22:48:45.969 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/obj 2022-09-09T22:48:45.971 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:GzrFfzxbzr/waDn4gpFRLcBZtPk= 2022-09-09T22:48:45.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:GzrFfzxbzr/waDn4gpFRLcBZtPk=', 'Content-Length': '0'} 2022-09-09T22:48:45.976 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005641ff4134b3ac82-00631bc2cd-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:48:45.978 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.979 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:45.980 DEBUG:boto:path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.981 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.982 DEBUG:boto:Method: GET 2022-09-09T22:48:45.983 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.984 DEBUG:boto:Data: 2022-09-09T22:48:45.985 DEBUG:boto:Headers: {} 2022-09-09T22:48:45.986 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:45.987 DEBUG:boto:Port: 8001 2022-09-09T22:48:45.988 DEBUG:boto:Params: {} 2022-09-09T22:48:45.989 DEBUG:boto:Token: None 2022-09-09T22:48:45.990 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:45 GMT /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:45.991 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jjvC7vRq+x9IjTsn4yCRXY2ScKQ= 2022-09-09T22:48:45.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jjvC7vRq+x9IjTsn4yCRXY2ScKQ=', 'Content-Length': '0'} 2022-09-09T22:48:45.996 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000899042c495e032f2-00631bc2cd-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:45.998 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:45.999 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:46.000 DEBUG:boto:path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:46.001 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:46.002 DEBUG:boto:Method: GET 2022-09-09T22:48:46.003 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:46.004 DEBUG:boto:Data: 2022-09-09T22:48:46.004 DEBUG:boto:Headers: {} 2022-09-09T22:48:46.005 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:46.006 DEBUG:boto:Port: 8000 2022-09-09T22:48:46.007 DEBUG:boto:Params: {} 2022-09-09T22:48:46.009 DEBUG:boto:Token: None 2022-09-09T22:48:46.010 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:46 GMT /jkcgbj-20/obj?versionId=i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj 2022-09-09T22:48:46.011 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JdNbOE9ph+yKsvBl/lo9WgFrvLE= 2022-09-09T22:48:46.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:46 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JdNbOE9ph+yKsvBl/lo9WgFrvLE=', 'Content-Length': '0'} 2022-09-09T22:48:46.016 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:17 GMT'), ('x-amz-version-id', 'i9PSHBUuQ8xfOhtiVzZhEqzCuln.uXj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000eff52650cf7c4eaf-00631bc2ce-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:48:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:46.018 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:46.018 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:46.019 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:46.020 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:46.020 DEBUG:boto:Method: GET 2022-09-09T22:48:46.021 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:46.022 DEBUG:boto:Data: 2022-09-09T22:48:46.022 DEBUG:boto:Headers: {} 2022-09-09T22:48:46.023 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:48:46.023 DEBUG:boto:Port: 8001 2022-09-09T22:48:46.024 DEBUG:boto:Params: {} 2022-09-09T22:48:46.024 DEBUG:boto:Token: None 2022-09-09T22:48:46.025 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:46 GMT /jkcgbj-20/ 2022-09-09T22:48:46.025 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:s19ekRIa0Ud7c/mw+7G43yTpi0E= 2022-09-09T22:48:46.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:46 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:s19ekRIa0Ud7c/mw+7G43yTpi0E=', 'Content-Length': '0'} 2022-09-09T22:48:46.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007003ef43e494b0cc-00631bc2ce-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:46.074 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:46.075 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:46.076 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:46.076 DEBUG:boto:path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:46.077 DEBUG:boto:auth_path=/jkcgbj-20/?prefix=obj 2022-09-09T22:48:46.077 DEBUG:boto:Method: GET 2022-09-09T22:48:46.078 DEBUG:boto:Path: /jkcgbj-20/?prefix=obj 2022-09-09T22:48:46.078 DEBUG:boto:Data: 2022-09-09T22:48:46.079 DEBUG:boto:Headers: {} 2022-09-09T22:48:46.079 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:48:46.080 DEBUG:boto:Port: 8000 2022-09-09T22:48:46.080 DEBUG:boto:Params: {} 2022-09-09T22:48:46.081 DEBUG:boto:Token: None 2022-09-09T22:48:46.082 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:48:46 GMT /jkcgbj-20/ 2022-09-09T22:48:46.083 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:s19ekRIa0Ud7c/mw+7G43yTpi0E= 2022-09-09T22:48:46.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:46 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:s19ekRIa0Ud7c/mw+7G43yTpi0E=', 'Content-Length': '0'} 2022-09-09T22:48:46.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024ee2c600d8e9ced-00631bc2ce-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:48:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:46.091 DEBUG:boto:b'jkcgbj-20obj1000falseobj2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:48:46.092 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-20 zones={test-zone3, test-zone2} 2022-09-09T22:48:46.092 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:46.093 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:46.093 DEBUG:boto:path=/jkcgbj-20/?versioning 2022-09-09T22:48:46.094 DEBUG:boto:auth_path=/jkcgbj-20/?versioning 2022-09-09T22:48:46.094 DEBUG:boto:Method: PUT 2022-09-09T22:48:46.095 DEBUG:boto:Path: /jkcgbj-20/?versioning 2022-09-09T22:48:46.095 DEBUG:boto:Data: Suspended Disabled 2022-09-09T22:48:46.096 DEBUG:boto:Headers: {} 2022-09-09T22:48:46.097 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:46.097 DEBUG:boto:Port: 8000 2022-09-09T22:48:46.098 DEBUG:boto:Params: {} 2022-09-09T22:48:46.098 DEBUG:boto:Token: None 2022-09-09T22:48:46.099 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:48:46 GMT /jkcgbj-20/?versioning 2022-09-09T22:48:46.099 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:NcdON1atk8tAy+qBlBylue7bsH4= 2022-09-09T22:48:46.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:46 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:NcdON1atk8tAy+qBlBylue7bsH4=', 'Content-Length': '230'} 2022-09-09T22:48:46.113 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002cf23caff3144c46-00631bc2ce-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:48:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:46.114 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:48:46.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:48:46.116 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:48:46.164 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:48:46.164 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:48:46.218 INFO:teuthology.orchestra.run.smithi005.stderr:No --period given, using current period=503b1d61-4513-48c8-a308-17b76219610c 2022-09-09T22:48:46.237 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T22:48:46.238 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:48:46.238 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:48:46.238 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:28:06.146147Z" 2022-09-09T22:48:46.238 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:48:46.239 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:48:46.239 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:48:46.239 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:48:46.240 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:48:46.240 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:48:46.240 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:48:46.240 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:30:37.757014Z" 2022-09-09T22:48:46.241 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:48:46.241 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:48:46.241 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662763726.109956_113.1", 2022-09-09T22:48:46.241 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:48:46.109956Z" 2022-09-09T22:48:46.242 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:48:46.242 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T22:48:46.243 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763726.109956_113.1'} 2022-09-09T22:48:46.244 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-09T22:48:46.244 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:48:46.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:48:46.245 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:48:46.292 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:48:46.292 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:48:46.351 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:48:46.345+0000 7fcaa1420dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:48:46.353 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:48:46.349+0000 7fcaa1420dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:48:46.384 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:48:46.384 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:48:46.385 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:48:46.385 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:48:46.385 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:48:46.385 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:48:46.386 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:48:46.386 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:48:46.386 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:48:46.386 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:48:46.387 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:48:46.387 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:48:46.387 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:48:46.387 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:48:46.388 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:48:46.388 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:48:46.388 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:48:46.388 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:48:46.389 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:48:46.389 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:48:46.389 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:48:46.390 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:48:46.390 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:48:46.391 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:48:46.391 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:48:46.391 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:48:46.391 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:48:46.392 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:48:46.392 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:48:46.392 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:48:46.393 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:48:46.393 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:48:46.393 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:48:46.393 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:48:46.394 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:48:46.394 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:48:46.394 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:48:46.394 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:48:46.395 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:48:46.395 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:48:46.395 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:48:46.395 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:48:46.396 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:48:46.396 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:48:46.396 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:48:46.396 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:48:46.397 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:48:46.397 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:48:46.397 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:48:46.397 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763726.109956_113.1", 2022-09-09T22:48:46.397 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:48:46.398 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:48:46.398 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:48:46.398 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:48:46.109956Z", 2022-09-09T22:48:46.399 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:48:46.399 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:48:46.399 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:48:46.399 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:48:46.399 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:48:46.400 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:48:46.401 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:48:46.401 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:48:46.401 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:48:46.401 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:48:46Z" 2022-09-09T22:48:46.402 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:48:46.403 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763726.109956_113.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:48:46.109956Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:48:46Z" } 2022-09-09T22:48:46.404 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763726.109956_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:48:46.109956Z', 'realm_epoch': 2}}] 2022-09-09T22:48:46.405 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763726.109956_113.1'} 2022-09-09T22:48:46.406 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763726.109956_113.1'} 2022-09-09T22:48:46.407 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-09T22:48:46.408 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-09T22:48:46.409 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:48:46.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:48:46.411 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:48:46.458 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:48:46.459 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:48:46.514 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:48:46.509+0000 7f4287462dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:48:46.516 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:48:46.513+0000 7f4287462dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:48:46.542 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:48:46.542 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T22:48:46.543 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T22:48:46.543 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T22:48:46.543 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T22:48:46.544 INFO:teuthology.orchestra.run.smithi005.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:48:46.544 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:48:46.544 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:48:46.544 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T22:48:46.545 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:48:46.545 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T22:48:46.545 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:48:46.546 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:48:46.546 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:48:46.546 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:48:46.546 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 1, 2022-09-09T22:48:46.547 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:48:46.547 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:48:46.547 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:48:46.547 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:48:46.548 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:48:46.548 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:48:46.548 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T22:48:46.548 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:48:46.549 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:48:46.549 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:48:46.549 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:48:46.550 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:48:46.551 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:48:46.551 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:48:46.551 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:48:46.552 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:48:46.552 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:48:46.552 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:48:46.552 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T22:48:46.553 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:48:46.553 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:48:46.553 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:48:46.555 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:48:46.555 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:48:46.555 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:48:46.555 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:48:46.556 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:48:46.556 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:48:46.556 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:48:46.557 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:48:46.557 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T22:48:46.557 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:48:46.558 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:48:46.558 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662763726.109956_113.1", 2022-09-09T22:48:46.558 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:48:46.558 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:48:46.559 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:48:46.559 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:48:46.109956Z", 2022-09-09T22:48:46.559 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:48:46.559 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:48:46.560 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:48:46.560 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:48:46.560 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:48:46.561 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T22:48:46.561 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 1, 2022-09-09T22:48:46.562 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 1 2022-09-09T22:48:46.563 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:48:46.563 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T22:48:46Z" 2022-09-09T22:48:46.564 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:48:46.565 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763726.109956_113.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:48:46.109956Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:48:46Z" } 2022-09-09T22:48:46.566 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763726.109956_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:48:46.109956Z', 'realm_epoch': 2}}] 2022-09-09T22:48:46.567 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763726.109956_113.1'} 2022-09-09T22:48:46.567 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763726.109956_113.1'} 2022-09-09T22:48:46.568 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-09T22:48:46.568 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-09T22:48:46.569 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:48:46.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:48:46.570 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:48:46.621 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:48:46.622 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:48:46.687 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:48:46.681+0000 7f3514edddc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:48:46.690 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:48:46.685+0000 7f3514edddc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:48:46.715 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:48:46.716 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:48:46.716 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:48:46.716 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:48:46.717 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:48:46.717 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:48:46.718 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:48:46.718 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:48:46.719 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:48:46.719 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:48:46.719 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:48:46.720 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:48:46.720 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:48:46.721 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:48:46.721 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:48:46.722 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:48:46.722 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:48:46.723 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:48:46.723 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:48:46.724 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:48:46.724 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:48:46.725 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:48:46.725 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:48:46.725 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:48:46.726 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:48:46.726 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:48:46.727 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:48:46.727 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:48:46.728 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:48:46.728 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:48:46.729 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:48:46.729 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:48:46.730 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:48:46.730 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:48:46.730 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:48:46.731 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:48:46.731 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:48:46.732 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:48:46.732 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:48:46.732 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:48:46.733 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:48:46.733 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:48:46.734 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:48:46.734 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:48:46.735 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:48:46.735 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:48:46.736 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:48:46.736 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:48:46.736 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:48:46.737 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763726.109956_113.1", 2022-09-09T22:48:46.737 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:48:46.737 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:48:46.738 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:48:46.738 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:48:46.109956Z", 2022-09-09T22:48:46.738 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:48:46.738 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:48:46.739 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:48:46.739 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:48:46.739 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:48:46.739 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:48:46.740 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:48:46.740 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:48:46.740 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:48:46.740 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:48:46Z" 2022-09-09T22:48:46.741 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:48:46.742 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763726.109956_113.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:48:46.109956Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:48:46Z" } 2022-09-09T22:48:46.743 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763726.109956_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:48:46.109956Z', 'realm_epoch': 2}}] 2022-09-09T22:48:46.743 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763726.109956_113.1'} 2022-09-09T22:48:46.744 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763726.109956_113.1'} 2022-09-09T22:48:46.744 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-09T22:48:46.745 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:48:46.745 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:48:46.746 DEBUG:boto:Method: HEAD 2022-09-09T22:48:46.746 DEBUG:boto:Path: /jkcgbj-20/ 2022-09-09T22:48:46.747 DEBUG:boto:Data: 2022-09-09T22:48:46.747 DEBUG:boto:Headers: {} 2022-09-09T22:48:46.748 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:46.748 DEBUG:boto:Port: 8000 2022-09-09T22:48:46.749 DEBUG:boto:Params: {} 2022-09-09T22:48:46.749 DEBUG:boto:Token: None 2022-09-09T22:48:46.750 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:48:46 GMT /jkcgbj-20/ 2022-09-09T22:48:46.751 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:yWIkqe529F1ExJsdBm56dxqzFio= 2022-09-09T22:48:46.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:48:46 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:yWIkqe529F1ExJsdBm56dxqzFio=', 'Content-Length': '0'} 2022-09-09T22:48:46.759 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000acc7162bd1e7ab2-00631bc2ce-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:48:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:46.760 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:48:46.760 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:48:46.761 DEBUG:boto:Method: PUT 2022-09-09T22:48:46.762 DEBUG:boto:Path: /jkcgbj-20/obj 2022-09-09T22:48:46.762 DEBUG:boto:Data: 2022-09-09T22:48:46.763 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-09T22:48:46.763 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:48:46.764 DEBUG:boto:Port: 8000 2022-09-09T22:48:46.764 DEBUG:boto:Params: {} 2022-09-09T22:48:46.765 DEBUG:boto:Token: None 2022-09-09T22:48:46.765 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 09 Sep 2022 22:48:46 GMT /jkcgbj-20/obj 2022-09-09T22:48:46.766 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0Thl0dQJzeXJ6OuuB3EJPaxqQcE= 2022-09-09T22:48:46.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:48:46 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0Thl0dQJzeXJ6OuuB3EJPaxqQcE='} 2022-09-09T22:48:46.782 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000087520fbf9bc9e233-00631bc2ce-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:48:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:48:46.783 DEBUG:rgw_multi.tests:created null version id=None 2022-09-09T22:48:46.783 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jkcgbj-20 2022-09-09T22:48:46.784 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:48:46.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:48:46.785 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:48:46.842 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:48:46.843 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:48:46.922 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:48:46.917+0000 7f9eadf6bdc0 1 waiting for incremental sync to catch up: 2022-09-09T22:48:46.923 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000003.26.12, , ] 2022-09-09T22:48:46.923 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000005.30.17, , ] 2022-09-09T22:48:51.925 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:48:51.921+0000 7f9eadf6bdc0 1 waiting for incremental sync to catch up: 2022-09-09T22:48:51.925 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000003.26.12, , ] 2022-09-09T22:48:51.926 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000005.30.17, , ] 2022-09-09T22:48:56.927 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:48:56.921+0000 7f9eadf6bdc0 1 waiting for incremental sync to catch up: 2022-09-09T22:48:56.927 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000003.26.12, , ] 2022-09-09T22:48:56.928 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000005.30.17, , ] 2022-09-09T22:49:01.930 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:49:01.925+0000 7f9eadf6bdc0 1 bucket sync caught up with source: 2022-09-09T22:49:01.930 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000005.30.17, , ] 2022-09-09T22:49:01.930 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000005.30.17, , ] 2022-09-09T22:49:01.931 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:49:01.925+0000 7f9eadf6bdc0 0 bucket checkpoint complete 2022-09-09T22:49:01.960 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jkcgbj-20 2022-09-09T22:49:02.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:49:02.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:49:02.078 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:49:02.134 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:49:02.134 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:49:02.198 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:49:02.196+0000 7fe9a5d22dc0 1 bucket sync caught up with source: 2022-09-09T22:49:02.199 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000005.30.17, , ] 2022-09-09T22:49:02.199 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000005.30.17, , ] 2022-09-09T22:49:02.199 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:49:02.196+0000 7fe9a5d22dc0 0 bucket checkpoint complete 2022-09-09T22:49:02.224 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jkcgbj-20 2022-09-09T22:49:02.225 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:49:02.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:49:02.226 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:49:02.279 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:49:02.280 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:49:02.364 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:49:02.357+0000 7fc4f8577dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:49:02.365 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000003.26.12, , ] 2022-09-09T22:49:02.365 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000005.30.17, , ] 2022-09-09T22:49:07.366 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:49:07.361+0000 7fc4f8577dc0 1 bucket sync caught up with source: 2022-09-09T22:49:07.366 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000005.30.17, , ] 2022-09-09T22:49:07.367 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000005.30.17, , ] 2022-09-09T22:49:07.367 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:49:07.361+0000 7fc4f8577dc0 0 bucket checkpoint complete 2022-09-09T22:49:07.393 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jkcgbj-20 2022-09-09T22:49:07.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:49:07.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:49:07.395 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:49:07.449 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:49:07.449 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:49:07.521 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:49:07.516+0000 7f7ad015cdc0 1 bucket sync caught up with source: 2022-09-09T22:49:07.522 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000006.42.17, , ] 2022-09-09T22:49:07.523 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000006.42.17, , ] 2022-09-09T22:49:07.523 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:49:07.516+0000 7f7ad015cdc0 0 bucket checkpoint complete 2022-09-09T22:49:07.548 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jkcgbj-20 2022-09-09T22:49:07.549 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:49:07.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:49:07.550 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:49:07.603 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:49:07.603 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:49:07.668 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:49:07.664+0000 7fead95a5dc0 1 bucket sync caught up with source: 2022-09-09T22:49:07.669 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000006.42.17, , ] 2022-09-09T22:49:07.669 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000006.42.17, , ] 2022-09-09T22:49:07.670 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:49:07.664+0000 7fead95a5dc0 0 bucket checkpoint complete 2022-09-09T22:49:07.695 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jkcgbj-20 2022-09-09T22:49:07.696 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jkcgbj-20 2022-09-09T22:49:07.696 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:49:07.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:49:07.697 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:49:07.747 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:49:07.748 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:49:07.816 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:49:07.812+0000 7fabff9a4dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:49:07.816 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000004.53.9, , ] 2022-09-09T22:49:07.816 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000007.60.14, , ] 2022-09-09T22:49:12.819 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:49:12.816+0000 7fabff9a4dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:49:12.819 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000004.53.9, , ] 2022-09-09T22:49:12.820 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000007.60.14, , ] 2022-09-09T22:49:17.822 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:49:17.816+0000 7fabff9a4dc0 1 waiting for incremental sync to catch up: 2022-09-09T22:49:17.822 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000004.53.9, , ] 2022-09-09T22:49:17.823 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000007.60.14, , ] 2022-09-09T22:49:22.824 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:49:22.820+0000 7fabff9a4dc0 1 bucket sync caught up with source: 2022-09-09T22:49:22.825 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000007.60.14, , ] 2022-09-09T22:49:22.825 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000007.60.14, , ] 2022-09-09T22:49:22.825 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:49:22.820+0000 7fabff9a4dc0 0 bucket checkpoint complete 2022-09-09T22:49:22.854 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jkcgbj-20 2022-09-09T22:49:22.855 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:49:22.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:49:22.856 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:49:22.908 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:49:22.909 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:49:22.984 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:49:22.980+0000 7f6c474e5dc0 1 bucket sync caught up with source: 2022-09-09T22:49:22.985 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000007.60.14, , ] 2022-09-09T22:49:22.985 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000007.60.14, , ] 2022-09-09T22:49:22.986 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:49:22.980+0000 7f6c474e5dc0 0 bucket checkpoint complete 2022-09-09T22:49:23.013 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jkcgbj-20 2022-09-09T22:49:23.014 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-20 zones={test-zone2, test-zone1} 2022-09-09T22:49:23.015 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:49:23.016 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:49:23.017 DEBUG:boto:Method: HEAD 2022-09-09T22:49:23.018 DEBUG:boto:Path: /jkcgbj-20/ 2022-09-09T22:49:23.019 DEBUG:boto:Data: 2022-09-09T22:49:23.019 DEBUG:boto:Headers: {} 2022-09-09T22:49:23.020 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:49:23.021 DEBUG:boto:Port: 8000 2022-09-09T22:49:23.022 DEBUG:boto:Params: {} 2022-09-09T22:49:23.023 DEBUG:boto:Token: None 2022-09-09T22:49:23.023 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:49:23 GMT /jkcgbj-20/ 2022-09-09T22:49:23.024 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:u8yflVP2+7O1Cc0hjzPHa+3Quak= 2022-09-09T22:49:23.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:49:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:u8yflVP2+7O1Cc0hjzPHa+3Quak=', 'Content-Length': '0'} 2022-09-09T22:49:23.029 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083692c2f4e0524eb-00631bc2f3-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:49:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:49:23.030 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:49:23.031 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:49:23.032 DEBUG:boto:Method: HEAD 2022-09-09T22:49:23.033 DEBUG:boto:Path: /jkcgbj-20/ 2022-09-09T22:49:23.034 DEBUG:boto:Data: 2022-09-09T22:49:23.035 DEBUG:boto:Headers: {} 2022-09-09T22:49:23.035 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:49:23.036 DEBUG:boto:Port: 8000 2022-09-09T22:49:23.037 DEBUG:boto:Params: {} 2022-09-09T22:49:23.038 DEBUG:boto:Token: None 2022-09-09T22:49:23.039 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:49:23 GMT /jkcgbj-20/ 2022-09-09T22:49:23.039 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:u8yflVP2+7O1Cc0hjzPHa+3Quak= 2022-09-09T22:49:23.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:49:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:u8yflVP2+7O1Cc0hjzPHa+3Quak=', 'Content-Length': '0'} 2022-09-09T22:49:23.046 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001b3e64904d7f60dd-00631bc2f3-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:49:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:49:23.047 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:49:23.047 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:49:23.048 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:49:23.048 DEBUG:boto:path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:49:23.049 DEBUG:boto:auth_path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:49:23.050 DEBUG:boto:Method: GET 2022-09-09T22:49:23.050 DEBUG:boto:Path: /jkcgbj-20/?versions&max-keys=999 2022-09-09T22:49:23.051 DEBUG:boto:Data: 2022-09-09T22:49:23.052 DEBUG:boto:Headers: {} 2022-09-09T22:49:23.053 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:49:23.053 DEBUG:boto:Port: 8000 2022-09-09T22:49:23.054 DEBUG:boto:Params: {} 2022-09-09T22:49:23.054 DEBUG:boto:Token: None 2022-09-09T22:49:23.055 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:49:23 GMT /jkcgbj-20/?versions 2022-09-09T22:49:23.056 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6rxVxA1TFYLLlAUA4uXQ3+UryOE= 2022-09-09T22:49:23.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:49:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6rxVxA1TFYLLlAUA4uXQ3+UryOE=', 'Content-Length': '0'} 2022-09-09T22:49:23.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049394b82fc352e92-00631bc2f3-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:49:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:49:23.063 DEBUG:boto:b'jkcgbj-20999falseobjnulltrue2022-09-09T22:48:46.768Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobji9PSHBUuQ8xfOhtiVzZhEqzCuln.uXjfalse2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:49:23.064 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:49:23.064 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:49:23.065 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:49:23.065 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:49:23.066 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:49:23.067 DEBUG:boto:path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:49:23.068 DEBUG:boto:auth_path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:49:23.068 DEBUG:boto:Method: GET 2022-09-09T22:49:23.069 DEBUG:boto:Path: /jkcgbj-20/?versions&max-keys=999 2022-09-09T22:49:23.069 DEBUG:boto:Data: 2022-09-09T22:49:23.070 DEBUG:boto:Headers: {} 2022-09-09T22:49:23.071 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:49:23.072 DEBUG:boto:Port: 8000 2022-09-09T22:49:23.072 DEBUG:boto:Params: {} 2022-09-09T22:49:23.073 DEBUG:boto:Token: None 2022-09-09T22:49:23.074 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:49:23 GMT /jkcgbj-20/?versions 2022-09-09T22:49:23.074 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6rxVxA1TFYLLlAUA4uXQ3+UryOE= 2022-09-09T22:49:23.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:49:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6rxVxA1TFYLLlAUA4uXQ3+UryOE=', 'Content-Length': '0'} 2022-09-09T22:49:23.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5e4ae2bb93df68c-00631bc2f3-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:49:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:49:23.082 DEBUG:boto:b'jkcgbj-20999falseobjnulltrue2022-09-09T22:48:46.768Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobji9PSHBUuQ8xfOhtiVzZhEqzCuln.uXjfalse2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:49:23.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:49:23.084 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:49:23.084 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:49:23.085 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:49:23.085 DEBUG:boto:path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:49:23.086 DEBUG:boto:auth_path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:49:23.086 DEBUG:boto:Method: GET 2022-09-09T22:49:23.087 DEBUG:boto:Path: /jkcgbj-20/?versions&max-keys=999 2022-09-09T22:49:23.087 DEBUG:boto:Data: 2022-09-09T22:49:23.088 DEBUG:boto:Headers: {} 2022-09-09T22:49:23.088 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:49:23.089 DEBUG:boto:Port: 8000 2022-09-09T22:49:23.089 DEBUG:boto:Params: {} 2022-09-09T22:49:23.090 DEBUG:boto:Token: None 2022-09-09T22:49:23.090 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:49:23 GMT /jkcgbj-20/?versions 2022-09-09T22:49:23.091 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6rxVxA1TFYLLlAUA4uXQ3+UryOE= 2022-09-09T22:49:23.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:49:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6rxVxA1TFYLLlAUA4uXQ3+UryOE=', 'Content-Length': '0'} 2022-09-09T22:49:23.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1ec9e7ce3833f5c-00631bc2f3-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:49:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:49:23.097 DEBUG:boto:b'jkcgbj-20999falseobjnulltrue2022-09-09T22:48:46.768Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobji9PSHBUuQ8xfOhtiVzZhEqzCuln.uXjfalse2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:49:23.098 DEBUG:boto:path=/jkcgbj-20/ 2022-09-09T22:49:23.098 DEBUG:boto:auth_path=/jkcgbj-20/ 2022-09-09T22:49:23.099 DEBUG:boto:path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:49:23.099 DEBUG:boto:auth_path=/jkcgbj-20/?versions&max-keys=999 2022-09-09T22:49:23.100 DEBUG:boto:Method: GET 2022-09-09T22:49:23.100 DEBUG:boto:Path: /jkcgbj-20/?versions&max-keys=999 2022-09-09T22:49:23.101 DEBUG:boto:Data: 2022-09-09T22:49:23.101 DEBUG:boto:Headers: {} 2022-09-09T22:49:23.102 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:49:23.102 DEBUG:boto:Port: 8000 2022-09-09T22:49:23.103 DEBUG:boto:Params: {} 2022-09-09T22:49:23.104 DEBUG:boto:Token: None 2022-09-09T22:49:23.104 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:49:23 GMT /jkcgbj-20/?versions 2022-09-09T22:49:23.105 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6rxVxA1TFYLLlAUA4uXQ3+UryOE= 2022-09-09T22:49:23.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:49:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6rxVxA1TFYLLlAUA4uXQ3+UryOE=', 'Content-Length': '0'} 2022-09-09T22:49:23.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc741ff5e5eeed9a-00631bc2f3-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:49:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:49:23.158 DEBUG:boto:b'jkcgbj-20999falseobjnulltrue2022-09-09T22:48:46.768Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobji9PSHBUuQ8xfOhtiVzZhEqzCuln.uXjfalse2022-09-09T22:48:17.853Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:49:23.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:49:23.160 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:49:23.161 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:49:23.161 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:49:23.162 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:49:23.162 DEBUG:boto:Method: GET 2022-09-09T22:49:23.163 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:49:23.163 DEBUG:boto:Data: 2022-09-09T22:49:23.164 DEBUG:boto:Headers: {} 2022-09-09T22:49:23.164 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:49:23.165 DEBUG:boto:Port: 8000 2022-09-09T22:49:23.165 DEBUG:boto:Params: {} 2022-09-09T22:49:23.166 DEBUG:boto:Token: None 2022-09-09T22:49:23.167 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:49:23 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:49:23.167 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:t0YwNmb2wMYKgN7m3PSYr4KaDNc= 2022-09-09T22:49:23.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:49:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:t0YwNmb2wMYKgN7m3PSYr4KaDNc=', 'Content-Length': '0'} 2022-09-09T22:49:23.171 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d17d2c5c1be54df6-00631bc2f3-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:49:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:49:23.172 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:49:23.173 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:49:23.174 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:49:23.174 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:49:23.175 DEBUG:boto:Method: GET 2022-09-09T22:49:23.175 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:49:23.176 DEBUG:boto:Data: 2022-09-09T22:49:23.176 DEBUG:boto:Headers: {} 2022-09-09T22:49:23.177 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:49:23.178 DEBUG:boto:Port: 8000 2022-09-09T22:49:23.179 DEBUG:boto:Params: {} 2022-09-09T22:49:23.179 DEBUG:boto:Token: None 2022-09-09T22:49:23.180 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:49:23 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:49:23.181 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:t0YwNmb2wMYKgN7m3PSYr4KaDNc= 2022-09-09T22:49:23.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:49:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:t0YwNmb2wMYKgN7m3PSYr4KaDNc=', 'Content-Length': '0'} 2022-09-09T22:49:23.184 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002a7d97b88cf1f3b6-00631bc2f3-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:49:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:49:23.185 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:49:23.186 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:49:23.186 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:49:23.187 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:49:23.187 DEBUG:boto:Method: HEAD 2022-09-09T22:49:23.188 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:49:23.188 DEBUG:boto:Data: 2022-09-09T22:49:23.189 DEBUG:boto:Headers: {} 2022-09-09T22:49:23.189 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:49:23.190 DEBUG:boto:Port: 8000 2022-09-09T22:49:23.190 DEBUG:boto:Params: {} 2022-09-09T22:49:23.191 DEBUG:boto:Token: None 2022-09-09T22:49:23.192 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:49:23 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:49:23.192 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OENqsrxGp+t+EaQSmH+JPp3VBmw= 2022-09-09T22:49:23.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:49:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OENqsrxGp+t+EaQSmH+JPp3VBmw=', 'Content-Length': '0'} 2022-09-09T22:49:23.196 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:47:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000445b213c645a9c0d-00631bc2f3-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:49:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:49:23.197 DEBUG:boto:path=/jkcgbj-20/obj 2022-09-09T22:49:23.197 DEBUG:boto:auth_path=/jkcgbj-20/obj 2022-09-09T22:49:23.198 DEBUG:boto:path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:49:23.198 DEBUG:boto:auth_path=/jkcgbj-20/obj?versionId=null 2022-09-09T22:49:23.199 DEBUG:boto:Method: HEAD 2022-09-09T22:49:23.199 DEBUG:boto:Path: /jkcgbj-20/obj?versionId=null 2022-09-09T22:49:23.200 DEBUG:boto:Data: 2022-09-09T22:49:23.200 DEBUG:boto:Headers: {} 2022-09-09T22:49:23.201 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:49:23.201 DEBUG:boto:Port: 8000 2022-09-09T22:49:23.202 DEBUG:boto:Params: {} 2022-09-09T22:49:23.202 DEBUG:boto:Token: None 2022-09-09T22:49:23.203 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:49:23 GMT /jkcgbj-20/obj?versionId=null 2022-09-09T22:49:23.203 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OENqsrxGp+t+EaQSmH+JPp3VBmw= 2022-09-09T22:49:23.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:49:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OENqsrxGp+t+EaQSmH+JPp3VBmw=', 'Content-Length': '0'} 2022-09-09T22:49:23.207 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:48:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000da38354b57887aa6-00631bc2f3-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:49:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:49:23.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:49:23.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... FAIL 2022-09-09T22:49:23.214 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jkcgbj-21 2022-09-09T22:49:23.214 DEBUG:boto:path=/jkcgbj-21/ 2022-09-09T22:49:23.215 DEBUG:boto:auth_path=/jkcgbj-21/ 2022-09-09T22:49:23.215 DEBUG:boto:Method: PUT 2022-09-09T22:49:23.216 DEBUG:boto:Path: /jkcgbj-21/ 2022-09-09T22:49:23.216 DEBUG:boto:Data: 2022-09-09T22:49:23.217 DEBUG:boto:Headers: {} 2022-09-09T22:49:23.217 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:49:23.218 DEBUG:boto:Port: 8000 2022-09-09T22:49:23.218 DEBUG:boto:Params: {} 2022-09-09T22:49:23.219 DEBUG:boto:Token: None 2022-09-09T22:49:23.219 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:49:23 GMT /jkcgbj-21/ 2022-09-09T22:49:23.220 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Im3OHxjuKulCdZtZ8FplmPc1ReE= 2022-09-09T22:49:23.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:49:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Im3OHxjuKulCdZtZ8FplmPc1ReE=', 'Content-Length': '0'} 2022-09-09T22:49:23.248 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e7611f601524f138-00631bc2f3-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:49:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:49:23.248 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jkcgbj-22 2022-09-09T22:49:23.249 DEBUG:boto:path=/jkcgbj-22/ 2022-09-09T22:49:23.249 DEBUG:boto:auth_path=/jkcgbj-22/ 2022-09-09T22:49:23.250 DEBUG:boto:Method: PUT 2022-09-09T22:49:23.250 DEBUG:boto:Path: /jkcgbj-22/ 2022-09-09T22:49:23.251 DEBUG:boto:Data: 2022-09-09T22:49:23.251 DEBUG:boto:Headers: {} 2022-09-09T22:49:23.252 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:49:23.252 DEBUG:boto:Port: 8000 2022-09-09T22:49:23.253 DEBUG:boto:Params: {} 2022-09-09T22:49:23.254 DEBUG:boto:Token: None 2022-09-09T22:49:23.254 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:49:23 GMT /jkcgbj-22/ 2022-09-09T22:49:23.255 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:moG6Z/iIoo1jCfYNmNmcGI2qrWI= 2022-09-09T22:49:23.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:49:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:moG6Z/iIoo1jCfYNmNmcGI2qrWI=', 'Content-Length': '0'} 2022-09-09T22:49:23.304 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005d8762dfd0bc9b48-00631bc2f3-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:49:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:49:23.305 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jkcgbj-23 2022-09-09T22:49:23.305 DEBUG:boto:path=/jkcgbj-23/ 2022-09-09T22:49:23.306 DEBUG:boto:auth_path=/jkcgbj-23/ 2022-09-09T22:49:23.306 DEBUG:boto:Method: PUT 2022-09-09T22:49:23.307 DEBUG:boto:Path: /jkcgbj-23/ 2022-09-09T22:49:23.307 DEBUG:boto:Data: 2022-09-09T22:49:23.308 DEBUG:boto:Headers: {} 2022-09-09T22:49:23.310 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:49:23.310 DEBUG:boto:Port: 8001 2022-09-09T22:49:23.311 DEBUG:boto:Params: {} 2022-09-09T22:49:23.311 DEBUG:boto:Token: None 2022-09-09T22:49:23.312 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:49:23 GMT /jkcgbj-23/ 2022-09-09T22:49:23.313 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0xteNTkIN6S0krDzL55G/SQYTpM= 2022-09-09T22:49:23.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:49:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0xteNTkIN6S0krDzL55G/SQYTpM=', 'Content-Length': '0'} 2022-09-09T22:49:23.359 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d85e3b95c3be69c9-00631bc2f3-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:49:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:49:23.360 DEBUG:boto:path=/jkcgbj-21/ 2022-09-09T22:49:23.360 DEBUG:boto:auth_path=/jkcgbj-21/ 2022-09-09T22:49:23.360 DEBUG:boto:path=/jkcgbj-21/?versioning 2022-09-09T22:49:23.361 DEBUG:boto:auth_path=/jkcgbj-21/?versioning 2022-09-09T22:49:23.361 DEBUG:boto:Method: PUT 2022-09-09T22:49:23.362 DEBUG:boto:Path: /jkcgbj-21/?versioning 2022-09-09T22:49:23.362 DEBUG:boto:Data: Enabled Disabled 2022-09-09T22:49:23.363 DEBUG:boto:Headers: {} 2022-09-09T22:49:23.363 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:49:23.364 DEBUG:boto:Port: 8000 2022-09-09T22:49:23.364 DEBUG:boto:Params: {} 2022-09-09T22:49:23.365 DEBUG:boto:Token: None 2022-09-09T22:49:23.365 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:49:23 GMT /jkcgbj-21/?versioning 2022-09-09T22:49:23.366 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:WNoM2gImVsrVCA6fPBfL/fdEVTs= 2022-09-09T22:49:23.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:49:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:WNoM2gImVsrVCA6fPBfL/fdEVTs=', 'Content-Length': '228'} 2022-09-09T22:49:23.377 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008cd25a8bd9262c75-00631bc2f3-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:49:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:49:23.378 DEBUG:boto:path=/jkcgbj-22/ 2022-09-09T22:49:23.378 DEBUG:boto:auth_path=/jkcgbj-22/ 2022-09-09T22:49:23.379 DEBUG:boto:path=/jkcgbj-22/?versioning 2022-09-09T22:49:23.379 DEBUG:boto:auth_path=/jkcgbj-22/?versioning 2022-09-09T22:49:23.380 DEBUG:boto:Method: PUT 2022-09-09T22:49:23.380 DEBUG:boto:Path: /jkcgbj-22/?versioning 2022-09-09T22:49:23.381 DEBUG:boto:Data: Enabled Disabled 2022-09-09T22:49:23.381 DEBUG:boto:Headers: {} 2022-09-09T22:49:23.382 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:49:23.382 DEBUG:boto:Port: 8000 2022-09-09T22:49:23.383 DEBUG:boto:Params: {} 2022-09-09T22:49:23.383 DEBUG:boto:Token: None 2022-09-09T22:49:23.384 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:49:23 GMT /jkcgbj-22/?versioning 2022-09-09T22:49:23.384 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:cyFk4Ms5eDxAvsCAsAlJ8X/ozdM= 2022-09-09T22:49:23.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:49:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:cyFk4Ms5eDxAvsCAsAlJ8X/ozdM=', 'Content-Length': '228'} 2022-09-09T22:49:23.404 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000566e48d36b26a6e3-00631bc2f3-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:49:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:49:23.405 DEBUG:boto:path=/jkcgbj-23/ 2022-09-09T22:49:23.405 DEBUG:boto:auth_path=/jkcgbj-23/ 2022-09-09T22:49:23.406 DEBUG:boto:path=/jkcgbj-23/?versioning 2022-09-09T22:49:23.407 DEBUG:boto:auth_path=/jkcgbj-23/?versioning 2022-09-09T22:49:23.407 DEBUG:boto:Method: PUT 2022-09-09T22:49:23.408 DEBUG:boto:Path: /jkcgbj-23/?versioning 2022-09-09T22:49:23.408 DEBUG:boto:Data: Enabled Disabled 2022-09-09T22:49:23.409 DEBUG:boto:Headers: {} 2022-09-09T22:49:23.410 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:49:23.410 DEBUG:boto:Port: 8001 2022-09-09T22:49:23.411 DEBUG:boto:Params: {} 2022-09-09T22:49:23.411 DEBUG:boto:Token: None 2022-09-09T22:49:23.412 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:49:23 GMT /jkcgbj-23/?versioning 2022-09-09T22:49:23.413 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UoGfjI1VsicxKQUi1mD+zbZe8rc= 2022-09-09T22:49:23.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:49:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UoGfjI1VsicxKQUi1mD+zbZe8rc=', 'Content-Length': '228'} 2022-09-09T22:49:23.437 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002b5bc3c07db2c6ea-00631bc2f3-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:49:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:49:23.438 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:49:23.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:49:23.439 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:49:23.485 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:49:23.485 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:49:23.534 INFO:teuthology.orchestra.run.smithi005.stderr:No --period given, using current period=503b1d61-4513-48c8-a308-17b76219610c 2022-09-09T22:49:23.563 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T22:49:23.563 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:49:23.563 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:49:23.564 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:28:06.146147Z" 2022-09-09T22:49:23.564 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:49:23.564 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:49:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:49:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:49:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:49:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:49:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:49:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:30:37.757014Z" 2022-09-09T22:49:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:49:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:49:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662763763.426933_131.1", 2022-09-09T22:49:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:49:23.426933Z" 2022-09-09T22:49:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:49:23.568 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T22:49:23.570 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763763.426933_131.1'} 2022-09-09T22:49:23.570 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-09T22:49:23.571 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:49:23.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:49:23.572 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:49:23.620 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:49:23.620 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:49:23.703 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:49:23.696+0000 7f19d81d3dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:49:23.705 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:49:23.700+0000 7f19d81d3dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:49:23.730 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:49:23.730 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:49:23.731 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:49:23.731 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:49:23.731 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:49:23.731 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:49:23.732 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:49:23.732 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:49:23.733 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:49:23.733 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:49:23.733 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:49:23.733 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:49:23.734 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:49:23.734 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:49:23.734 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:49:23.735 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:49:23.735 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:49:23.735 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:49:23.735 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:49:23.736 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:49:23.736 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:49:23.736 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:49:23.736 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:49:23.737 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:49:23.737 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:49:23.737 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:49:23.737 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:49:23.738 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:49:23.738 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:49:23.738 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:49:23.739 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:49:23.739 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:49:23.739 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:49:23.739 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:49:23.740 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:49:23.740 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:49:23.740 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:49:23.740 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:49:23.741 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:49:23.741 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:49:23.741 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:49:23.741 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:49:23.742 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:49:23.742 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:49:23.742 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:49:23.743 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:49:23.743 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:49:23.744 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:49:23.744 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:49:23.744 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763763.239008_115.1", 2022-09-09T22:49:23.744 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:49:23.745 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:49:23.745 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:49:23.745 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.239008Z", 2022-09-09T22:49:23.745 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:49:23.746 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:49:23.746 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:49:23.746 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:49:23.747 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:49:23.747 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:49:23.747 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:49:23.747 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:49:23.748 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:49:23.748 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:49:23Z" 2022-09-09T22:49:23.748 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:49:23.750 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763763.239008_115.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:49:23.239008Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:49:23Z" } 2022-09-09T22:49:23.750 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763763.239008_115.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:49:23.239008Z', 'realm_epoch': 2}}] 2022-09-09T22:49:23.751 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763763.426933_131.1'} 2022-09-09T22:49:23.751 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763763.239008_115.1'} 2022-09-09T22:49:23.752 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763763.426933_131.1 target=1_1662763763.239008_115.1 2022-09-09T22:49:28.753 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:49:28.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:49:28.754 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:49:28.802 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:49:28.803 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:49:28.855 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:49:28.848+0000 7f3764d7cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:49:28.856 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:49:28.852+0000 7f3764d7cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:49:28.883 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:49:28.883 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:49:28.884 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:49:28.884 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:49:28.884 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:49:28.884 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:49:28.885 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:49:28.885 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:49:28.885 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:49:28.885 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:49:28.886 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:49:28.886 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:49:28.886 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:49:28.887 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:49:28.887 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:49:28.887 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:49:28.888 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:49:28.888 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:49:28.889 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:49:28.889 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:49:28.890 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:49:28.890 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:49:28.890 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:49:28.891 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:49:28.891 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:49:28.891 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:49:28.891 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:49:28.892 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:49:28.892 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:49:28.892 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:49:28.892 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:49:28.893 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:49:28.893 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:49:28.893 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:49:28.894 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:49:28.894 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:49:28.894 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:49:28.894 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:49:28.895 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:49:28.895 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:49:28.895 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:49:28.895 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:49:28.896 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:49:28.896 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:49:28.896 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:49:28.897 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:49:28.897 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:49:28.897 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:49:28.898 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:49:28.898 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763763.426933_131.1", 2022-09-09T22:49:28.899 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:49:28.899 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:49:28.899 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:49:28.899 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.426933Z", 2022-09-09T22:49:28.900 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:49:28.900 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:49:28.900 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:49:28.900 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:49:28.901 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:49:28.901 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:49:28.901 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:49:28.901 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:49:28.902 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:49:28.902 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:49:28Z" 2022-09-09T22:49:28.902 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:49:28.904 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763763.426933_131.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:49:23.426933Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:49:28Z" } 2022-09-09T22:49:28.904 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763763.426933_131.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:49:23.426933Z', 'realm_epoch': 2}}] 2022-09-09T22:49:28.905 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763763.426933_131.1'} 2022-09-09T22:49:28.905 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763763.426933_131.1'} 2022-09-09T22:49:28.906 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-09T22:49:28.907 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-09T22:49:28.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:49:28.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:49:28.908 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:49:28.955 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:49:28.955 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:49:29.009 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:49:29.004+0000 7f39466d5dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:49:29.011 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:49:29.008+0000 7f39466d5dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:49:29.036 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:49:29.036 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T22:49:29.036 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T22:49:29.037 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T22:49:29.037 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T22:49:29.037 INFO:teuthology.orchestra.run.smithi005.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:49:29.038 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:49:29.038 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:49:29.038 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T22:49:29.039 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:49:29.039 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T22:49:29.039 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:49:29.039 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:49:29.040 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:49:29.040 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:49:29.040 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 1, 2022-09-09T22:49:29.040 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:49:29.041 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:49:29.041 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:49:29.041 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:49:29.041 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:49:29.042 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:49:29.042 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T22:49:29.042 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:49:29.042 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:49:29.043 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:49:29.043 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:49:29.043 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:49:29.043 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:49:29.044 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:49:29.044 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:49:29.044 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:49:29.044 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:49:29.045 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:49:29.045 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T22:49:29.045 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:49:29.045 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:49:29.046 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:49:29.046 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:49:29.046 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:49:29.046 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:49:29.047 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:49:29.047 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:49:29.047 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:49:29.047 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:49:29.048 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:49:29.048 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T22:49:29.048 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:49:29.048 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:49:29.049 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662763763.426933_131.1", 2022-09-09T22:49:29.049 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:49:29.049 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:49:29.049 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:49:29.050 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.426933Z", 2022-09-09T22:49:29.050 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:49:29.050 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:49:29.051 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:49:29.051 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:49:29.051 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:49:29.052 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T22:49:29.052 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 1, 2022-09-09T22:49:29.052 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 1 2022-09-09T22:49:29.052 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:49:29.053 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T22:49:29Z" 2022-09-09T22:49:29.053 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:49:29.054 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763763.426933_131.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:49:23.426933Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:49:29Z" } 2022-09-09T22:49:29.055 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763763.426933_131.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:49:23.426933Z', 'realm_epoch': 2}}] 2022-09-09T22:49:29.056 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763763.426933_131.1'} 2022-09-09T22:49:29.056 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763763.426933_131.1'} 2022-09-09T22:49:29.057 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-09T22:49:29.057 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-09T22:49:29.058 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:49:29.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:49:29.059 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:49:29.106 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:49:29.107 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:49:29.179 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:49:29.172+0000 7fa584a2adc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:49:29.182 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:49:29.176+0000 7fa584a2adc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:49:29.209 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:49:29.210 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:49:29.211 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:49:29.211 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:49:29.212 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:49:29.212 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:49:29.212 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:49:29.213 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:49:29.213 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:49:29.213 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:49:29.213 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:49:29.214 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:49:29.214 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:49:29.214 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:49:29.215 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:49:29.215 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:49:29.216 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:49:29.216 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:49:29.217 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:49:29.217 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:49:29.218 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:49:29.218 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:49:29.219 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:49:29.219 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:49:29.220 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:49:29.220 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:49:29.221 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:49:29.221 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:49:29.222 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:49:29.222 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:49:29.222 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:49:29.223 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:49:29.223 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:49:29.223 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:49:29.223 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:49:29.224 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:49:29.224 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:49:29.224 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:49:29.224 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:49:29.225 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:49:29.225 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:49:29.225 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:49:29.225 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:49:29.226 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:49:29.226 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:49:29.226 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:49:29.226 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:49:29.227 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:49:29.227 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:49:29.227 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763763.426933_131.1", 2022-09-09T22:49:29.227 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:49:29.228 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:49:29.228 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:49:29.228 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.426933Z", 2022-09-09T22:49:29.228 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:49:29.229 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:49:29.229 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:49:29.229 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:49:29.229 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:49:29.230 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:49:29.230 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:49:29.230 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:49:29.231 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:49:29.231 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:49:29Z" 2022-09-09T22:49:29.231 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:49:29.232 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763763.426933_131.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:49:23.426933Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:49:29Z" } 2022-09-09T22:49:29.233 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763763.426933_131.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:49:23.426933Z', 'realm_epoch': 2}}] 2022-09-09T22:49:29.234 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763763.426933_131.1'} 2022-09-09T22:49:29.234 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763763.426933_131.1'} 2022-09-09T22:49:29.235 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-09T22:49:29.235 DEBUG:boto:path=/jkcgbj-21/ 2022-09-09T22:49:29.236 DEBUG:boto:auth_path=/jkcgbj-21/ 2022-09-09T22:49:29.236 DEBUG:boto:Method: HEAD 2022-09-09T22:49:29.237 DEBUG:boto:Path: /jkcgbj-21/ 2022-09-09T22:49:29.237 DEBUG:boto:Data: 2022-09-09T22:49:29.238 DEBUG:boto:Headers: {} 2022-09-09T22:49:29.238 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:49:29.239 DEBUG:boto:Port: 8000 2022-09-09T22:49:29.239 DEBUG:boto:Params: {} 2022-09-09T22:49:29.240 DEBUG:boto:Token: None 2022-09-09T22:49:29.241 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:49:29 GMT /jkcgbj-21/ 2022-09-09T22:49:29.241 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:zBWGJlhKnIHluIoySHNJ3S78sQ0= 2022-09-09T22:49:29.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:49:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:zBWGJlhKnIHluIoySHNJ3S78sQ0=', 'Content-Length': '0'} 2022-09-09T22:49:29.248 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c004d2e4f95338ff-00631bc2f9-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:49:29.249 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:49:29.250 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:49:29.251 DEBUG:boto:Method: PUT 2022-09-09T22:49:29.252 DEBUG:boto:Path: /jkcgbj-21/obj 2022-09-09T22:49:29.253 DEBUG:boto:Data: 2022-09-09T22:49:29.254 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-09T22:49:29.255 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:49:29.256 DEBUG:boto:Port: 8000 2022-09-09T22:49:29.257 DEBUG:boto:Params: {} 2022-09-09T22:49:29.258 DEBUG:boto:Token: None 2022-09-09T22:49:29.258 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 09 Sep 2022 22:49:29 GMT /jkcgbj-21/obj 2022-09-09T22:49:29.259 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:xTOrG4baj7t71etXbYOmwJqdSs0= 2022-09-09T22:49:29.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:49:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:xTOrG4baj7t71etXbYOmwJqdSs0='} 2022-09-09T22:49:29.310 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'V-UiCIOWwBdw247KH0Oq0akAJzGvI5G'), ('x-amz-request-id', 'tx00000189b61941fd98d51-00631bc2f9-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:49:29.311 DEBUG:boto:path=/jkcgbj-21/ 2022-09-09T22:49:29.312 DEBUG:boto:auth_path=/jkcgbj-21/ 2022-09-09T22:49:29.312 DEBUG:boto:Method: HEAD 2022-09-09T22:49:29.313 DEBUG:boto:Path: /jkcgbj-21/ 2022-09-09T22:49:29.313 DEBUG:boto:Data: 2022-09-09T22:49:29.314 DEBUG:boto:Headers: {} 2022-09-09T22:49:29.315 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:49:29.316 DEBUG:boto:Port: 8000 2022-09-09T22:49:29.316 DEBUG:boto:Params: {} 2022-09-09T22:49:29.317 DEBUG:boto:Token: None 2022-09-09T22:49:29.317 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:49:29 GMT /jkcgbj-21/ 2022-09-09T22:49:29.318 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:zBWGJlhKnIHluIoySHNJ3S78sQ0= 2022-09-09T22:49:29.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:49:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:zBWGJlhKnIHluIoySHNJ3S78sQ0=', 'Content-Length': '0'} 2022-09-09T22:49:29.329 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb306d0903ce8082-00631bc2f9-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:49:29.331 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:49:29.332 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:49:29.333 DEBUG:boto:Method: DELETE 2022-09-09T22:49:29.334 DEBUG:boto:Path: /jkcgbj-21/obj 2022-09-09T22:49:29.335 DEBUG:boto:Data: 2022-09-09T22:49:29.336 DEBUG:boto:Headers: {} 2022-09-09T22:49:29.337 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:49:29.338 DEBUG:boto:Port: 8000 2022-09-09T22:49:29.339 DEBUG:boto:Params: {} 2022-09-09T22:49:29.340 DEBUG:boto:Token: None 2022-09-09T22:49:29.341 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:49:29 GMT /jkcgbj-21/obj 2022-09-09T22:49:29.342 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ardbbgpHwyAPERyw1w3k00dcmKU= 2022-09-09T22:49:29.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:49:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ardbbgpHwyAPERyw1w3k00dcmKU=', 'Content-Length': '0'} 2022-09-09T22:49:29.361 DEBUG:boto:Response headers: [('x-amz-version-id', 'o.3h3n2G6TMQ7YQ1vk1ad3FnvFpVNTg'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000cdd1fa09f1a8c87f-00631bc2f9-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:49:29.362 DEBUG:boto:path=/jkcgbj-22/ 2022-09-09T22:49:29.363 DEBUG:boto:auth_path=/jkcgbj-22/ 2022-09-09T22:49:29.363 DEBUG:boto:Method: HEAD 2022-09-09T22:49:29.364 DEBUG:boto:Path: /jkcgbj-22/ 2022-09-09T22:49:29.364 DEBUG:boto:Data: 2022-09-09T22:49:29.365 DEBUG:boto:Headers: {} 2022-09-09T22:49:29.365 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:49:29.365 DEBUG:boto:Port: 8000 2022-09-09T22:49:29.366 DEBUG:boto:Params: {} 2022-09-09T22:49:29.367 DEBUG:boto:Token: None 2022-09-09T22:49:29.367 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:49:29 GMT /jkcgbj-22/ 2022-09-09T22:49:29.368 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:r2z5d/LNi4z3+tFSXtTFuJxGwZU= 2022-09-09T22:49:29.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:49:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:r2z5d/LNi4z3+tFSXtTFuJxGwZU=', 'Content-Length': '0'} 2022-09-09T22:49:29.375 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003806e2bab803f7d2-00631bc2f9-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:49:29.376 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:49:29.377 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:49:29.378 DEBUG:boto:Method: PUT 2022-09-09T22:49:29.379 DEBUG:boto:Path: /jkcgbj-22/obj 2022-09-09T22:49:29.380 DEBUG:boto:Data: 2022-09-09T22:49:29.382 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-09T22:49:29.384 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:49:29.385 DEBUG:boto:Port: 8000 2022-09-09T22:49:29.386 DEBUG:boto:Params: {} 2022-09-09T22:49:29.387 DEBUG:boto:Token: None 2022-09-09T22:49:29.388 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 09 Sep 2022 22:49:29 GMT /jkcgbj-22/obj 2022-09-09T22:49:29.390 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:DaPWAYvX1AdfT2bTlxJhTg8zfq8= 2022-09-09T22:49:29.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:49:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:DaPWAYvX1AdfT2bTlxJhTg8zfq8='} 2022-09-09T22:49:29.421 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'wldLOHvZkprMnEge7.vgzLGeDVDkMUR'), ('x-amz-request-id', 'tx00000028a41ce7ba3f572-00631bc2f9-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:49:29.422 DEBUG:boto:path=/jkcgbj-22/ 2022-09-09T22:49:29.422 DEBUG:boto:auth_path=/jkcgbj-22/ 2022-09-09T22:49:29.423 DEBUG:boto:Method: HEAD 2022-09-09T22:49:29.423 DEBUG:boto:Path: /jkcgbj-22/ 2022-09-09T22:49:29.424 DEBUG:boto:Data: 2022-09-09T22:49:29.424 DEBUG:boto:Headers: {} 2022-09-09T22:49:29.425 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:49:29.425 DEBUG:boto:Port: 8000 2022-09-09T22:49:29.426 DEBUG:boto:Params: {} 2022-09-09T22:49:29.426 DEBUG:boto:Token: None 2022-09-09T22:49:29.427 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:49:29 GMT /jkcgbj-22/ 2022-09-09T22:49:29.427 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:r2z5d/LNi4z3+tFSXtTFuJxGwZU= 2022-09-09T22:49:29.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:49:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:r2z5d/LNi4z3+tFSXtTFuJxGwZU=', 'Content-Length': '0'} 2022-09-09T22:49:29.432 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3528b83f34394df-00631bc2f9-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:49:29.433 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:49:29.434 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:49:29.434 DEBUG:boto:Method: DELETE 2022-09-09T22:49:29.435 DEBUG:boto:Path: /jkcgbj-22/obj 2022-09-09T22:49:29.435 DEBUG:boto:Data: 2022-09-09T22:49:29.436 DEBUG:boto:Headers: {} 2022-09-09T22:49:29.436 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:49:29.437 DEBUG:boto:Port: 8000 2022-09-09T22:49:29.437 DEBUG:boto:Params: {} 2022-09-09T22:49:29.438 DEBUG:boto:Token: None 2022-09-09T22:49:29.439 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:49:29 GMT /jkcgbj-22/obj 2022-09-09T22:49:29.439 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:EU0kix2AZYNmbnY1AGD1POj4BYs= 2022-09-09T22:49:29.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:49:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:EU0kix2AZYNmbnY1AGD1POj4BYs=', 'Content-Length': '0'} 2022-09-09T22:49:29.457 DEBUG:boto:Response headers: [('x-amz-version-id', '9KGDz7kz0-0inGbhR-9Nw2HQLGID0YD'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000008fc2587001713f60-00631bc2f9-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:49:29.458 DEBUG:boto:path=/jkcgbj-23/ 2022-09-09T22:49:29.458 DEBUG:boto:auth_path=/jkcgbj-23/ 2022-09-09T22:49:29.459 DEBUG:boto:Method: HEAD 2022-09-09T22:49:29.460 DEBUG:boto:Path: /jkcgbj-23/ 2022-09-09T22:49:29.460 DEBUG:boto:Data: 2022-09-09T22:49:29.460 DEBUG:boto:Headers: {} 2022-09-09T22:49:29.461 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:49:29.461 DEBUG:boto:Port: 8001 2022-09-09T22:49:29.462 DEBUG:boto:Params: {} 2022-09-09T22:49:29.463 DEBUG:boto:Token: None 2022-09-09T22:49:29.463 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:49:29 GMT /jkcgbj-23/ 2022-09-09T22:49:29.464 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:iBH8FUQtplEr65axzbTmrxv+HrI= 2022-09-09T22:49:29.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:49:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:iBH8FUQtplEr65axzbTmrxv+HrI=', 'Content-Length': '0'} 2022-09-09T22:49:29.469 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007482c0119616394f-00631bc2f9-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:49:29.470 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:49:29.471 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:49:29.471 DEBUG:boto:Method: PUT 2022-09-09T22:49:29.472 DEBUG:boto:Path: /jkcgbj-23/obj 2022-09-09T22:49:29.472 DEBUG:boto:Data: 2022-09-09T22:49:29.473 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-09T22:49:29.473 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:49:29.474 DEBUG:boto:Port: 8001 2022-09-09T22:49:29.474 DEBUG:boto:Params: {} 2022-09-09T22:49:29.475 DEBUG:boto:Token: None 2022-09-09T22:49:29.476 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 09 Sep 2022 22:49:29 GMT /jkcgbj-23/obj 2022-09-09T22:49:29.476 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:4ve/BEkYhvnnLC9kAE8FDp9bXYg= 2022-09-09T22:49:29.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:49:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:4ve/BEkYhvnnLC9kAE8FDp9bXYg='} 2022-09-09T22:49:29.506 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'jqA7MkcivIkZnCalrNyyNmjK3EMR-GF'), ('x-amz-request-id', 'tx00000dc0b91b386ea6a95-00631bc2f9-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:49:29.506 DEBUG:boto:path=/jkcgbj-23/ 2022-09-09T22:49:29.507 DEBUG:boto:auth_path=/jkcgbj-23/ 2022-09-09T22:49:29.508 DEBUG:boto:Method: HEAD 2022-09-09T22:49:29.508 DEBUG:boto:Path: /jkcgbj-23/ 2022-09-09T22:49:29.509 DEBUG:boto:Data: 2022-09-09T22:49:29.509 DEBUG:boto:Headers: {} 2022-09-09T22:49:29.510 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:49:29.510 DEBUG:boto:Port: 8001 2022-09-09T22:49:29.511 DEBUG:boto:Params: {} 2022-09-09T22:49:29.511 DEBUG:boto:Token: None 2022-09-09T22:49:29.512 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:49:29 GMT /jkcgbj-23/ 2022-09-09T22:49:29.512 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:iBH8FUQtplEr65axzbTmrxv+HrI= 2022-09-09T22:49:29.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:49:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:iBH8FUQtplEr65axzbTmrxv+HrI=', 'Content-Length': '0'} 2022-09-09T22:49:29.517 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000505cd941be04aa75-00631bc2f9-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:49:29.518 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:49:29.519 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:49:29.519 DEBUG:boto:Method: DELETE 2022-09-09T22:49:29.520 DEBUG:boto:Path: /jkcgbj-23/obj 2022-09-09T22:49:29.520 DEBUG:boto:Data: 2022-09-09T22:49:29.521 DEBUG:boto:Headers: {} 2022-09-09T22:49:29.521 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:49:29.522 DEBUG:boto:Port: 8001 2022-09-09T22:49:29.523 DEBUG:boto:Params: {} 2022-09-09T22:49:29.523 DEBUG:boto:Token: None 2022-09-09T22:49:29.524 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:49:29 GMT /jkcgbj-23/obj 2022-09-09T22:49:29.524 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Ga2hJeGJSlEQSz+NXYCpNxN+XSU= 2022-09-09T22:49:29.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:49:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Ga2hJeGJSlEQSz+NXYCpNxN+XSU=', 'Content-Length': '0'} 2022-09-09T22:49:29.543 DEBUG:boto:Response headers: [('x-amz-version-id', '8Z5c4vwfWvpNpRLMwFGV-tguoijURW5'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000431fc52329113090-00631bc2f9-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:49:29.544 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jkcgbj-21 2022-09-09T22:49:29.545 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:49:29.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:49:29.546 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:49:29.601 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:49:29.602 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:49:29.674 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:49:29.668+0000 7fc57439adc0 1 waiting to reach incremental sync.. 2022-09-09T22:49:34.675 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:49:34.672+0000 7fc57439adc0 1 waiting to reach incremental sync.. 2022-09-09T22:49:39.675 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:49:39.668+0000 7fc57439adc0 1 waiting to reach incremental sync.. 2022-09-09T22:49:44.679 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:49:44.672+0000 7fc57439adc0 1 bucket sync caught up with source: 2022-09-09T22:49:44.680 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000003.14.11, , ] 2022-09-09T22:49:44.680 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000003.14.11, , ] 2022-09-09T22:49:44.680 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:49:44.672+0000 7fc57439adc0 0 bucket checkpoint complete 2022-09-09T22:49:44.708 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jkcgbj-21 2022-09-09T22:49:44.709 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:49:44.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:49:44.710 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:49:44.765 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:49:44.766 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:49:44.830 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:49:44.828+0000 7f311fb6ddc0 1 bucket sync caught up with source: 2022-09-09T22:49:44.831 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000003.14.11, , ] 2022-09-09T22:49:44.831 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000003.14.11, , ] 2022-09-09T22:49:44.831 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:49:44.828+0000 7f311fb6ddc0 0 bucket checkpoint complete 2022-09-09T22:49:44.855 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jkcgbj-21 2022-09-09T22:49:44.855 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:49:44.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:49:44.857 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:49:44.910 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:49:44.911 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:49:44.980 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:49:44.976+0000 7f0d2389ddc0 1 waiting to reach incremental sync.. 2022-09-09T22:49:49.985 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:49:49.980+0000 7f0d2389ddc0 1 bucket sync caught up with source: 2022-09-09T22:49:49.985 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000003.14.11, , ] 2022-09-09T22:49:49.985 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000003.14.11, , ] 2022-09-09T22:49:49.986 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:49:49.980+0000 7f0d2389ddc0 0 bucket checkpoint complete 2022-09-09T22:49:50.017 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jkcgbj-21 2022-09-09T22:49:50.018 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:49:50.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:49:50.020 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:49:50.073 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:49:50.073 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:49:50.135 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:49:50.132+0000 7f679e250dc0 1 bucket sync caught up with source: 2022-09-09T22:49:50.136 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000002.62.5, , ] 2022-09-09T22:49:50.136 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000002.62.5, , ] 2022-09-09T22:49:50.137 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:49:50.132+0000 7f679e250dc0 0 bucket checkpoint complete 2022-09-09T22:49:50.164 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jkcgbj-21 2022-09-09T22:49:50.165 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:49:50.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:49:50.167 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:49:50.220 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:49:50.221 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:49:50.285 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:49:50.280+0000 7f69c44cedc0 1 waiting to reach incremental sync.. 2022-09-09T22:49:55.286 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:49:55.283+0000 7f69c44cedc0 1 waiting to reach incremental sync.. 2022-09-09T22:50:00.290 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:50:00.283+0000 7f69c44cedc0 1 bucket sync caught up with source: 2022-09-09T22:50:00.291 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000002.62.5, , ] 2022-09-09T22:50:00.291 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000002.62.5, , ] 2022-09-09T22:50:00.292 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:50:00.283+0000 7f69c44cedc0 0 bucket checkpoint complete 2022-09-09T22:50:00.317 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jkcgbj-21 2022-09-09T22:50:00.317 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jkcgbj-21 2022-09-09T22:50:00.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:00.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:00.319 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:50:00.373 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:50:00.373 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:50:00.439 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:50:00.435+0000 7fae168d8dc0 1 bucket sync caught up with source: 2022-09-09T22:50:00.439 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000002.73.5, , ] 2022-09-09T22:50:00.440 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000002.73.5, , ] 2022-09-09T22:50:00.440 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:50:00.435+0000 7fae168d8dc0 0 bucket checkpoint complete 2022-09-09T22:50:00.466 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jkcgbj-21 2022-09-09T22:50:00.467 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:00.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:00.468 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:50:00.521 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:50:00.522 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:50:00.600 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:50:00.596+0000 7f55caaa9dc0 1 waiting to reach incremental sync.. 2022-09-09T22:50:05.601 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:50:05.596+0000 7f55caaa9dc0 1 waiting to reach incremental sync.. 2022-09-09T22:50:10.602 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:50:10.596+0000 7f55caaa9dc0 1 waiting to reach incremental sync.. 2022-09-09T22:50:15.603 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:50:15.595+0000 7f55caaa9dc0 1 waiting to reach incremental sync.. 2022-09-09T22:50:20.606 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:50:20.599+0000 7f55caaa9dc0 1 bucket sync caught up with source: 2022-09-09T22:50:20.607 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000002.73.5, , ] 2022-09-09T22:50:20.607 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000002.73.5, , ] 2022-09-09T22:50:20.607 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:50:20.599+0000 7f55caaa9dc0 0 bucket checkpoint complete 2022-09-09T22:50:20.639 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jkcgbj-21 2022-09-09T22:50:20.640 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-21 zones={test-zone2, test-zone1} 2022-09-09T22:50:20.641 DEBUG:boto:path=/jkcgbj-21/ 2022-09-09T22:50:20.643 DEBUG:boto:auth_path=/jkcgbj-21/ 2022-09-09T22:50:20.644 DEBUG:boto:Method: HEAD 2022-09-09T22:50:20.645 DEBUG:boto:Path: /jkcgbj-21/ 2022-09-09T22:50:20.646 DEBUG:boto:Data: 2022-09-09T22:50:20.647 DEBUG:boto:Headers: {} 2022-09-09T22:50:20.648 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:20.649 DEBUG:boto:Port: 8000 2022-09-09T22:50:20.650 DEBUG:boto:Params: {} 2022-09-09T22:50:20.651 DEBUG:boto:Token: None 2022-09-09T22:50:20.652 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:20 GMT /jkcgbj-21/ 2022-09-09T22:50:20.653 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vCh6ZAr+w2XEt15A8U1z4dkXeO0= 2022-09-09T22:50:20.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vCh6ZAr+w2XEt15A8U1z4dkXeO0=', 'Content-Length': '0'} 2022-09-09T22:50:20.659 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8f369dda58c5123-00631bc32c-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:20.660 DEBUG:boto:path=/jkcgbj-21/ 2022-09-09T22:50:20.660 DEBUG:boto:auth_path=/jkcgbj-21/ 2022-09-09T22:50:20.661 DEBUG:boto:Method: HEAD 2022-09-09T22:50:20.661 DEBUG:boto:Path: /jkcgbj-21/ 2022-09-09T22:50:20.662 DEBUG:boto:Data: 2022-09-09T22:50:20.662 DEBUG:boto:Headers: {} 2022-09-09T22:50:20.663 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:20.664 DEBUG:boto:Port: 8000 2022-09-09T22:50:20.664 DEBUG:boto:Params: {} 2022-09-09T22:50:20.665 DEBUG:boto:Token: None 2022-09-09T22:50:20.666 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:20 GMT /jkcgbj-21/ 2022-09-09T22:50:20.666 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vCh6ZAr+w2XEt15A8U1z4dkXeO0= 2022-09-09T22:50:20.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vCh6ZAr+w2XEt15A8U1z4dkXeO0=', 'Content-Length': '0'} 2022-09-09T22:50:20.672 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000078eff2a6be895d91-00631bc32c-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:20.673 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:50:20.674 DEBUG:boto:path=/jkcgbj-21/ 2022-09-09T22:50:20.674 DEBUG:boto:auth_path=/jkcgbj-21/ 2022-09-09T22:50:20.675 DEBUG:boto:path=/jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:20.676 DEBUG:boto:auth_path=/jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:20.677 DEBUG:boto:Method: GET 2022-09-09T22:50:20.677 DEBUG:boto:Path: /jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:20.678 DEBUG:boto:Data: 2022-09-09T22:50:20.679 DEBUG:boto:Headers: {} 2022-09-09T22:50:20.680 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:20.681 DEBUG:boto:Port: 8000 2022-09-09T22:50:20.682 DEBUG:boto:Params: {} 2022-09-09T22:50:20.683 DEBUG:boto:Token: None 2022-09-09T22:50:20.684 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:20 GMT /jkcgbj-21/?versions 2022-09-09T22:50:20.685 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Nu4siC2XTqCDO8Wdd6OII3OVdVw= 2022-09-09T22:50:20.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Nu4siC2XTqCDO8Wdd6OII3OVdVw=', 'Content-Length': '0'} 2022-09-09T22:50:20.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9c46458522168e2-00631bc32c-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:20.693 DEBUG:boto:b'jkcgbj-21999falseobjo.3h3n2G6TMQ7YQ1vk1ad3FnvFpVNTgtrue2022-09-09T22:49:29.347Zrgw-multisite-test-userMultisite Test UserNormalobjV-UiCIOWwBdw247KH0Oq0akAJzGvI5Gfalse2022-09-09T22:49:29.264Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:20.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:20.695 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:20.696 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:50:20.696 DEBUG:boto:path=/jkcgbj-21/ 2022-09-09T22:50:20.697 DEBUG:boto:auth_path=/jkcgbj-21/ 2022-09-09T22:50:20.697 DEBUG:boto:path=/jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:20.698 DEBUG:boto:auth_path=/jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:20.699 DEBUG:boto:Method: GET 2022-09-09T22:50:20.699 DEBUG:boto:Path: /jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:20.700 DEBUG:boto:Data: 2022-09-09T22:50:20.700 DEBUG:boto:Headers: {} 2022-09-09T22:50:20.701 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:20.701 DEBUG:boto:Port: 8000 2022-09-09T22:50:20.702 DEBUG:boto:Params: {} 2022-09-09T22:50:20.702 DEBUG:boto:Token: None 2022-09-09T22:50:20.703 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:20 GMT /jkcgbj-21/?versions 2022-09-09T22:50:20.704 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Nu4siC2XTqCDO8Wdd6OII3OVdVw= 2022-09-09T22:50:20.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Nu4siC2XTqCDO8Wdd6OII3OVdVw=', 'Content-Length': '0'} 2022-09-09T22:50:20.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d02dc8279ac3d48-00631bc32c-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:20.754 DEBUG:boto:b'jkcgbj-21999falseobjo.3h3n2G6TMQ7YQ1vk1ad3FnvFpVNTgtrue2022-09-09T22:49:29.347Zrgw-multisite-test-userMultisite Test UserNormalobjV-UiCIOWwBdw247KH0Oq0akAJzGvI5Gfalse2022-09-09T22:49:29.264Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:20.755 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:20.756 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:20.757 DEBUG:boto:path=/jkcgbj-21/ 2022-09-09T22:50:20.757 DEBUG:boto:auth_path=/jkcgbj-21/ 2022-09-09T22:50:20.758 DEBUG:boto:path=/jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:20.758 DEBUG:boto:auth_path=/jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:20.759 DEBUG:boto:Method: GET 2022-09-09T22:50:20.759 DEBUG:boto:Path: /jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:20.760 DEBUG:boto:Data: 2022-09-09T22:50:20.761 DEBUG:boto:Headers: {} 2022-09-09T22:50:20.761 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:20.762 DEBUG:boto:Port: 8000 2022-09-09T22:50:20.763 DEBUG:boto:Params: {} 2022-09-09T22:50:20.764 DEBUG:boto:Token: None 2022-09-09T22:50:20.764 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:20 GMT /jkcgbj-21/?versions 2022-09-09T22:50:20.765 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Nu4siC2XTqCDO8Wdd6OII3OVdVw= 2022-09-09T22:50:20.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Nu4siC2XTqCDO8Wdd6OII3OVdVw=', 'Content-Length': '0'} 2022-09-09T22:50:20.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e6d8238354ba33a-00631bc32c-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:20.772 DEBUG:boto:b'jkcgbj-21999falseobjo.3h3n2G6TMQ7YQ1vk1ad3FnvFpVNTgtrue2022-09-09T22:49:29.347Zrgw-multisite-test-userMultisite Test UserNormalobjV-UiCIOWwBdw247KH0Oq0akAJzGvI5Gfalse2022-09-09T22:49:29.264Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:20.773 DEBUG:boto:path=/jkcgbj-21/ 2022-09-09T22:50:20.774 DEBUG:boto:auth_path=/jkcgbj-21/ 2022-09-09T22:50:20.774 DEBUG:boto:path=/jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:20.775 DEBUG:boto:auth_path=/jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:20.775 DEBUG:boto:Method: GET 2022-09-09T22:50:20.776 DEBUG:boto:Path: /jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:20.776 DEBUG:boto:Data: 2022-09-09T22:50:20.777 DEBUG:boto:Headers: {} 2022-09-09T22:50:20.777 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:20.778 DEBUG:boto:Port: 8000 2022-09-09T22:50:20.778 DEBUG:boto:Params: {} 2022-09-09T22:50:20.779 DEBUG:boto:Token: None 2022-09-09T22:50:20.779 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:20 GMT /jkcgbj-21/?versions 2022-09-09T22:50:20.781 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Nu4siC2XTqCDO8Wdd6OII3OVdVw= 2022-09-09T22:50:20.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Nu4siC2XTqCDO8Wdd6OII3OVdVw=', 'Content-Length': '0'} 2022-09-09T22:50:20.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0a9273c1ffbcdbd-00631bc32c-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:20.787 DEBUG:boto:b'jkcgbj-21999falseobjo.3h3n2G6TMQ7YQ1vk1ad3FnvFpVNTgtrue2022-09-09T22:49:29.347Zrgw-multisite-test-userMultisite Test UserNormalobjV-UiCIOWwBdw247KH0Oq0akAJzGvI5Gfalse2022-09-09T22:49:29.264Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:20.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:50:20.789 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:50:20.789 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:50:20.790 DEBUG:boto:Method: HEAD 2022-09-09T22:50:20.790 DEBUG:boto:Path: /jkcgbj-21/obj 2022-09-09T22:50:20.791 DEBUG:boto:Data: 2022-09-09T22:50:20.791 DEBUG:boto:Headers: {} 2022-09-09T22:50:20.792 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:20.792 DEBUG:boto:Port: 8000 2022-09-09T22:50:20.793 DEBUG:boto:Params: {} 2022-09-09T22:50:20.794 DEBUG:boto:Token: None 2022-09-09T22:50:20.794 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:20 GMT /jkcgbj-21/obj 2022-09-09T22:50:20.795 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:41JIBoG8+a8BAi+kki2MditAzKw= 2022-09-09T22:50:20.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:41JIBoG8+a8BAi+kki2MditAzKw=', 'Content-Length': '0'} 2022-09-09T22:50:20.800 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000007faaac6f3d94b1ed-00631bc32c-10e8-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:20.801 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:50:20.802 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:50:20.803 DEBUG:boto:Method: HEAD 2022-09-09T22:50:20.803 DEBUG:boto:Path: /jkcgbj-21/obj 2022-09-09T22:50:20.804 DEBUG:boto:Data: 2022-09-09T22:50:20.804 DEBUG:boto:Headers: {} 2022-09-09T22:50:20.805 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:20.805 DEBUG:boto:Port: 8000 2022-09-09T22:50:20.806 DEBUG:boto:Params: {} 2022-09-09T22:50:20.806 DEBUG:boto:Token: None 2022-09-09T22:50:20.807 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:20 GMT /jkcgbj-21/obj 2022-09-09T22:50:20.808 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:41JIBoG8+a8BAi+kki2MditAzKw= 2022-09-09T22:50:20.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:41JIBoG8+a8BAi+kki2MditAzKw=', 'Content-Length': '0'} 2022-09-09T22:50:20.812 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000379c577e13f780c2-00631bc32c-10ff-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:20.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:50:20.814 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:50:20.815 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:50:20.815 DEBUG:boto:path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:20.816 DEBUG:boto:auth_path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:20.816 DEBUG:boto:Method: GET 2022-09-09T22:50:20.817 DEBUG:boto:Path: /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:20.817 DEBUG:boto:Data: 2022-09-09T22:50:20.818 DEBUG:boto:Headers: {} 2022-09-09T22:50:20.819 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:20.819 DEBUG:boto:Port: 8000 2022-09-09T22:50:20.820 DEBUG:boto:Params: {} 2022-09-09T22:50:20.821 DEBUG:boto:Token: None 2022-09-09T22:50:20.821 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:20 GMT /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:20.822 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:lBHOxwjCyQInbjs9f6CUzUaqFRg= 2022-09-09T22:50:20.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:lBHOxwjCyQInbjs9f6CUzUaqFRg=', 'Content-Length': '0'} 2022-09-09T22:50:20.826 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'V-UiCIOWwBdw247KH0Oq0akAJzGvI5G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a9df6096e901ec15-00631bc32c-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:20.828 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:50:20.828 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:50:20.829 DEBUG:boto:path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:20.830 DEBUG:boto:auth_path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:20.831 DEBUG:boto:Method: GET 2022-09-09T22:50:20.831 DEBUG:boto:Path: /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:20.832 DEBUG:boto:Data: 2022-09-09T22:50:20.832 DEBUG:boto:Headers: {} 2022-09-09T22:50:20.833 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:20.833 DEBUG:boto:Port: 8000 2022-09-09T22:50:20.834 DEBUG:boto:Params: {} 2022-09-09T22:50:20.835 DEBUG:boto:Token: None 2022-09-09T22:50:20.835 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:20 GMT /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:20.836 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:lBHOxwjCyQInbjs9f6CUzUaqFRg= 2022-09-09T22:50:20.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:lBHOxwjCyQInbjs9f6CUzUaqFRg=', 'Content-Length': '0'} 2022-09-09T22:50:20.840 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'V-UiCIOWwBdw247KH0Oq0akAJzGvI5G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000083ce15ff0ed62b73-00631bc32c-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:20.841 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:50:20.842 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:50:20.843 DEBUG:boto:path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:20.843 DEBUG:boto:auth_path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:20.844 DEBUG:boto:Method: HEAD 2022-09-09T22:50:20.844 DEBUG:boto:Path: /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:20.845 DEBUG:boto:Data: 2022-09-09T22:50:20.845 DEBUG:boto:Headers: {} 2022-09-09T22:50:20.846 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:20.847 DEBUG:boto:Port: 8000 2022-09-09T22:50:20.847 DEBUG:boto:Params: {} 2022-09-09T22:50:20.848 DEBUG:boto:Token: None 2022-09-09T22:50:20.849 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:20 GMT /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:20.850 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rMQlbT7GuKdSh1oLBcPpDBhMYWg= 2022-09-09T22:50:20.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rMQlbT7GuKdSh1oLBcPpDBhMYWg=', 'Content-Length': '0'} 2022-09-09T22:50:20.855 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'V-UiCIOWwBdw247KH0Oq0akAJzGvI5G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000448097b71acfccc2-00631bc32c-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:20.856 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:50:20.857 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:50:20.857 DEBUG:boto:path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:20.858 DEBUG:boto:auth_path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:20.858 DEBUG:boto:Method: HEAD 2022-09-09T22:50:20.859 DEBUG:boto:Path: /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:20.859 DEBUG:boto:Data: 2022-09-09T22:50:20.860 DEBUG:boto:Headers: {} 2022-09-09T22:50:20.860 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:20.861 DEBUG:boto:Port: 8000 2022-09-09T22:50:20.861 DEBUG:boto:Params: {} 2022-09-09T22:50:20.862 DEBUG:boto:Token: None 2022-09-09T22:50:20.862 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:20 GMT /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:20.863 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rMQlbT7GuKdSh1oLBcPpDBhMYWg= 2022-09-09T22:50:20.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rMQlbT7GuKdSh1oLBcPpDBhMYWg=', 'Content-Length': '0'} 2022-09-09T22:50:20.866 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'V-UiCIOWwBdw247KH0Oq0akAJzGvI5G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005850bd2958244bae-00631bc32c-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:20.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:50:20.868 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:50:20.869 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:50:20.870 DEBUG:boto:path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:20.871 DEBUG:boto:auth_path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:20.872 DEBUG:boto:Method: GET 2022-09-09T22:50:20.873 DEBUG:boto:Path: /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:20.874 DEBUG:boto:Data: 2022-09-09T22:50:20.875 DEBUG:boto:Headers: {} 2022-09-09T22:50:20.876 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:20.877 DEBUG:boto:Port: 8000 2022-09-09T22:50:20.878 DEBUG:boto:Params: {} 2022-09-09T22:50:20.879 DEBUG:boto:Token: None 2022-09-09T22:50:20.880 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:20 GMT /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:20.881 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:lBHOxwjCyQInbjs9f6CUzUaqFRg= 2022-09-09T22:50:20.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:lBHOxwjCyQInbjs9f6CUzUaqFRg=', 'Content-Length': '0'} 2022-09-09T22:50:20.885 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'V-UiCIOWwBdw247KH0Oq0akAJzGvI5G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004b74af6a857f8847-00631bc32c-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:20.886 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:50:20.886 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:50:20.887 DEBUG:boto:path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:20.888 DEBUG:boto:auth_path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:20.888 DEBUG:boto:Method: GET 2022-09-09T22:50:20.889 DEBUG:boto:Path: /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:20.889 DEBUG:boto:Data: 2022-09-09T22:50:20.890 DEBUG:boto:Headers: {} 2022-09-09T22:50:20.900 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:20.901 DEBUG:boto:Port: 8000 2022-09-09T22:50:20.902 DEBUG:boto:Params: {} 2022-09-09T22:50:20.903 DEBUG:boto:Token: None 2022-09-09T22:50:20.904 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:20 GMT /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:20.906 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:lBHOxwjCyQInbjs9f6CUzUaqFRg= 2022-09-09T22:50:20.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:lBHOxwjCyQInbjs9f6CUzUaqFRg=', 'Content-Length': '0'} 2022-09-09T22:50:20.911 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'V-UiCIOWwBdw247KH0Oq0akAJzGvI5G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000079240331f20d46f4-00631bc32c-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:20.912 DEBUG:boto:path=/jkcgbj-21/ 2022-09-09T22:50:20.913 DEBUG:boto:auth_path=/jkcgbj-21/ 2022-09-09T22:50:20.914 DEBUG:boto:path=/jkcgbj-21/?prefix=obj 2022-09-09T22:50:20.914 DEBUG:boto:auth_path=/jkcgbj-21/?prefix=obj 2022-09-09T22:50:20.915 DEBUG:boto:Method: GET 2022-09-09T22:50:20.916 DEBUG:boto:Path: /jkcgbj-21/?prefix=obj 2022-09-09T22:50:20.917 DEBUG:boto:Data: 2022-09-09T22:50:20.917 DEBUG:boto:Headers: {} 2022-09-09T22:50:20.918 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:20.919 DEBUG:boto:Port: 8000 2022-09-09T22:50:20.919 DEBUG:boto:Params: {} 2022-09-09T22:50:20.920 DEBUG:boto:Token: None 2022-09-09T22:50:20.921 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:20 GMT /jkcgbj-21/ 2022-09-09T22:50:20.921 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jHTuVvWI+8Kmg9VnakFLi3XPHVM= 2022-09-09T22:50:20.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jHTuVvWI+8Kmg9VnakFLi3XPHVM=', 'Content-Length': '0'} 2022-09-09T22:50:20.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d0b14ba50d02f52-00631bc32c-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:20.970 DEBUG:boto:b'jkcgbj-21obj1000false' 2022-09-09T22:50:20.972 DEBUG:boto:path=/jkcgbj-21/ 2022-09-09T22:50:20.972 DEBUG:boto:auth_path=/jkcgbj-21/ 2022-09-09T22:50:20.973 DEBUG:boto:path=/jkcgbj-21/?prefix=obj 2022-09-09T22:50:20.973 DEBUG:boto:auth_path=/jkcgbj-21/?prefix=obj 2022-09-09T22:50:20.974 DEBUG:boto:Method: GET 2022-09-09T22:50:20.974 DEBUG:boto:Path: /jkcgbj-21/?prefix=obj 2022-09-09T22:50:20.975 DEBUG:boto:Data: 2022-09-09T22:50:20.976 DEBUG:boto:Headers: {} 2022-09-09T22:50:20.976 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:20.977 DEBUG:boto:Port: 8000 2022-09-09T22:50:20.978 DEBUG:boto:Params: {} 2022-09-09T22:50:20.978 DEBUG:boto:Token: None 2022-09-09T22:50:20.979 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:20 GMT /jkcgbj-21/ 2022-09-09T22:50:20.980 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jHTuVvWI+8Kmg9VnakFLi3XPHVM= 2022-09-09T22:50:20.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jHTuVvWI+8Kmg9VnakFLi3XPHVM=', 'Content-Length': '0'} 2022-09-09T22:50:20.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c73cde5fbd6682f5-00631bc32c-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:20.988 DEBUG:boto:b'jkcgbj-21obj1000false' 2022-09-09T22:50:20.989 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:50:20.990 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:50:20.991 DEBUG:boto:Method: HEAD 2022-09-09T22:50:20.992 DEBUG:boto:Path: /jkcgbj-21/obj 2022-09-09T22:50:20.993 DEBUG:boto:Data: 2022-09-09T22:50:20.994 DEBUG:boto:Headers: {} 2022-09-09T22:50:20.994 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:20.995 DEBUG:boto:Port: 8000 2022-09-09T22:50:20.995 DEBUG:boto:Params: {} 2022-09-09T22:50:20.996 DEBUG:boto:Token: None 2022-09-09T22:50:20.996 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:20 GMT /jkcgbj-21/obj 2022-09-09T22:50:20.997 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:41JIBoG8+a8BAi+kki2MditAzKw= 2022-09-09T22:50:20.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:41JIBoG8+a8BAi+kki2MditAzKw=', 'Content-Length': '0'} 2022-09-09T22:50:21.002 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000050ac579402a36620-00631bc32c-10e8-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.003 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:50:21.004 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:50:21.005 DEBUG:boto:Method: HEAD 2022-09-09T22:50:21.005 DEBUG:boto:Path: /jkcgbj-21/obj 2022-09-09T22:50:21.006 DEBUG:boto:Data: 2022-09-09T22:50:21.007 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.007 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:21.008 DEBUG:boto:Port: 8000 2022-09-09T22:50:21.009 DEBUG:boto:Params: {} 2022-09-09T22:50:21.010 DEBUG:boto:Token: None 2022-09-09T22:50:21.011 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/obj 2022-09-09T22:50:21.012 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1doSerDBkIfPNMakKdXDLs5N4/s= 2022-09-09T22:50:21.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1doSerDBkIfPNMakKdXDLs5N4/s=', 'Content-Length': '0'} 2022-09-09T22:50:21.017 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000fd4d1610aa96e9df-00631bc32d-10ff-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.018 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-21 zones={test-zone2, test-zone1} 2022-09-09T22:50:21.019 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-21 zones={test-zone3, test-zone1} 2022-09-09T22:50:21.020 DEBUG:boto:path=/jkcgbj-21/ 2022-09-09T22:50:21.020 DEBUG:boto:auth_path=/jkcgbj-21/ 2022-09-09T22:50:21.021 DEBUG:boto:Method: HEAD 2022-09-09T22:50:21.021 DEBUG:boto:Path: /jkcgbj-21/ 2022-09-09T22:50:21.022 DEBUG:boto:Data: 2022-09-09T22:50:21.022 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.023 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:21.023 DEBUG:boto:Port: 8001 2022-09-09T22:50:21.024 DEBUG:boto:Params: {} 2022-09-09T22:50:21.024 DEBUG:boto:Token: None 2022-09-09T22:50:21.025 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/ 2022-09-09T22:50:21.026 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:yFBmUKeGzyd00BI+yBYixHmBcC4= 2022-09-09T22:50:21.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:yFBmUKeGzyd00BI+yBYixHmBcC4=', 'Content-Length': '0'} 2022-09-09T22:50:21.038 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002fbc87e44609e9d0-00631bc32d-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.039 DEBUG:boto:path=/jkcgbj-21/ 2022-09-09T22:50:21.040 DEBUG:boto:auth_path=/jkcgbj-21/ 2022-09-09T22:50:21.040 DEBUG:boto:Method: HEAD 2022-09-09T22:50:21.041 DEBUG:boto:Path: /jkcgbj-21/ 2022-09-09T22:50:21.042 DEBUG:boto:Data: 2022-09-09T22:50:21.042 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.043 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:21.044 DEBUG:boto:Port: 8000 2022-09-09T22:50:21.045 DEBUG:boto:Params: {} 2022-09-09T22:50:21.046 DEBUG:boto:Token: None 2022-09-09T22:50:21.046 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/ 2022-09-09T22:50:21.047 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:yFBmUKeGzyd00BI+yBYixHmBcC4= 2022-09-09T22:50:21.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:yFBmUKeGzyd00BI+yBYixHmBcC4=', 'Content-Length': '0'} 2022-09-09T22:50:21.054 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001aa2c614fe6a5e0a-00631bc32d-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.054 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:50:21.055 DEBUG:boto:path=/jkcgbj-21/ 2022-09-09T22:50:21.055 DEBUG:boto:auth_path=/jkcgbj-21/ 2022-09-09T22:50:21.056 DEBUG:boto:path=/jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:21.056 DEBUG:boto:auth_path=/jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:21.057 DEBUG:boto:Method: GET 2022-09-09T22:50:21.057 DEBUG:boto:Path: /jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:21.058 DEBUG:boto:Data: 2022-09-09T22:50:21.058 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.059 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:21.059 DEBUG:boto:Port: 8001 2022-09-09T22:50:21.060 DEBUG:boto:Params: {} 2022-09-09T22:50:21.061 DEBUG:boto:Token: None 2022-09-09T22:50:21.061 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/?versions 2022-09-09T22:50:21.062 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7JfgDkdoErtZtvjdI3C5wI6sH40= 2022-09-09T22:50:21.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7JfgDkdoErtZtvjdI3C5wI6sH40=', 'Content-Length': '0'} 2022-09-09T22:50:21.069 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006822611831a10bfe-00631bc32d-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.110 DEBUG:boto:b'jkcgbj-21999falseobjo.3h3n2G6TMQ7YQ1vk1ad3FnvFpVNTgtrue2022-09-09T22:49:29.347Zrgw-multisite-test-userMultisite Test UserNormalobjV-UiCIOWwBdw247KH0Oq0akAJzGvI5Gfalse2022-09-09T22:49:29.264Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:21.111 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:21.112 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:21.112 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:50:21.113 DEBUG:boto:path=/jkcgbj-21/ 2022-09-09T22:50:21.114 DEBUG:boto:auth_path=/jkcgbj-21/ 2022-09-09T22:50:21.115 DEBUG:boto:path=/jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:21.115 DEBUG:boto:auth_path=/jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:21.116 DEBUG:boto:Method: GET 2022-09-09T22:50:21.117 DEBUG:boto:Path: /jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:21.118 DEBUG:boto:Data: 2022-09-09T22:50:21.119 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.120 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:21.120 DEBUG:boto:Port: 8000 2022-09-09T22:50:21.121 DEBUG:boto:Params: {} 2022-09-09T22:50:21.122 DEBUG:boto:Token: None 2022-09-09T22:50:21.122 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/?versions 2022-09-09T22:50:21.123 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7JfgDkdoErtZtvjdI3C5wI6sH40= 2022-09-09T22:50:21.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7JfgDkdoErtZtvjdI3C5wI6sH40=', 'Content-Length': '0'} 2022-09-09T22:50:21.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d89d277f08a6d663-00631bc32d-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.132 DEBUG:boto:b'jkcgbj-21999falseobjo.3h3n2G6TMQ7YQ1vk1ad3FnvFpVNTgtrue2022-09-09T22:49:29.347Zrgw-multisite-test-userMultisite Test UserNormalobjV-UiCIOWwBdw247KH0Oq0akAJzGvI5Gfalse2022-09-09T22:49:29.264Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:21.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:21.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:21.134 DEBUG:boto:path=/jkcgbj-21/ 2022-09-09T22:50:21.135 DEBUG:boto:auth_path=/jkcgbj-21/ 2022-09-09T22:50:21.135 DEBUG:boto:path=/jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:21.136 DEBUG:boto:auth_path=/jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:21.136 DEBUG:boto:Method: GET 2022-09-09T22:50:21.137 DEBUG:boto:Path: /jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:21.137 DEBUG:boto:Data: 2022-09-09T22:50:21.138 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.138 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:21.139 DEBUG:boto:Port: 8001 2022-09-09T22:50:21.139 DEBUG:boto:Params: {} 2022-09-09T22:50:21.140 DEBUG:boto:Token: None 2022-09-09T22:50:21.141 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/?versions 2022-09-09T22:50:21.141 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7JfgDkdoErtZtvjdI3C5wI6sH40= 2022-09-09T22:50:21.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7JfgDkdoErtZtvjdI3C5wI6sH40=', 'Content-Length': '0'} 2022-09-09T22:50:21.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e421b9433396786-00631bc32d-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.148 DEBUG:boto:b'jkcgbj-21999falseobjo.3h3n2G6TMQ7YQ1vk1ad3FnvFpVNTgtrue2022-09-09T22:49:29.347Zrgw-multisite-test-userMultisite Test UserNormalobjV-UiCIOWwBdw247KH0Oq0akAJzGvI5Gfalse2022-09-09T22:49:29.264Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:21.149 DEBUG:boto:path=/jkcgbj-21/ 2022-09-09T22:50:21.149 DEBUG:boto:auth_path=/jkcgbj-21/ 2022-09-09T22:50:21.150 DEBUG:boto:path=/jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:21.151 DEBUG:boto:auth_path=/jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:21.151 DEBUG:boto:Method: GET 2022-09-09T22:50:21.152 DEBUG:boto:Path: /jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:21.153 DEBUG:boto:Data: 2022-09-09T22:50:21.154 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.155 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:21.156 DEBUG:boto:Port: 8000 2022-09-09T22:50:21.157 DEBUG:boto:Params: {} 2022-09-09T22:50:21.158 DEBUG:boto:Token: None 2022-09-09T22:50:21.159 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/?versions 2022-09-09T22:50:21.160 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7JfgDkdoErtZtvjdI3C5wI6sH40= 2022-09-09T22:50:21.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7JfgDkdoErtZtvjdI3C5wI6sH40=', 'Content-Length': '0'} 2022-09-09T22:50:21.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcf6d97330a46907-00631bc32d-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.169 DEBUG:boto:b'jkcgbj-21999falseobjo.3h3n2G6TMQ7YQ1vk1ad3FnvFpVNTgtrue2022-09-09T22:49:29.347Zrgw-multisite-test-userMultisite Test UserNormalobjV-UiCIOWwBdw247KH0Oq0akAJzGvI5Gfalse2022-09-09T22:49:29.264Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:21.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:50:21.171 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:50:21.172 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:50:21.172 DEBUG:boto:Method: HEAD 2022-09-09T22:50:21.173 DEBUG:boto:Path: /jkcgbj-21/obj 2022-09-09T22:50:21.174 DEBUG:boto:Data: 2022-09-09T22:50:21.175 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.176 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:21.177 DEBUG:boto:Port: 8001 2022-09-09T22:50:21.178 DEBUG:boto:Params: {} 2022-09-09T22:50:21.179 DEBUG:boto:Token: None 2022-09-09T22:50:21.180 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/obj 2022-09-09T22:50:21.181 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1doSerDBkIfPNMakKdXDLs5N4/s= 2022-09-09T22:50:21.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1doSerDBkIfPNMakKdXDLs5N4/s=', 'Content-Length': '0'} 2022-09-09T22:50:21.186 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000d1194ff1653d1747-00631bc32d-1103-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.186 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:50:21.187 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:50:21.187 DEBUG:boto:Method: HEAD 2022-09-09T22:50:21.188 DEBUG:boto:Path: /jkcgbj-21/obj 2022-09-09T22:50:21.188 DEBUG:boto:Data: 2022-09-09T22:50:21.189 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.189 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:21.190 DEBUG:boto:Port: 8000 2022-09-09T22:50:21.190 DEBUG:boto:Params: {} 2022-09-09T22:50:21.191 DEBUG:boto:Token: None 2022-09-09T22:50:21.192 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/obj 2022-09-09T22:50:21.192 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1doSerDBkIfPNMakKdXDLs5N4/s= 2022-09-09T22:50:21.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1doSerDBkIfPNMakKdXDLs5N4/s=', 'Content-Length': '0'} 2022-09-09T22:50:21.198 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000001a05d3cf8112c4d2-00631bc32d-10ff-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:50:21.200 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:50:21.201 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:50:21.202 DEBUG:boto:path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.203 DEBUG:boto:auth_path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.204 DEBUG:boto:Method: GET 2022-09-09T22:50:21.205 DEBUG:boto:Path: /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.206 DEBUG:boto:Data: 2022-09-09T22:50:21.207 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.208 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:21.209 DEBUG:boto:Port: 8001 2022-09-09T22:50:21.210 DEBUG:boto:Params: {} 2022-09-09T22:50:21.211 DEBUG:boto:Token: None 2022-09-09T22:50:21.212 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.212 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+F5h3UfSIgglTZi1UV/OUa+VkTU= 2022-09-09T22:50:21.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+F5h3UfSIgglTZi1UV/OUa+VkTU=', 'Content-Length': '0'} 2022-09-09T22:50:21.217 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'V-UiCIOWwBdw247KH0Oq0akAJzGvI5G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002d90a86431fc8af0-00631bc32d-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.218 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:50:21.219 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:50:21.219 DEBUG:boto:path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.220 DEBUG:boto:auth_path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.220 DEBUG:boto:Method: GET 2022-09-09T22:50:21.221 DEBUG:boto:Path: /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.221 DEBUG:boto:Data: 2022-09-09T22:50:21.222 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.223 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:21.223 DEBUG:boto:Port: 8000 2022-09-09T22:50:21.224 DEBUG:boto:Params: {} 2022-09-09T22:50:21.224 DEBUG:boto:Token: None 2022-09-09T22:50:21.225 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.226 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+F5h3UfSIgglTZi1UV/OUa+VkTU= 2022-09-09T22:50:21.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+F5h3UfSIgglTZi1UV/OUa+VkTU=', 'Content-Length': '0'} 2022-09-09T22:50:21.230 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'V-UiCIOWwBdw247KH0Oq0akAJzGvI5G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000116c145e57aa9256-00631bc32d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.232 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:50:21.233 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:50:21.233 DEBUG:boto:path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.234 DEBUG:boto:auth_path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.235 DEBUG:boto:Method: HEAD 2022-09-09T22:50:21.236 DEBUG:boto:Path: /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.237 DEBUG:boto:Data: 2022-09-09T22:50:21.238 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.238 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:21.239 DEBUG:boto:Port: 8001 2022-09-09T22:50:21.240 DEBUG:boto:Params: {} 2022-09-09T22:50:21.240 DEBUG:boto:Token: None 2022-09-09T22:50:21.241 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.242 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2sEznL9fjetJ0DKfnI9nJ3InJW0= 2022-09-09T22:50:21.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2sEznL9fjetJ0DKfnI9nJ3InJW0=', 'Content-Length': '0'} 2022-09-09T22:50:21.246 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'V-UiCIOWwBdw247KH0Oq0akAJzGvI5G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005f69e1a5c3149473-00631bc32d-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.246 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:50:21.247 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:50:21.248 DEBUG:boto:path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.248 DEBUG:boto:auth_path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.249 DEBUG:boto:Method: HEAD 2022-09-09T22:50:21.249 DEBUG:boto:Path: /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.250 DEBUG:boto:Data: 2022-09-09T22:50:21.250 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.251 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:21.251 DEBUG:boto:Port: 8000 2022-09-09T22:50:21.252 DEBUG:boto:Params: {} 2022-09-09T22:50:21.253 DEBUG:boto:Token: None 2022-09-09T22:50:21.253 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.254 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2sEznL9fjetJ0DKfnI9nJ3InJW0= 2022-09-09T22:50:21.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2sEznL9fjetJ0DKfnI9nJ3InJW0=', 'Content-Length': '0'} 2022-09-09T22:50:21.258 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'V-UiCIOWwBdw247KH0Oq0akAJzGvI5G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000edf0bdaef8ab27c9-00631bc32d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:50:21.259 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:50:21.260 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:50:21.260 DEBUG:boto:path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.261 DEBUG:boto:auth_path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.262 DEBUG:boto:Method: GET 2022-09-09T22:50:21.263 DEBUG:boto:Path: /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.263 DEBUG:boto:Data: 2022-09-09T22:50:21.264 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.264 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:21.265 DEBUG:boto:Port: 8001 2022-09-09T22:50:21.266 DEBUG:boto:Params: {} 2022-09-09T22:50:21.267 DEBUG:boto:Token: None 2022-09-09T22:50:21.268 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.269 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+F5h3UfSIgglTZi1UV/OUa+VkTU= 2022-09-09T22:50:21.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+F5h3UfSIgglTZi1UV/OUa+VkTU=', 'Content-Length': '0'} 2022-09-09T22:50:21.274 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'V-UiCIOWwBdw247KH0Oq0akAJzGvI5G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000262f581b43c5f344-00631bc32d-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.275 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:50:21.275 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:50:21.276 DEBUG:boto:path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.277 DEBUG:boto:auth_path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.277 DEBUG:boto:Method: GET 2022-09-09T22:50:21.278 DEBUG:boto:Path: /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.278 DEBUG:boto:Data: 2022-09-09T22:50:21.279 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.279 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:21.280 DEBUG:boto:Port: 8000 2022-09-09T22:50:21.281 DEBUG:boto:Params: {} 2022-09-09T22:50:21.282 DEBUG:boto:Token: None 2022-09-09T22:50:21.283 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.284 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+F5h3UfSIgglTZi1UV/OUa+VkTU= 2022-09-09T22:50:21.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+F5h3UfSIgglTZi1UV/OUa+VkTU=', 'Content-Length': '0'} 2022-09-09T22:50:21.290 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'V-UiCIOWwBdw247KH0Oq0akAJzGvI5G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fbe70540087b1905-00631bc32d-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.292 DEBUG:boto:path=/jkcgbj-21/ 2022-09-09T22:50:21.293 DEBUG:boto:auth_path=/jkcgbj-21/ 2022-09-09T22:50:21.294 DEBUG:boto:path=/jkcgbj-21/?prefix=obj 2022-09-09T22:50:21.295 DEBUG:boto:auth_path=/jkcgbj-21/?prefix=obj 2022-09-09T22:50:21.296 DEBUG:boto:Method: GET 2022-09-09T22:50:21.297 DEBUG:boto:Path: /jkcgbj-21/?prefix=obj 2022-09-09T22:50:21.298 DEBUG:boto:Data: 2022-09-09T22:50:21.299 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.300 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:21.301 DEBUG:boto:Port: 8001 2022-09-09T22:50:21.302 DEBUG:boto:Params: {} 2022-09-09T22:50:21.304 DEBUG:boto:Token: None 2022-09-09T22:50:21.305 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/ 2022-09-09T22:50:21.306 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:yiRdXe0SS8TRRkbXat3tFqa6Qvw= 2022-09-09T22:50:21.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:yiRdXe0SS8TRRkbXat3tFqa6Qvw=', 'Content-Length': '0'} 2022-09-09T22:50:21.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000993274b09a4e7533-00631bc32d-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.316 DEBUG:boto:b'jkcgbj-21obj1000false' 2022-09-09T22:50:21.317 DEBUG:boto:path=/jkcgbj-21/ 2022-09-09T22:50:21.318 DEBUG:boto:auth_path=/jkcgbj-21/ 2022-09-09T22:50:21.319 DEBUG:boto:path=/jkcgbj-21/?prefix=obj 2022-09-09T22:50:21.320 DEBUG:boto:auth_path=/jkcgbj-21/?prefix=obj 2022-09-09T22:50:21.321 DEBUG:boto:Method: GET 2022-09-09T22:50:21.322 DEBUG:boto:Path: /jkcgbj-21/?prefix=obj 2022-09-09T22:50:21.323 DEBUG:boto:Data: 2022-09-09T22:50:21.324 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.324 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:21.325 DEBUG:boto:Port: 8000 2022-09-09T22:50:21.326 DEBUG:boto:Params: {} 2022-09-09T22:50:21.327 DEBUG:boto:Token: None 2022-09-09T22:50:21.328 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/ 2022-09-09T22:50:21.328 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:yiRdXe0SS8TRRkbXat3tFqa6Qvw= 2022-09-09T22:50:21.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:yiRdXe0SS8TRRkbXat3tFqa6Qvw=', 'Content-Length': '0'} 2022-09-09T22:50:21.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082826771f4a34cb7-00631bc32d-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.382 DEBUG:boto:b'jkcgbj-21obj1000false' 2022-09-09T22:50:21.383 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:50:21.384 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:50:21.384 DEBUG:boto:Method: HEAD 2022-09-09T22:50:21.385 DEBUG:boto:Path: /jkcgbj-21/obj 2022-09-09T22:50:21.385 DEBUG:boto:Data: 2022-09-09T22:50:21.386 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.386 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:21.387 DEBUG:boto:Port: 8001 2022-09-09T22:50:21.387 DEBUG:boto:Params: {} 2022-09-09T22:50:21.388 DEBUG:boto:Token: None 2022-09-09T22:50:21.389 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/obj 2022-09-09T22:50:21.389 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1doSerDBkIfPNMakKdXDLs5N4/s= 2022-09-09T22:50:21.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1doSerDBkIfPNMakKdXDLs5N4/s=', 'Content-Length': '0'} 2022-09-09T22:50:21.394 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000084c2d8e315512e56-00631bc32d-1103-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.395 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:50:21.396 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:50:21.396 DEBUG:boto:Method: HEAD 2022-09-09T22:50:21.397 DEBUG:boto:Path: /jkcgbj-21/obj 2022-09-09T22:50:21.397 DEBUG:boto:Data: 2022-09-09T22:50:21.398 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.399 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:21.399 DEBUG:boto:Port: 8000 2022-09-09T22:50:21.400 DEBUG:boto:Params: {} 2022-09-09T22:50:21.400 DEBUG:boto:Token: None 2022-09-09T22:50:21.401 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/obj 2022-09-09T22:50:21.401 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1doSerDBkIfPNMakKdXDLs5N4/s= 2022-09-09T22:50:21.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1doSerDBkIfPNMakKdXDLs5N4/s=', 'Content-Length': '0'} 2022-09-09T22:50:21.406 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000418bfc5814994344-00631bc32d-10ff-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.407 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-21 zones={test-zone3, test-zone1} 2022-09-09T22:50:21.407 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-21 zones={test-zone3, test-zone2} 2022-09-09T22:50:21.408 DEBUG:boto:path=/jkcgbj-21/ 2022-09-09T22:50:21.408 DEBUG:boto:auth_path=/jkcgbj-21/ 2022-09-09T22:50:21.409 DEBUG:boto:Method: HEAD 2022-09-09T22:50:21.410 DEBUG:boto:Path: /jkcgbj-21/ 2022-09-09T22:50:21.410 DEBUG:boto:Data: 2022-09-09T22:50:21.411 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.411 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:21.412 DEBUG:boto:Port: 8001 2022-09-09T22:50:21.414 DEBUG:boto:Params: {} 2022-09-09T22:50:21.415 DEBUG:boto:Token: None 2022-09-09T22:50:21.416 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/ 2022-09-09T22:50:21.417 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:yFBmUKeGzyd00BI+yBYixHmBcC4= 2022-09-09T22:50:21.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:yFBmUKeGzyd00BI+yBYixHmBcC4=', 'Content-Length': '0'} 2022-09-09T22:50:21.424 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055d75c608487740a-00631bc32d-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.424 DEBUG:boto:path=/jkcgbj-21/ 2022-09-09T22:50:21.425 DEBUG:boto:auth_path=/jkcgbj-21/ 2022-09-09T22:50:21.425 DEBUG:boto:Method: HEAD 2022-09-09T22:50:21.426 DEBUG:boto:Path: /jkcgbj-21/ 2022-09-09T22:50:21.427 DEBUG:boto:Data: 2022-09-09T22:50:21.427 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.428 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:21.428 DEBUG:boto:Port: 8000 2022-09-09T22:50:21.429 DEBUG:boto:Params: {} 2022-09-09T22:50:21.430 DEBUG:boto:Token: None 2022-09-09T22:50:21.430 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/ 2022-09-09T22:50:21.431 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:yFBmUKeGzyd00BI+yBYixHmBcC4= 2022-09-09T22:50:21.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:yFBmUKeGzyd00BI+yBYixHmBcC4=', 'Content-Length': '0'} 2022-09-09T22:50:21.436 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a1ac8e2a08b4b94-00631bc32d-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.437 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:50:21.438 DEBUG:boto:path=/jkcgbj-21/ 2022-09-09T22:50:21.438 DEBUG:boto:auth_path=/jkcgbj-21/ 2022-09-09T22:50:21.439 DEBUG:boto:path=/jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:21.439 DEBUG:boto:auth_path=/jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:21.440 DEBUG:boto:Method: GET 2022-09-09T22:50:21.440 DEBUG:boto:Path: /jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:21.441 DEBUG:boto:Data: 2022-09-09T22:50:21.441 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.442 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:21.442 DEBUG:boto:Port: 8001 2022-09-09T22:50:21.443 DEBUG:boto:Params: {} 2022-09-09T22:50:21.443 DEBUG:boto:Token: None 2022-09-09T22:50:21.444 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/?versions 2022-09-09T22:50:21.445 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7JfgDkdoErtZtvjdI3C5wI6sH40= 2022-09-09T22:50:21.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7JfgDkdoErtZtvjdI3C5wI6sH40=', 'Content-Length': '0'} 2022-09-09T22:50:21.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ede85ec2adcd84e-00631bc32d-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.494 DEBUG:boto:b'jkcgbj-21999falseobjo.3h3n2G6TMQ7YQ1vk1ad3FnvFpVNTgtrue2022-09-09T22:49:29.347Zrgw-multisite-test-userMultisite Test UserNormalobjV-UiCIOWwBdw247KH0Oq0akAJzGvI5Gfalse2022-09-09T22:49:29.264Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:21.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:21.496 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:21.496 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:50:21.497 DEBUG:boto:path=/jkcgbj-21/ 2022-09-09T22:50:21.498 DEBUG:boto:auth_path=/jkcgbj-21/ 2022-09-09T22:50:21.498 DEBUG:boto:path=/jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:21.499 DEBUG:boto:auth_path=/jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:21.500 DEBUG:boto:Method: GET 2022-09-09T22:50:21.500 DEBUG:boto:Path: /jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:21.501 DEBUG:boto:Data: 2022-09-09T22:50:21.502 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.502 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:21.503 DEBUG:boto:Port: 8000 2022-09-09T22:50:21.503 DEBUG:boto:Params: {} 2022-09-09T22:50:21.504 DEBUG:boto:Token: None 2022-09-09T22:50:21.505 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/?versions 2022-09-09T22:50:21.506 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7JfgDkdoErtZtvjdI3C5wI6sH40= 2022-09-09T22:50:21.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7JfgDkdoErtZtvjdI3C5wI6sH40=', 'Content-Length': '0'} 2022-09-09T22:50:21.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9abe8c103cec604-00631bc32d-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.515 DEBUG:boto:b'jkcgbj-21999falseobjo.3h3n2G6TMQ7YQ1vk1ad3FnvFpVNTgtrue2022-09-09T22:49:29.347Zrgw-multisite-test-userMultisite Test UserNormalobjV-UiCIOWwBdw247KH0Oq0akAJzGvI5Gfalse2022-09-09T22:49:29.264Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:21.516 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:21.517 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:21.517 DEBUG:boto:path=/jkcgbj-21/ 2022-09-09T22:50:21.518 DEBUG:boto:auth_path=/jkcgbj-21/ 2022-09-09T22:50:21.519 DEBUG:boto:path=/jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:21.519 DEBUG:boto:auth_path=/jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:21.520 DEBUG:boto:Method: GET 2022-09-09T22:50:21.520 DEBUG:boto:Path: /jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:21.521 DEBUG:boto:Data: 2022-09-09T22:50:21.521 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.522 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:21.522 DEBUG:boto:Port: 8001 2022-09-09T22:50:21.523 DEBUG:boto:Params: {} 2022-09-09T22:50:21.524 DEBUG:boto:Token: None 2022-09-09T22:50:21.524 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/?versions 2022-09-09T22:50:21.525 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7JfgDkdoErtZtvjdI3C5wI6sH40= 2022-09-09T22:50:21.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7JfgDkdoErtZtvjdI3C5wI6sH40=', 'Content-Length': '0'} 2022-09-09T22:50:21.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041895d01bc4df0c0-00631bc32d-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.533 DEBUG:boto:b'jkcgbj-21999falseobjo.3h3n2G6TMQ7YQ1vk1ad3FnvFpVNTgtrue2022-09-09T22:49:29.347Zrgw-multisite-test-userMultisite Test UserNormalobjV-UiCIOWwBdw247KH0Oq0akAJzGvI5Gfalse2022-09-09T22:49:29.264Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:21.533 DEBUG:boto:path=/jkcgbj-21/ 2022-09-09T22:50:21.534 DEBUG:boto:auth_path=/jkcgbj-21/ 2022-09-09T22:50:21.535 DEBUG:boto:path=/jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:21.535 DEBUG:boto:auth_path=/jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:21.536 DEBUG:boto:Method: GET 2022-09-09T22:50:21.536 DEBUG:boto:Path: /jkcgbj-21/?versions&max-keys=999 2022-09-09T22:50:21.537 DEBUG:boto:Data: 2022-09-09T22:50:21.538 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.538 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:21.539 DEBUG:boto:Port: 8000 2022-09-09T22:50:21.539 DEBUG:boto:Params: {} 2022-09-09T22:50:21.540 DEBUG:boto:Token: None 2022-09-09T22:50:21.540 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/?versions 2022-09-09T22:50:21.541 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7JfgDkdoErtZtvjdI3C5wI6sH40= 2022-09-09T22:50:21.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7JfgDkdoErtZtvjdI3C5wI6sH40=', 'Content-Length': '0'} 2022-09-09T22:50:21.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006353df0d2163cee2-00631bc32d-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.548 DEBUG:boto:b'jkcgbj-21999falseobjo.3h3n2G6TMQ7YQ1vk1ad3FnvFpVNTgtrue2022-09-09T22:49:29.347Zrgw-multisite-test-userMultisite Test UserNormalobjV-UiCIOWwBdw247KH0Oq0akAJzGvI5Gfalse2022-09-09T22:49:29.264Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:21.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:50:21.549 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:50:21.550 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:50:21.550 DEBUG:boto:Method: HEAD 2022-09-09T22:50:21.551 DEBUG:boto:Path: /jkcgbj-21/obj 2022-09-09T22:50:21.551 DEBUG:boto:Data: 2022-09-09T22:50:21.552 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.552 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:21.553 DEBUG:boto:Port: 8001 2022-09-09T22:50:21.553 DEBUG:boto:Params: {} 2022-09-09T22:50:21.554 DEBUG:boto:Token: None 2022-09-09T22:50:21.555 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/obj 2022-09-09T22:50:21.555 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1doSerDBkIfPNMakKdXDLs5N4/s= 2022-09-09T22:50:21.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1doSerDBkIfPNMakKdXDLs5N4/s=', 'Content-Length': '0'} 2022-09-09T22:50:21.560 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000057d52585f5d1cb6c-00631bc32d-1103-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.561 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:50:21.561 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:50:21.562 DEBUG:boto:Method: HEAD 2022-09-09T22:50:21.562 DEBUG:boto:Path: /jkcgbj-21/obj 2022-09-09T22:50:21.563 DEBUG:boto:Data: 2022-09-09T22:50:21.564 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.564 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:21.565 DEBUG:boto:Port: 8000 2022-09-09T22:50:21.565 DEBUG:boto:Params: {} 2022-09-09T22:50:21.566 DEBUG:boto:Token: None 2022-09-09T22:50:21.567 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/obj 2022-09-09T22:50:21.568 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1doSerDBkIfPNMakKdXDLs5N4/s= 2022-09-09T22:50:21.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1doSerDBkIfPNMakKdXDLs5N4/s=', 'Content-Length': '0'} 2022-09-09T22:50:21.574 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000007ba7476657e5de35-00631bc32d-10e8-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:50:21.575 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:50:21.576 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:50:21.576 DEBUG:boto:path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.577 DEBUG:boto:auth_path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.577 DEBUG:boto:Method: GET 2022-09-09T22:50:21.578 DEBUG:boto:Path: /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.578 DEBUG:boto:Data: 2022-09-09T22:50:21.579 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.579 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:21.580 DEBUG:boto:Port: 8001 2022-09-09T22:50:21.580 DEBUG:boto:Params: {} 2022-09-09T22:50:21.581 DEBUG:boto:Token: None 2022-09-09T22:50:21.581 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.582 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+F5h3UfSIgglTZi1UV/OUa+VkTU= 2022-09-09T22:50:21.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+F5h3UfSIgglTZi1UV/OUa+VkTU=', 'Content-Length': '0'} 2022-09-09T22:50:21.586 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'V-UiCIOWwBdw247KH0Oq0akAJzGvI5G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008c92a12a2add6fdf-00631bc32d-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.587 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:50:21.587 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:50:21.588 DEBUG:boto:path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.588 DEBUG:boto:auth_path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.589 DEBUG:boto:Method: GET 2022-09-09T22:50:21.589 DEBUG:boto:Path: /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.590 DEBUG:boto:Data: 2022-09-09T22:50:21.590 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.591 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:21.591 DEBUG:boto:Port: 8000 2022-09-09T22:50:21.592 DEBUG:boto:Params: {} 2022-09-09T22:50:21.593 DEBUG:boto:Token: None 2022-09-09T22:50:21.593 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.594 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+F5h3UfSIgglTZi1UV/OUa+VkTU= 2022-09-09T22:50:21.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+F5h3UfSIgglTZi1UV/OUa+VkTU=', 'Content-Length': '0'} 2022-09-09T22:50:21.598 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'V-UiCIOWwBdw247KH0Oq0akAJzGvI5G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002cddc8c575d9cae8-00631bc32d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.599 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:50:21.600 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:50:21.600 DEBUG:boto:path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.601 DEBUG:boto:auth_path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.601 DEBUG:boto:Method: HEAD 2022-09-09T22:50:21.602 DEBUG:boto:Path: /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.603 DEBUG:boto:Data: 2022-09-09T22:50:21.603 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.604 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:21.604 DEBUG:boto:Port: 8001 2022-09-09T22:50:21.605 DEBUG:boto:Params: {} 2022-09-09T22:50:21.605 DEBUG:boto:Token: None 2022-09-09T22:50:21.606 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.606 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2sEznL9fjetJ0DKfnI9nJ3InJW0= 2022-09-09T22:50:21.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2sEznL9fjetJ0DKfnI9nJ3InJW0=', 'Content-Length': '0'} 2022-09-09T22:50:21.610 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'V-UiCIOWwBdw247KH0Oq0akAJzGvI5G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d9e75100db545d4a-00631bc32d-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.611 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:50:21.612 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:50:21.612 DEBUG:boto:path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.613 DEBUG:boto:auth_path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.613 DEBUG:boto:Method: HEAD 2022-09-09T22:50:21.614 DEBUG:boto:Path: /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.614 DEBUG:boto:Data: 2022-09-09T22:50:21.615 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.615 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:21.616 DEBUG:boto:Port: 8000 2022-09-09T22:50:21.616 DEBUG:boto:Params: {} 2022-09-09T22:50:21.617 DEBUG:boto:Token: None 2022-09-09T22:50:21.617 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.618 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2sEznL9fjetJ0DKfnI9nJ3InJW0= 2022-09-09T22:50:21.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2sEznL9fjetJ0DKfnI9nJ3InJW0=', 'Content-Length': '0'} 2022-09-09T22:50:21.621 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'V-UiCIOWwBdw247KH0Oq0akAJzGvI5G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000001102f7e96fbb8ce-00631bc32d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:50:21.623 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:50:21.623 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:50:21.624 DEBUG:boto:path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.624 DEBUG:boto:auth_path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.625 DEBUG:boto:Method: GET 2022-09-09T22:50:21.625 DEBUG:boto:Path: /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.626 DEBUG:boto:Data: 2022-09-09T22:50:21.627 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.628 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:21.629 DEBUG:boto:Port: 8001 2022-09-09T22:50:21.630 DEBUG:boto:Params: {} 2022-09-09T22:50:21.631 DEBUG:boto:Token: None 2022-09-09T22:50:21.631 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.632 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+F5h3UfSIgglTZi1UV/OUa+VkTU= 2022-09-09T22:50:21.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+F5h3UfSIgglTZi1UV/OUa+VkTU=', 'Content-Length': '0'} 2022-09-09T22:50:21.636 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'V-UiCIOWwBdw247KH0Oq0akAJzGvI5G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e5e0224f245c15c3-00631bc32d-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.638 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:50:21.639 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:50:21.640 DEBUG:boto:path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.640 DEBUG:boto:auth_path=/jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.641 DEBUG:boto:Method: GET 2022-09-09T22:50:21.642 DEBUG:boto:Path: /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.643 DEBUG:boto:Data: 2022-09-09T22:50:21.644 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.645 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:21.646 DEBUG:boto:Port: 8000 2022-09-09T22:50:21.647 DEBUG:boto:Params: {} 2022-09-09T22:50:21.648 DEBUG:boto:Token: None 2022-09-09T22:50:21.649 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/obj?versionId=V-UiCIOWwBdw247KH0Oq0akAJzGvI5G 2022-09-09T22:50:21.650 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+F5h3UfSIgglTZi1UV/OUa+VkTU= 2022-09-09T22:50:21.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+F5h3UfSIgglTZi1UV/OUa+VkTU=', 'Content-Length': '0'} 2022-09-09T22:50:21.654 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'V-UiCIOWwBdw247KH0Oq0akAJzGvI5G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009f54f30e86f15e50-00631bc32d-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.655 DEBUG:boto:path=/jkcgbj-21/ 2022-09-09T22:50:21.655 DEBUG:boto:auth_path=/jkcgbj-21/ 2022-09-09T22:50:21.656 DEBUG:boto:path=/jkcgbj-21/?prefix=obj 2022-09-09T22:50:21.656 DEBUG:boto:auth_path=/jkcgbj-21/?prefix=obj 2022-09-09T22:50:21.657 DEBUG:boto:Method: GET 2022-09-09T22:50:21.657 DEBUG:boto:Path: /jkcgbj-21/?prefix=obj 2022-09-09T22:50:21.658 DEBUG:boto:Data: 2022-09-09T22:50:21.658 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.659 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:21.660 DEBUG:boto:Port: 8001 2022-09-09T22:50:21.660 DEBUG:boto:Params: {} 2022-09-09T22:50:21.661 DEBUG:boto:Token: None 2022-09-09T22:50:21.661 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/ 2022-09-09T22:50:21.662 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:yiRdXe0SS8TRRkbXat3tFqa6Qvw= 2022-09-09T22:50:21.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:yiRdXe0SS8TRRkbXat3tFqa6Qvw=', 'Content-Length': '0'} 2022-09-09T22:50:21.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007685b99730a1ee1c-00631bc32d-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.710 DEBUG:boto:b'jkcgbj-21obj1000false' 2022-09-09T22:50:21.711 DEBUG:boto:path=/jkcgbj-21/ 2022-09-09T22:50:21.712 DEBUG:boto:auth_path=/jkcgbj-21/ 2022-09-09T22:50:21.713 DEBUG:boto:path=/jkcgbj-21/?prefix=obj 2022-09-09T22:50:21.714 DEBUG:boto:auth_path=/jkcgbj-21/?prefix=obj 2022-09-09T22:50:21.715 DEBUG:boto:Method: GET 2022-09-09T22:50:21.716 DEBUG:boto:Path: /jkcgbj-21/?prefix=obj 2022-09-09T22:50:21.717 DEBUG:boto:Data: 2022-09-09T22:50:21.717 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.718 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:21.719 DEBUG:boto:Port: 8000 2022-09-09T22:50:21.721 DEBUG:boto:Params: {} 2022-09-09T22:50:21.722 DEBUG:boto:Token: None 2022-09-09T22:50:21.723 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/ 2022-09-09T22:50:21.724 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:yiRdXe0SS8TRRkbXat3tFqa6Qvw= 2022-09-09T22:50:21.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:yiRdXe0SS8TRRkbXat3tFqa6Qvw=', 'Content-Length': '0'} 2022-09-09T22:50:21.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026614ecde3749a90-00631bc32d-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.731 DEBUG:boto:b'jkcgbj-21obj1000false' 2022-09-09T22:50:21.732 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:50:21.733 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:50:21.733 DEBUG:boto:Method: HEAD 2022-09-09T22:50:21.734 DEBUG:boto:Path: /jkcgbj-21/obj 2022-09-09T22:50:21.734 DEBUG:boto:Data: 2022-09-09T22:50:21.735 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.735 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:21.736 DEBUG:boto:Port: 8001 2022-09-09T22:50:21.736 DEBUG:boto:Params: {} 2022-09-09T22:50:21.737 DEBUG:boto:Token: None 2022-09-09T22:50:21.738 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/obj 2022-09-09T22:50:21.738 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1doSerDBkIfPNMakKdXDLs5N4/s= 2022-09-09T22:50:21.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1doSerDBkIfPNMakKdXDLs5N4/s=', 'Content-Length': '0'} 2022-09-09T22:50:21.744 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000a5eb27314b0a1e8a-00631bc32d-1103-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.745 DEBUG:boto:path=/jkcgbj-21/obj 2022-09-09T22:50:21.745 DEBUG:boto:auth_path=/jkcgbj-21/obj 2022-09-09T22:50:21.746 DEBUG:boto:Method: HEAD 2022-09-09T22:50:21.747 DEBUG:boto:Path: /jkcgbj-21/obj 2022-09-09T22:50:21.747 DEBUG:boto:Data: 2022-09-09T22:50:21.748 DEBUG:boto:Headers: {} 2022-09-09T22:50:21.748 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:21.749 DEBUG:boto:Port: 8000 2022-09-09T22:50:21.749 DEBUG:boto:Params: {} 2022-09-09T22:50:21.750 DEBUG:boto:Token: None 2022-09-09T22:50:21.750 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:21 GMT /jkcgbj-21/obj 2022-09-09T22:50:21.751 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1doSerDBkIfPNMakKdXDLs5N4/s= 2022-09-09T22:50:21.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:21 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1doSerDBkIfPNMakKdXDLs5N4/s=', 'Content-Length': '0'} 2022-09-09T22:50:21.756 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000375024d380039bab-00631bc32d-10e8-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:21.757 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-21 zones={test-zone3, test-zone2} 2022-09-09T22:50:21.758 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jkcgbj-22 2022-09-09T22:50:21.758 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:21.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:21.759 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:50:21.814 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:50:21.814 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:50:21.893 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:50:21.887+0000 7fe73bd19dc0 1 bucket sync caught up with source: 2022-09-09T22:50:21.894 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000002.10.5, , ] 2022-09-09T22:50:21.894 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000002.10.5, , ] 2022-09-09T22:50:21.894 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:50:21.887+0000 7fe73bd19dc0 0 bucket checkpoint complete 2022-09-09T22:50:21.926 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jkcgbj-22 2022-09-09T22:50:21.927 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:21.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:21.929 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:50:21.984 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:50:21.985 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:50:22.054 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:50:22.051+0000 7f6014a3adc0 1 bucket sync caught up with source: 2022-09-09T22:50:22.055 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000002.10.5, , ] 2022-09-09T22:50:22.055 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000002.10.5, , ] 2022-09-09T22:50:22.055 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:50:22.051+0000 7f6014a3adc0 0 bucket checkpoint complete 2022-09-09T22:50:22.080 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jkcgbj-22 2022-09-09T22:50:22.081 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:22.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:22.082 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:50:22.135 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:50:22.136 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:50:22.209 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:50:22.203+0000 7f02060d0dc0 1 bucket sync caught up with source: 2022-09-09T22:50:22.210 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000002.10.5, , ] 2022-09-09T22:50:22.210 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000002.10.5, , ] 2022-09-09T22:50:22.210 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:50:22.203+0000 7f02060d0dc0 0 bucket checkpoint complete 2022-09-09T22:50:22.236 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jkcgbj-22 2022-09-09T22:50:22.237 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:22.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:22.248 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:50:22.302 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:50:22.303 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:50:22.373 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:50:22.367+0000 7f3f6ea14dc0 1 bucket sync caught up with source: 2022-09-09T22:50:22.374 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000003.93.11, , ] 2022-09-09T22:50:22.374 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000003.93.11, , ] 2022-09-09T22:50:22.374 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:50:22.367+0000 7f3f6ea14dc0 0 bucket checkpoint complete 2022-09-09T22:50:22.403 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jkcgbj-22 2022-09-09T22:50:22.404 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:22.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:22.407 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:50:22.459 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:50:22.460 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:50:22.528 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:50:22.523+0000 7f901137fdc0 1 bucket sync caught up with source: 2022-09-09T22:50:22.528 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000003.93.11, , ] 2022-09-09T22:50:22.529 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000003.93.11, , ] 2022-09-09T22:50:22.529 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:50:22.523+0000 7f901137fdc0 0 bucket checkpoint complete 2022-09-09T22:50:22.556 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jkcgbj-22 2022-09-09T22:50:22.557 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jkcgbj-22 2022-09-09T22:50:22.558 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:22.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:22.559 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:50:22.610 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:50:22.612 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:50:22.681 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:50:22.675+0000 7f44e01a7dc0 1 bucket sync caught up with source: 2022-09-09T22:50:22.682 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000002.96.5, , ] 2022-09-09T22:50:22.682 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000002.96.5, , ] 2022-09-09T22:50:22.683 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:50:22.675+0000 7f44e01a7dc0 0 bucket checkpoint complete 2022-09-09T22:50:22.709 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jkcgbj-22 2022-09-09T22:50:22.710 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:22.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:22.711 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:50:22.766 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:50:22.767 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:50:22.842 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:50:22.835+0000 7f8a56f75dc0 1 bucket sync caught up with source: 2022-09-09T22:50:22.842 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000002.96.5, , ] 2022-09-09T22:50:22.843 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000002.96.5, , ] 2022-09-09T22:50:22.843 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:50:22.835+0000 7f8a56f75dc0 0 bucket checkpoint complete 2022-09-09T22:50:22.887 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jkcgbj-22 2022-09-09T22:50:22.888 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-22 zones={test-zone2, test-zone1} 2022-09-09T22:50:22.889 DEBUG:boto:path=/jkcgbj-22/ 2022-09-09T22:50:22.890 DEBUG:boto:auth_path=/jkcgbj-22/ 2022-09-09T22:50:22.891 DEBUG:boto:Method: HEAD 2022-09-09T22:50:22.892 DEBUG:boto:Path: /jkcgbj-22/ 2022-09-09T22:50:22.893 DEBUG:boto:Data: 2022-09-09T22:50:22.894 DEBUG:boto:Headers: {} 2022-09-09T22:50:22.895 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:22.896 DEBUG:boto:Port: 8000 2022-09-09T22:50:22.896 DEBUG:boto:Params: {} 2022-09-09T22:50:22.897 DEBUG:boto:Token: None 2022-09-09T22:50:22.899 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:22 GMT /jkcgbj-22/ 2022-09-09T22:50:22.900 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:F1MkO/yrZMERAOlVwMe45aXbi2E= 2022-09-09T22:50:22.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:22 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:F1MkO/yrZMERAOlVwMe45aXbi2E=', 'Content-Length': '0'} 2022-09-09T22:50:22.912 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077589fd9d96c85ad-00631bc32e-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:22.914 DEBUG:boto:path=/jkcgbj-22/ 2022-09-09T22:50:22.915 DEBUG:boto:auth_path=/jkcgbj-22/ 2022-09-09T22:50:22.916 DEBUG:boto:Method: HEAD 2022-09-09T22:50:22.917 DEBUG:boto:Path: /jkcgbj-22/ 2022-09-09T22:50:22.919 DEBUG:boto:Data: 2022-09-09T22:50:22.919 DEBUG:boto:Headers: {} 2022-09-09T22:50:22.920 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:22.921 DEBUG:boto:Port: 8000 2022-09-09T22:50:22.922 DEBUG:boto:Params: {} 2022-09-09T22:50:22.923 DEBUG:boto:Token: None 2022-09-09T22:50:22.924 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:22 GMT /jkcgbj-22/ 2022-09-09T22:50:22.925 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:F1MkO/yrZMERAOlVwMe45aXbi2E= 2022-09-09T22:50:22.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:22 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:F1MkO/yrZMERAOlVwMe45aXbi2E=', 'Content-Length': '0'} 2022-09-09T22:50:22.932 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b111861726b3daf-00631bc32e-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:22.933 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:50:22.934 DEBUG:boto:path=/jkcgbj-22/ 2022-09-09T22:50:22.935 DEBUG:boto:auth_path=/jkcgbj-22/ 2022-09-09T22:50:22.936 DEBUG:boto:path=/jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:22.937 DEBUG:boto:auth_path=/jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:22.938 DEBUG:boto:Method: GET 2022-09-09T22:50:22.939 DEBUG:boto:Path: /jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:22.940 DEBUG:boto:Data: 2022-09-09T22:50:22.941 DEBUG:boto:Headers: {} 2022-09-09T22:50:22.942 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:22.943 DEBUG:boto:Port: 8000 2022-09-09T22:50:22.944 DEBUG:boto:Params: {} 2022-09-09T22:50:22.945 DEBUG:boto:Token: None 2022-09-09T22:50:22.946 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:22 GMT /jkcgbj-22/?versions 2022-09-09T22:50:22.947 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:tShbQuOuibzClK+McvpdAw6k7ug= 2022-09-09T22:50:22.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:22 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:tShbQuOuibzClK+McvpdAw6k7ug=', 'Content-Length': '0'} 2022-09-09T22:50:22.955 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe104d16c82ce463-00631bc32e-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:22.956 DEBUG:boto:b'jkcgbj-22999falseobj9KGDz7kz0-0inGbhR-9Nw2HQLGID0YDtrue2022-09-09T22:49:29.443Zrgw-multisite-test-userMultisite Test UserNormalobjwldLOHvZkprMnEge7.vgzLGeDVDkMURfalse2022-09-09T22:49:29.396Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:22.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:22.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:22.957 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:50:22.958 DEBUG:boto:path=/jkcgbj-22/ 2022-09-09T22:50:22.959 DEBUG:boto:auth_path=/jkcgbj-22/ 2022-09-09T22:50:22.959 DEBUG:boto:path=/jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:22.960 DEBUG:boto:auth_path=/jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:22.960 DEBUG:boto:Method: GET 2022-09-09T22:50:22.961 DEBUG:boto:Path: /jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:22.962 DEBUG:boto:Data: 2022-09-09T22:50:22.962 DEBUG:boto:Headers: {} 2022-09-09T22:50:22.963 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:22.963 DEBUG:boto:Port: 8000 2022-09-09T22:50:22.964 DEBUG:boto:Params: {} 2022-09-09T22:50:22.964 DEBUG:boto:Token: None 2022-09-09T22:50:22.965 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:22 GMT /jkcgbj-22/?versions 2022-09-09T22:50:22.966 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:tShbQuOuibzClK+McvpdAw6k7ug= 2022-09-09T22:50:22.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:22 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:tShbQuOuibzClK+McvpdAw6k7ug=', 'Content-Length': '0'} 2022-09-09T22:50:22.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a80a08e5b57a11d-00631bc32e-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:22.974 DEBUG:boto:b'jkcgbj-22999falseobj9KGDz7kz0-0inGbhR-9Nw2HQLGID0YDtrue2022-09-09T22:49:29.443Zrgw-multisite-test-userMultisite Test UserNormalobjwldLOHvZkprMnEge7.vgzLGeDVDkMURfalse2022-09-09T22:49:29.396Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:22.975 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:22.975 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:22.976 DEBUG:boto:path=/jkcgbj-22/ 2022-09-09T22:50:22.976 DEBUG:boto:auth_path=/jkcgbj-22/ 2022-09-09T22:50:22.977 DEBUG:boto:path=/jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:22.977 DEBUG:boto:auth_path=/jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:22.978 DEBUG:boto:Method: GET 2022-09-09T22:50:22.978 DEBUG:boto:Path: /jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:22.979 DEBUG:boto:Data: 2022-09-09T22:50:22.979 DEBUG:boto:Headers: {} 2022-09-09T22:50:22.980 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:22.980 DEBUG:boto:Port: 8000 2022-09-09T22:50:22.981 DEBUG:boto:Params: {} 2022-09-09T22:50:22.981 DEBUG:boto:Token: None 2022-09-09T22:50:22.982 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:22 GMT /jkcgbj-22/?versions 2022-09-09T22:50:22.983 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:tShbQuOuibzClK+McvpdAw6k7ug= 2022-09-09T22:50:22.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:22 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:tShbQuOuibzClK+McvpdAw6k7ug=', 'Content-Length': '0'} 2022-09-09T22:50:22.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9dd277c25258f73-00631bc32e-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:22.990 DEBUG:boto:b'jkcgbj-22999falseobj9KGDz7kz0-0inGbhR-9Nw2HQLGID0YDtrue2022-09-09T22:49:29.443Zrgw-multisite-test-userMultisite Test UserNormalobjwldLOHvZkprMnEge7.vgzLGeDVDkMURfalse2022-09-09T22:49:29.396Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:22.991 DEBUG:boto:path=/jkcgbj-22/ 2022-09-09T22:50:22.992 DEBUG:boto:auth_path=/jkcgbj-22/ 2022-09-09T22:50:22.992 DEBUG:boto:path=/jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:22.993 DEBUG:boto:auth_path=/jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:22.993 DEBUG:boto:Method: GET 2022-09-09T22:50:22.994 DEBUG:boto:Path: /jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:22.994 DEBUG:boto:Data: 2022-09-09T22:50:22.995 DEBUG:boto:Headers: {} 2022-09-09T22:50:22.995 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:22.996 DEBUG:boto:Port: 8000 2022-09-09T22:50:22.996 DEBUG:boto:Params: {} 2022-09-09T22:50:22.997 DEBUG:boto:Token: None 2022-09-09T22:50:22.997 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:22 GMT /jkcgbj-22/?versions 2022-09-09T22:50:22.998 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:tShbQuOuibzClK+McvpdAw6k7ug= 2022-09-09T22:50:22.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:22 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:tShbQuOuibzClK+McvpdAw6k7ug=', 'Content-Length': '0'} 2022-09-09T22:50:23.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000204b035c3867b103-00631bc32e-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.046 DEBUG:boto:b'jkcgbj-22999falseobj9KGDz7kz0-0inGbhR-9Nw2HQLGID0YDtrue2022-09-09T22:49:29.443Zrgw-multisite-test-userMultisite Test UserNormalobjwldLOHvZkprMnEge7.vgzLGeDVDkMURfalse2022-09-09T22:49:29.396Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:23.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:50:23.049 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:50:23.050 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:50:23.050 DEBUG:boto:Method: HEAD 2022-09-09T22:50:23.051 DEBUG:boto:Path: /jkcgbj-22/obj 2022-09-09T22:50:23.052 DEBUG:boto:Data: 2022-09-09T22:50:23.053 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.053 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:23.054 DEBUG:boto:Port: 8000 2022-09-09T22:50:23.055 DEBUG:boto:Params: {} 2022-09-09T22:50:23.055 DEBUG:boto:Token: None 2022-09-09T22:50:23.056 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/obj 2022-09-09T22:50:23.057 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TaW1Rcjge4G5lviNxwySw2eT+L4= 2022-09-09T22:50:23.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TaW1Rcjge4G5lviNxwySw2eT+L4=', 'Content-Length': '0'} 2022-09-09T22:50:23.062 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000e8f705d43f66ec60-00631bc32f-10e8-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.063 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:50:23.064 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:50:23.065 DEBUG:boto:Method: HEAD 2022-09-09T22:50:23.066 DEBUG:boto:Path: /jkcgbj-22/obj 2022-09-09T22:50:23.067 DEBUG:boto:Data: 2022-09-09T22:50:23.068 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.069 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:23.070 DEBUG:boto:Port: 8000 2022-09-09T22:50:23.071 DEBUG:boto:Params: {} 2022-09-09T22:50:23.072 DEBUG:boto:Token: None 2022-09-09T22:50:23.073 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/obj 2022-09-09T22:50:23.074 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TaW1Rcjge4G5lviNxwySw2eT+L4= 2022-09-09T22:50:23.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TaW1Rcjge4G5lviNxwySw2eT+L4=', 'Content-Length': '0'} 2022-09-09T22:50:23.080 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000036a67924fc1daf0c-00631bc32f-10ff-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:50:23.083 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:50:23.084 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:50:23.084 DEBUG:boto:path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.085 DEBUG:boto:auth_path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.086 DEBUG:boto:Method: GET 2022-09-09T22:50:23.087 DEBUG:boto:Path: /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.088 DEBUG:boto:Data: 2022-09-09T22:50:23.089 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.090 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:23.091 DEBUG:boto:Port: 8000 2022-09-09T22:50:23.092 DEBUG:boto:Params: {} 2022-09-09T22:50:23.093 DEBUG:boto:Token: None 2022-09-09T22:50:23.095 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.096 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:J7HFWjuwpQCvKx219ULDP/tzcPU= 2022-09-09T22:50:23.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:J7HFWjuwpQCvKx219ULDP/tzcPU=', 'Content-Length': '0'} 2022-09-09T22:50:23.101 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'wldLOHvZkprMnEge7.vgzLGeDVDkMUR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000048669f7471440609-00631bc32f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.103 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:50:23.104 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:50:23.105 DEBUG:boto:path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.106 DEBUG:boto:auth_path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.107 DEBUG:boto:Method: GET 2022-09-09T22:50:23.108 DEBUG:boto:Path: /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.109 DEBUG:boto:Data: 2022-09-09T22:50:23.109 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.110 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:23.111 DEBUG:boto:Port: 8000 2022-09-09T22:50:23.112 DEBUG:boto:Params: {} 2022-09-09T22:50:23.114 DEBUG:boto:Token: None 2022-09-09T22:50:23.115 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.116 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:J7HFWjuwpQCvKx219ULDP/tzcPU= 2022-09-09T22:50:23.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:J7HFWjuwpQCvKx219ULDP/tzcPU=', 'Content-Length': '0'} 2022-09-09T22:50:23.121 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'wldLOHvZkprMnEge7.vgzLGeDVDkMUR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f3e5ead7a7a5f51a-00631bc32f-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.122 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:50:23.123 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:50:23.123 DEBUG:boto:path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.124 DEBUG:boto:auth_path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.124 DEBUG:boto:Method: HEAD 2022-09-09T22:50:23.125 DEBUG:boto:Path: /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.125 DEBUG:boto:Data: 2022-09-09T22:50:23.126 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.126 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:23.127 DEBUG:boto:Port: 8000 2022-09-09T22:50:23.128 DEBUG:boto:Params: {} 2022-09-09T22:50:23.128 DEBUG:boto:Token: None 2022-09-09T22:50:23.129 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.129 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Sz8rPqIvDl6wBPDO9ly/2E2PRGU= 2022-09-09T22:50:23.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Sz8rPqIvDl6wBPDO9ly/2E2PRGU=', 'Content-Length': '0'} 2022-09-09T22:50:23.134 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'wldLOHvZkprMnEge7.vgzLGeDVDkMUR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d38dbe00af369721-00631bc32f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.135 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:50:23.135 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:50:23.136 DEBUG:boto:path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.136 DEBUG:boto:auth_path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.137 DEBUG:boto:Method: HEAD 2022-09-09T22:50:23.137 DEBUG:boto:Path: /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.138 DEBUG:boto:Data: 2022-09-09T22:50:23.138 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.139 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:23.139 DEBUG:boto:Port: 8000 2022-09-09T22:50:23.140 DEBUG:boto:Params: {} 2022-09-09T22:50:23.140 DEBUG:boto:Token: None 2022-09-09T22:50:23.141 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.142 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Sz8rPqIvDl6wBPDO9ly/2E2PRGU= 2022-09-09T22:50:23.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Sz8rPqIvDl6wBPDO9ly/2E2PRGU=', 'Content-Length': '0'} 2022-09-09T22:50:23.145 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'wldLOHvZkprMnEge7.vgzLGeDVDkMUR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cb6c1a7e9ed1c536-00631bc32f-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:50:23.147 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:50:23.147 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:50:23.148 DEBUG:boto:path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.148 DEBUG:boto:auth_path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.149 DEBUG:boto:Method: GET 2022-09-09T22:50:23.149 DEBUG:boto:Path: /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.150 DEBUG:boto:Data: 2022-09-09T22:50:23.150 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.151 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:23.151 DEBUG:boto:Port: 8000 2022-09-09T22:50:23.152 DEBUG:boto:Params: {} 2022-09-09T22:50:23.152 DEBUG:boto:Token: None 2022-09-09T22:50:23.153 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.154 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:J7HFWjuwpQCvKx219ULDP/tzcPU= 2022-09-09T22:50:23.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:J7HFWjuwpQCvKx219ULDP/tzcPU=', 'Content-Length': '0'} 2022-09-09T22:50:23.157 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'wldLOHvZkprMnEge7.vgzLGeDVDkMUR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004f27063bd979d921-00631bc32f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.159 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:50:23.159 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:50:23.160 DEBUG:boto:path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.160 DEBUG:boto:auth_path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.161 DEBUG:boto:Method: GET 2022-09-09T22:50:23.161 DEBUG:boto:Path: /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.162 DEBUG:boto:Data: 2022-09-09T22:50:23.162 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.163 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:23.163 DEBUG:boto:Port: 8000 2022-09-09T22:50:23.164 DEBUG:boto:Params: {} 2022-09-09T22:50:23.164 DEBUG:boto:Token: None 2022-09-09T22:50:23.165 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.166 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:J7HFWjuwpQCvKx219ULDP/tzcPU= 2022-09-09T22:50:23.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:J7HFWjuwpQCvKx219ULDP/tzcPU=', 'Content-Length': '0'} 2022-09-09T22:50:23.169 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'wldLOHvZkprMnEge7.vgzLGeDVDkMUR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bdeee712275012dc-00631bc32f-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.171 DEBUG:boto:path=/jkcgbj-22/ 2022-09-09T22:50:23.172 DEBUG:boto:auth_path=/jkcgbj-22/ 2022-09-09T22:50:23.173 DEBUG:boto:path=/jkcgbj-22/?prefix=obj 2022-09-09T22:50:23.173 DEBUG:boto:auth_path=/jkcgbj-22/?prefix=obj 2022-09-09T22:50:23.174 DEBUG:boto:Method: GET 2022-09-09T22:50:23.174 DEBUG:boto:Path: /jkcgbj-22/?prefix=obj 2022-09-09T22:50:23.175 DEBUG:boto:Data: 2022-09-09T22:50:23.176 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.176 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:23.177 DEBUG:boto:Port: 8000 2022-09-09T22:50:23.177 DEBUG:boto:Params: {} 2022-09-09T22:50:23.178 DEBUG:boto:Token: None 2022-09-09T22:50:23.179 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/ 2022-09-09T22:50:23.180 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:s9u1DM1KVhSUVQ/s0S0HVG8ds+k= 2022-09-09T22:50:23.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:s9u1DM1KVhSUVQ/s0S0HVG8ds+k=', 'Content-Length': '0'} 2022-09-09T22:50:23.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eba1bd8760c0e380-00631bc32f-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.189 DEBUG:boto:b'jkcgbj-22obj1000false' 2022-09-09T22:50:23.190 DEBUG:boto:path=/jkcgbj-22/ 2022-09-09T22:50:23.191 DEBUG:boto:auth_path=/jkcgbj-22/ 2022-09-09T22:50:23.192 DEBUG:boto:path=/jkcgbj-22/?prefix=obj 2022-09-09T22:50:23.193 DEBUG:boto:auth_path=/jkcgbj-22/?prefix=obj 2022-09-09T22:50:23.194 DEBUG:boto:Method: GET 2022-09-09T22:50:23.195 DEBUG:boto:Path: /jkcgbj-22/?prefix=obj 2022-09-09T22:50:23.196 DEBUG:boto:Data: 2022-09-09T22:50:23.197 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.198 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:23.199 DEBUG:boto:Port: 8000 2022-09-09T22:50:23.199 DEBUG:boto:Params: {} 2022-09-09T22:50:23.200 DEBUG:boto:Token: None 2022-09-09T22:50:23.201 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/ 2022-09-09T22:50:23.202 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:s9u1DM1KVhSUVQ/s0S0HVG8ds+k= 2022-09-09T22:50:23.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:s9u1DM1KVhSUVQ/s0S0HVG8ds+k=', 'Content-Length': '0'} 2022-09-09T22:50:23.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e370a27be1b7e5a0-00631bc32f-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.211 DEBUG:boto:b'jkcgbj-22obj1000false' 2022-09-09T22:50:23.212 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:50:23.213 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:50:23.214 DEBUG:boto:Method: HEAD 2022-09-09T22:50:23.215 DEBUG:boto:Path: /jkcgbj-22/obj 2022-09-09T22:50:23.216 DEBUG:boto:Data: 2022-09-09T22:50:23.217 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.218 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:23.219 DEBUG:boto:Port: 8000 2022-09-09T22:50:23.220 DEBUG:boto:Params: {} 2022-09-09T22:50:23.221 DEBUG:boto:Token: None 2022-09-09T22:50:23.221 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/obj 2022-09-09T22:50:23.222 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TaW1Rcjge4G5lviNxwySw2eT+L4= 2022-09-09T22:50:23.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TaW1Rcjge4G5lviNxwySw2eT+L4=', 'Content-Length': '0'} 2022-09-09T22:50:23.228 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000006a73db000e92d59-00631bc32f-10e8-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.229 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:50:23.230 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:50:23.231 DEBUG:boto:Method: HEAD 2022-09-09T22:50:23.232 DEBUG:boto:Path: /jkcgbj-22/obj 2022-09-09T22:50:23.233 DEBUG:boto:Data: 2022-09-09T22:50:23.234 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.234 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:23.235 DEBUG:boto:Port: 8000 2022-09-09T22:50:23.236 DEBUG:boto:Params: {} 2022-09-09T22:50:23.237 DEBUG:boto:Token: None 2022-09-09T22:50:23.238 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/obj 2022-09-09T22:50:23.238 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TaW1Rcjge4G5lviNxwySw2eT+L4= 2022-09-09T22:50:23.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TaW1Rcjge4G5lviNxwySw2eT+L4=', 'Content-Length': '0'} 2022-09-09T22:50:23.243 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000f4463c9d8d55eab9-00631bc32f-10ff-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.244 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-22 zones={test-zone2, test-zone1} 2022-09-09T22:50:23.245 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-22 zones={test-zone3, test-zone1} 2022-09-09T22:50:23.246 DEBUG:boto:path=/jkcgbj-22/ 2022-09-09T22:50:23.246 DEBUG:boto:auth_path=/jkcgbj-22/ 2022-09-09T22:50:23.247 DEBUG:boto:Method: HEAD 2022-09-09T22:50:23.247 DEBUG:boto:Path: /jkcgbj-22/ 2022-09-09T22:50:23.248 DEBUG:boto:Data: 2022-09-09T22:50:23.248 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.249 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:23.249 DEBUG:boto:Port: 8001 2022-09-09T22:50:23.250 DEBUG:boto:Params: {} 2022-09-09T22:50:23.251 DEBUG:boto:Token: None 2022-09-09T22:50:23.251 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/ 2022-09-09T22:50:23.252 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vw47YBCc7WtCtI4w4aa2F2Y9CQ8= 2022-09-09T22:50:23.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vw47YBCc7WtCtI4w4aa2F2Y9CQ8=', 'Content-Length': '0'} 2022-09-09T22:50:23.258 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000061ba9c57bdf3a447-00631bc32f-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.259 DEBUG:boto:path=/jkcgbj-22/ 2022-09-09T22:50:23.260 DEBUG:boto:auth_path=/jkcgbj-22/ 2022-09-09T22:50:23.261 DEBUG:boto:Method: HEAD 2022-09-09T22:50:23.262 DEBUG:boto:Path: /jkcgbj-22/ 2022-09-09T22:50:23.263 DEBUG:boto:Data: 2022-09-09T22:50:23.264 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.265 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:23.265 DEBUG:boto:Port: 8000 2022-09-09T22:50:23.266 DEBUG:boto:Params: {} 2022-09-09T22:50:23.267 DEBUG:boto:Token: None 2022-09-09T22:50:23.268 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/ 2022-09-09T22:50:23.269 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vw47YBCc7WtCtI4w4aa2F2Y9CQ8= 2022-09-09T22:50:23.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vw47YBCc7WtCtI4w4aa2F2Y9CQ8=', 'Content-Length': '0'} 2022-09-09T22:50:23.275 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000080b020a953a23b64-00631bc32f-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.277 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:50:23.278 DEBUG:boto:path=/jkcgbj-22/ 2022-09-09T22:50:23.279 DEBUG:boto:auth_path=/jkcgbj-22/ 2022-09-09T22:50:23.280 DEBUG:boto:path=/jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:23.281 DEBUG:boto:auth_path=/jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:23.282 DEBUG:boto:Method: GET 2022-09-09T22:50:23.282 DEBUG:boto:Path: /jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:23.283 DEBUG:boto:Data: 2022-09-09T22:50:23.284 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.285 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:23.286 DEBUG:boto:Port: 8001 2022-09-09T22:50:23.286 DEBUG:boto:Params: {} 2022-09-09T22:50:23.287 DEBUG:boto:Token: None 2022-09-09T22:50:23.288 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/?versions 2022-09-09T22:50:23.289 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:kl5kDiPPLHunxgiBdjXGPu70v90= 2022-09-09T22:50:23.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:kl5kDiPPLHunxgiBdjXGPu70v90=', 'Content-Length': '0'} 2022-09-09T22:50:23.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008676ef35bacb582-00631bc32f-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.298 DEBUG:boto:b'jkcgbj-22999falseobj9KGDz7kz0-0inGbhR-9Nw2HQLGID0YDtrue2022-09-09T22:49:29.443Zrgw-multisite-test-userMultisite Test UserNormalobjwldLOHvZkprMnEge7.vgzLGeDVDkMURfalse2022-09-09T22:49:29.396Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:23.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:23.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:23.300 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:50:23.300 DEBUG:boto:path=/jkcgbj-22/ 2022-09-09T22:50:23.301 DEBUG:boto:auth_path=/jkcgbj-22/ 2022-09-09T22:50:23.301 DEBUG:boto:path=/jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:23.302 DEBUG:boto:auth_path=/jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:23.302 DEBUG:boto:Method: GET 2022-09-09T22:50:23.303 DEBUG:boto:Path: /jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:23.303 DEBUG:boto:Data: 2022-09-09T22:50:23.304 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.304 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:23.305 DEBUG:boto:Port: 8000 2022-09-09T22:50:23.305 DEBUG:boto:Params: {} 2022-09-09T22:50:23.306 DEBUG:boto:Token: None 2022-09-09T22:50:23.307 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/?versions 2022-09-09T22:50:23.307 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:kl5kDiPPLHunxgiBdjXGPu70v90= 2022-09-09T22:50:23.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:kl5kDiPPLHunxgiBdjXGPu70v90=', 'Content-Length': '0'} 2022-09-09T22:50:23.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4136e2bff022936-00631bc32f-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.358 DEBUG:boto:b'jkcgbj-22999falseobj9KGDz7kz0-0inGbhR-9Nw2HQLGID0YDtrue2022-09-09T22:49:29.443Zrgw-multisite-test-userMultisite Test UserNormalobjwldLOHvZkprMnEge7.vgzLGeDVDkMURfalse2022-09-09T22:49:29.396Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:23.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:23.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:23.360 DEBUG:boto:path=/jkcgbj-22/ 2022-09-09T22:50:23.361 DEBUG:boto:auth_path=/jkcgbj-22/ 2022-09-09T22:50:23.361 DEBUG:boto:path=/jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:23.362 DEBUG:boto:auth_path=/jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:23.363 DEBUG:boto:Method: GET 2022-09-09T22:50:23.363 DEBUG:boto:Path: /jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:23.364 DEBUG:boto:Data: 2022-09-09T22:50:23.364 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.365 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:23.366 DEBUG:boto:Port: 8001 2022-09-09T22:50:23.366 DEBUG:boto:Params: {} 2022-09-09T22:50:23.367 DEBUG:boto:Token: None 2022-09-09T22:50:23.368 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/?versions 2022-09-09T22:50:23.369 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:kl5kDiPPLHunxgiBdjXGPu70v90= 2022-09-09T22:50:23.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:kl5kDiPPLHunxgiBdjXGPu70v90=', 'Content-Length': '0'} 2022-09-09T22:50:23.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd4000e0f4f739f1-00631bc32f-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.376 DEBUG:boto:b'jkcgbj-22999falseobj9KGDz7kz0-0inGbhR-9Nw2HQLGID0YDtrue2022-09-09T22:49:29.443Zrgw-multisite-test-userMultisite Test UserNormalobjwldLOHvZkprMnEge7.vgzLGeDVDkMURfalse2022-09-09T22:49:29.396Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:23.377 DEBUG:boto:path=/jkcgbj-22/ 2022-09-09T22:50:23.377 DEBUG:boto:auth_path=/jkcgbj-22/ 2022-09-09T22:50:23.378 DEBUG:boto:path=/jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:23.378 DEBUG:boto:auth_path=/jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:23.379 DEBUG:boto:Method: GET 2022-09-09T22:50:23.379 DEBUG:boto:Path: /jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:23.380 DEBUG:boto:Data: 2022-09-09T22:50:23.380 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.381 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:23.381 DEBUG:boto:Port: 8000 2022-09-09T22:50:23.382 DEBUG:boto:Params: {} 2022-09-09T22:50:23.382 DEBUG:boto:Token: None 2022-09-09T22:50:23.383 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/?versions 2022-09-09T22:50:23.384 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:kl5kDiPPLHunxgiBdjXGPu70v90= 2022-09-09T22:50:23.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:kl5kDiPPLHunxgiBdjXGPu70v90=', 'Content-Length': '0'} 2022-09-09T22:50:23.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f00b14590ebdac0-00631bc32f-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.391 DEBUG:boto:b'jkcgbj-22999falseobj9KGDz7kz0-0inGbhR-9Nw2HQLGID0YDtrue2022-09-09T22:49:29.443Zrgw-multisite-test-userMultisite Test UserNormalobjwldLOHvZkprMnEge7.vgzLGeDVDkMURfalse2022-09-09T22:49:29.396Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:23.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:50:23.392 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:50:23.393 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:50:23.393 DEBUG:boto:Method: HEAD 2022-09-09T22:50:23.394 DEBUG:boto:Path: /jkcgbj-22/obj 2022-09-09T22:50:23.394 DEBUG:boto:Data: 2022-09-09T22:50:23.395 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.395 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:23.396 DEBUG:boto:Port: 8001 2022-09-09T22:50:23.396 DEBUG:boto:Params: {} 2022-09-09T22:50:23.397 DEBUG:boto:Token: None 2022-09-09T22:50:23.397 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/obj 2022-09-09T22:50:23.398 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TaW1Rcjge4G5lviNxwySw2eT+L4= 2022-09-09T22:50:23.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TaW1Rcjge4G5lviNxwySw2eT+L4=', 'Content-Length': '0'} 2022-09-09T22:50:23.403 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000f1e87febe6ae887d-00631bc32f-1103-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.404 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:50:23.405 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:50:23.406 DEBUG:boto:Method: HEAD 2022-09-09T22:50:23.406 DEBUG:boto:Path: /jkcgbj-22/obj 2022-09-09T22:50:23.407 DEBUG:boto:Data: 2022-09-09T22:50:23.408 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.409 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:23.410 DEBUG:boto:Port: 8000 2022-09-09T22:50:23.410 DEBUG:boto:Params: {} 2022-09-09T22:50:23.411 DEBUG:boto:Token: None 2022-09-09T22:50:23.412 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/obj 2022-09-09T22:50:23.412 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TaW1Rcjge4G5lviNxwySw2eT+L4= 2022-09-09T22:50:23.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TaW1Rcjge4G5lviNxwySw2eT+L4=', 'Content-Length': '0'} 2022-09-09T22:50:23.417 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000003c39099618658a13-00631bc32f-10ff-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:50:23.418 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:50:23.419 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:50:23.419 DEBUG:boto:path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.420 DEBUG:boto:auth_path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.420 DEBUG:boto:Method: GET 2022-09-09T22:50:23.421 DEBUG:boto:Path: /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.422 DEBUG:boto:Data: 2022-09-09T22:50:23.423 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.423 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:23.424 DEBUG:boto:Port: 8001 2022-09-09T22:50:23.424 DEBUG:boto:Params: {} 2022-09-09T22:50:23.425 DEBUG:boto:Token: None 2022-09-09T22:50:23.426 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.426 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:J7HFWjuwpQCvKx219ULDP/tzcPU= 2022-09-09T22:50:23.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:J7HFWjuwpQCvKx219ULDP/tzcPU=', 'Content-Length': '0'} 2022-09-09T22:50:23.430 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'wldLOHvZkprMnEge7.vgzLGeDVDkMUR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000da0df804adf7b09a-00631bc32f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.431 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:50:23.432 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:50:23.432 DEBUG:boto:path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.433 DEBUG:boto:auth_path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.434 DEBUG:boto:Method: GET 2022-09-09T22:50:23.434 DEBUG:boto:Path: /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.435 DEBUG:boto:Data: 2022-09-09T22:50:23.435 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.436 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:23.437 DEBUG:boto:Port: 8000 2022-09-09T22:50:23.437 DEBUG:boto:Params: {} 2022-09-09T22:50:23.438 DEBUG:boto:Token: None 2022-09-09T22:50:23.439 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.439 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:J7HFWjuwpQCvKx219ULDP/tzcPU= 2022-09-09T22:50:23.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:J7HFWjuwpQCvKx219ULDP/tzcPU=', 'Content-Length': '0'} 2022-09-09T22:50:23.443 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'wldLOHvZkprMnEge7.vgzLGeDVDkMUR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000955eb00b2176e8ef-00631bc32f-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.444 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:50:23.444 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:50:23.445 DEBUG:boto:path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.445 DEBUG:boto:auth_path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.446 DEBUG:boto:Method: HEAD 2022-09-09T22:50:23.447 DEBUG:boto:Path: /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.448 DEBUG:boto:Data: 2022-09-09T22:50:23.449 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.450 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:23.450 DEBUG:boto:Port: 8001 2022-09-09T22:50:23.451 DEBUG:boto:Params: {} 2022-09-09T22:50:23.451 DEBUG:boto:Token: None 2022-09-09T22:50:23.452 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.453 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Sz8rPqIvDl6wBPDO9ly/2E2PRGU= 2022-09-09T22:50:23.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Sz8rPqIvDl6wBPDO9ly/2E2PRGU=', 'Content-Length': '0'} 2022-09-09T22:50:23.456 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'wldLOHvZkprMnEge7.vgzLGeDVDkMUR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000167cf71a729255e0-00631bc32f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.458 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:50:23.459 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:50:23.460 DEBUG:boto:path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.461 DEBUG:boto:auth_path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.461 DEBUG:boto:Method: HEAD 2022-09-09T22:50:23.462 DEBUG:boto:Path: /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.463 DEBUG:boto:Data: 2022-09-09T22:50:23.464 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.465 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:23.466 DEBUG:boto:Port: 8000 2022-09-09T22:50:23.467 DEBUG:boto:Params: {} 2022-09-09T22:50:23.468 DEBUG:boto:Token: None 2022-09-09T22:50:23.469 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.470 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Sz8rPqIvDl6wBPDO9ly/2E2PRGU= 2022-09-09T22:50:23.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Sz8rPqIvDl6wBPDO9ly/2E2PRGU=', 'Content-Length': '0'} 2022-09-09T22:50:23.474 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'wldLOHvZkprMnEge7.vgzLGeDVDkMUR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b0f66caa40d96b7e-00631bc32f-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:50:23.475 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:50:23.476 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:50:23.476 DEBUG:boto:path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.477 DEBUG:boto:auth_path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.477 DEBUG:boto:Method: GET 2022-09-09T22:50:23.478 DEBUG:boto:Path: /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.479 DEBUG:boto:Data: 2022-09-09T22:50:23.479 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.480 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:23.480 DEBUG:boto:Port: 8001 2022-09-09T22:50:23.481 DEBUG:boto:Params: {} 2022-09-09T22:50:23.481 DEBUG:boto:Token: None 2022-09-09T22:50:23.482 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.483 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:J7HFWjuwpQCvKx219ULDP/tzcPU= 2022-09-09T22:50:23.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:J7HFWjuwpQCvKx219ULDP/tzcPU=', 'Content-Length': '0'} 2022-09-09T22:50:23.487 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'wldLOHvZkprMnEge7.vgzLGeDVDkMUR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000089eed19c144dc41e-00631bc32f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.488 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:50:23.489 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:50:23.490 DEBUG:boto:path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.491 DEBUG:boto:auth_path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.492 DEBUG:boto:Method: GET 2022-09-09T22:50:23.493 DEBUG:boto:Path: /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.493 DEBUG:boto:Data: 2022-09-09T22:50:23.494 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.494 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:23.495 DEBUG:boto:Port: 8000 2022-09-09T22:50:23.495 DEBUG:boto:Params: {} 2022-09-09T22:50:23.496 DEBUG:boto:Token: None 2022-09-09T22:50:23.497 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.497 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:J7HFWjuwpQCvKx219ULDP/tzcPU= 2022-09-09T22:50:23.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:J7HFWjuwpQCvKx219ULDP/tzcPU=', 'Content-Length': '0'} 2022-09-09T22:50:23.501 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'wldLOHvZkprMnEge7.vgzLGeDVDkMUR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000652fbb042c80cb30-00631bc32f-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.502 DEBUG:boto:path=/jkcgbj-22/ 2022-09-09T22:50:23.503 DEBUG:boto:auth_path=/jkcgbj-22/ 2022-09-09T22:50:23.503 DEBUG:boto:path=/jkcgbj-22/?prefix=obj 2022-09-09T22:50:23.504 DEBUG:boto:auth_path=/jkcgbj-22/?prefix=obj 2022-09-09T22:50:23.504 DEBUG:boto:Method: GET 2022-09-09T22:50:23.505 DEBUG:boto:Path: /jkcgbj-22/?prefix=obj 2022-09-09T22:50:23.505 DEBUG:boto:Data: 2022-09-09T22:50:23.506 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.507 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:23.508 DEBUG:boto:Port: 8001 2022-09-09T22:50:23.509 DEBUG:boto:Params: {} 2022-09-09T22:50:23.510 DEBUG:boto:Token: None 2022-09-09T22:50:23.511 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/ 2022-09-09T22:50:23.512 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:s9u1DM1KVhSUVQ/s0S0HVG8ds+k= 2022-09-09T22:50:23.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:s9u1DM1KVhSUVQ/s0S0HVG8ds+k=', 'Content-Length': '0'} 2022-09-09T22:50:23.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000644792779ca52c25-00631bc32f-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.558 DEBUG:boto:b'jkcgbj-22obj1000false' 2022-09-09T22:50:23.559 DEBUG:boto:path=/jkcgbj-22/ 2022-09-09T22:50:23.561 DEBUG:boto:auth_path=/jkcgbj-22/ 2022-09-09T22:50:23.562 DEBUG:boto:path=/jkcgbj-22/?prefix=obj 2022-09-09T22:50:23.563 DEBUG:boto:auth_path=/jkcgbj-22/?prefix=obj 2022-09-09T22:50:23.564 DEBUG:boto:Method: GET 2022-09-09T22:50:23.565 DEBUG:boto:Path: /jkcgbj-22/?prefix=obj 2022-09-09T22:50:23.566 DEBUG:boto:Data: 2022-09-09T22:50:23.567 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.568 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:23.569 DEBUG:boto:Port: 8000 2022-09-09T22:50:23.570 DEBUG:boto:Params: {} 2022-09-09T22:50:23.571 DEBUG:boto:Token: None 2022-09-09T22:50:23.572 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/ 2022-09-09T22:50:23.572 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:s9u1DM1KVhSUVQ/s0S0HVG8ds+k= 2022-09-09T22:50:23.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:s9u1DM1KVhSUVQ/s0S0HVG8ds+k=', 'Content-Length': '0'} 2022-09-09T22:50:23.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000208625b38c12650b-00631bc32f-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.582 DEBUG:boto:b'jkcgbj-22obj1000false' 2022-09-09T22:50:23.583 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:50:23.583 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:50:23.584 DEBUG:boto:Method: HEAD 2022-09-09T22:50:23.585 DEBUG:boto:Path: /jkcgbj-22/obj 2022-09-09T22:50:23.585 DEBUG:boto:Data: 2022-09-09T22:50:23.586 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.586 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:23.587 DEBUG:boto:Port: 8001 2022-09-09T22:50:23.587 DEBUG:boto:Params: {} 2022-09-09T22:50:23.588 DEBUG:boto:Token: None 2022-09-09T22:50:23.589 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/obj 2022-09-09T22:50:23.589 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TaW1Rcjge4G5lviNxwySw2eT+L4= 2022-09-09T22:50:23.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TaW1Rcjge4G5lviNxwySw2eT+L4=', 'Content-Length': '0'} 2022-09-09T22:50:23.595 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000fc9ac70d389d47a5-00631bc32f-1103-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.596 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:50:23.597 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:50:23.597 DEBUG:boto:Method: HEAD 2022-09-09T22:50:23.598 DEBUG:boto:Path: /jkcgbj-22/obj 2022-09-09T22:50:23.599 DEBUG:boto:Data: 2022-09-09T22:50:23.600 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.601 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:23.602 DEBUG:boto:Port: 8000 2022-09-09T22:50:23.603 DEBUG:boto:Params: {} 2022-09-09T22:50:23.604 DEBUG:boto:Token: None 2022-09-09T22:50:23.605 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/obj 2022-09-09T22:50:23.606 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TaW1Rcjge4G5lviNxwySw2eT+L4= 2022-09-09T22:50:23.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TaW1Rcjge4G5lviNxwySw2eT+L4=', 'Content-Length': '0'} 2022-09-09T22:50:23.611 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000f009a84653ea803b-00631bc32f-10ff-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.612 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-22 zones={test-zone3, test-zone1} 2022-09-09T22:50:23.613 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-22 zones={test-zone3, test-zone2} 2022-09-09T22:50:23.614 DEBUG:boto:path=/jkcgbj-22/ 2022-09-09T22:50:23.614 DEBUG:boto:auth_path=/jkcgbj-22/ 2022-09-09T22:50:23.615 DEBUG:boto:Method: HEAD 2022-09-09T22:50:23.615 DEBUG:boto:Path: /jkcgbj-22/ 2022-09-09T22:50:23.616 DEBUG:boto:Data: 2022-09-09T22:50:23.616 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.617 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:23.618 DEBUG:boto:Port: 8001 2022-09-09T22:50:23.618 DEBUG:boto:Params: {} 2022-09-09T22:50:23.619 DEBUG:boto:Token: None 2022-09-09T22:50:23.620 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/ 2022-09-09T22:50:23.620 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vw47YBCc7WtCtI4w4aa2F2Y9CQ8= 2022-09-09T22:50:23.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vw47YBCc7WtCtI4w4aa2F2Y9CQ8=', 'Content-Length': '0'} 2022-09-09T22:50:23.626 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007aca9efd2def73f9-00631bc32f-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.628 DEBUG:boto:path=/jkcgbj-22/ 2022-09-09T22:50:23.628 DEBUG:boto:auth_path=/jkcgbj-22/ 2022-09-09T22:50:23.629 DEBUG:boto:Method: HEAD 2022-09-09T22:50:23.630 DEBUG:boto:Path: /jkcgbj-22/ 2022-09-09T22:50:23.631 DEBUG:boto:Data: 2022-09-09T22:50:23.632 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.633 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:23.633 DEBUG:boto:Port: 8000 2022-09-09T22:50:23.634 DEBUG:boto:Params: {} 2022-09-09T22:50:23.635 DEBUG:boto:Token: None 2022-09-09T22:50:23.636 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/ 2022-09-09T22:50:23.637 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vw47YBCc7WtCtI4w4aa2F2Y9CQ8= 2022-09-09T22:50:23.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vw47YBCc7WtCtI4w4aa2F2Y9CQ8=', 'Content-Length': '0'} 2022-09-09T22:50:23.644 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf01f4fad6f0a661-00631bc32f-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.646 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:50:23.647 DEBUG:boto:path=/jkcgbj-22/ 2022-09-09T22:50:23.647 DEBUG:boto:auth_path=/jkcgbj-22/ 2022-09-09T22:50:23.648 DEBUG:boto:path=/jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:23.649 DEBUG:boto:auth_path=/jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:23.650 DEBUG:boto:Method: GET 2022-09-09T22:50:23.651 DEBUG:boto:Path: /jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:23.652 DEBUG:boto:Data: 2022-09-09T22:50:23.653 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.653 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:23.654 DEBUG:boto:Port: 8001 2022-09-09T22:50:23.655 DEBUG:boto:Params: {} 2022-09-09T22:50:23.657 DEBUG:boto:Token: None 2022-09-09T22:50:23.658 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/?versions 2022-09-09T22:50:23.659 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:kl5kDiPPLHunxgiBdjXGPu70v90= 2022-09-09T22:50:23.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:kl5kDiPPLHunxgiBdjXGPu70v90=', 'Content-Length': '0'} 2022-09-09T22:50:23.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b65b414268c8acf6-00631bc32f-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.714 DEBUG:boto:b'jkcgbj-22999falseobj9KGDz7kz0-0inGbhR-9Nw2HQLGID0YDtrue2022-09-09T22:49:29.443Zrgw-multisite-test-userMultisite Test UserNormalobjwldLOHvZkprMnEge7.vgzLGeDVDkMURfalse2022-09-09T22:49:29.396Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:23.715 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:23.716 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:23.716 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:50:23.717 DEBUG:boto:path=/jkcgbj-22/ 2022-09-09T22:50:23.717 DEBUG:boto:auth_path=/jkcgbj-22/ 2022-09-09T22:50:23.718 DEBUG:boto:path=/jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:23.718 DEBUG:boto:auth_path=/jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:23.719 DEBUG:boto:Method: GET 2022-09-09T22:50:23.720 DEBUG:boto:Path: /jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:23.720 DEBUG:boto:Data: 2022-09-09T22:50:23.721 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.722 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:23.722 DEBUG:boto:Port: 8000 2022-09-09T22:50:23.723 DEBUG:boto:Params: {} 2022-09-09T22:50:23.723 DEBUG:boto:Token: None 2022-09-09T22:50:23.724 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/?versions 2022-09-09T22:50:23.725 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:kl5kDiPPLHunxgiBdjXGPu70v90= 2022-09-09T22:50:23.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:kl5kDiPPLHunxgiBdjXGPu70v90=', 'Content-Length': '0'} 2022-09-09T22:50:23.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000806bb9bc3f31339c-00631bc32f-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.732 DEBUG:boto:b'jkcgbj-22999falseobj9KGDz7kz0-0inGbhR-9Nw2HQLGID0YDtrue2022-09-09T22:49:29.443Zrgw-multisite-test-userMultisite Test UserNormalobjwldLOHvZkprMnEge7.vgzLGeDVDkMURfalse2022-09-09T22:49:29.396Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:23.733 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:23.733 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:23.734 DEBUG:boto:path=/jkcgbj-22/ 2022-09-09T22:50:23.734 DEBUG:boto:auth_path=/jkcgbj-22/ 2022-09-09T22:50:23.735 DEBUG:boto:path=/jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:23.735 DEBUG:boto:auth_path=/jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:23.736 DEBUG:boto:Method: GET 2022-09-09T22:50:23.736 DEBUG:boto:Path: /jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:23.737 DEBUG:boto:Data: 2022-09-09T22:50:23.738 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.739 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:23.739 DEBUG:boto:Port: 8001 2022-09-09T22:50:23.740 DEBUG:boto:Params: {} 2022-09-09T22:50:23.740 DEBUG:boto:Token: None 2022-09-09T22:50:23.742 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/?versions 2022-09-09T22:50:23.743 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:kl5kDiPPLHunxgiBdjXGPu70v90= 2022-09-09T22:50:23.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:kl5kDiPPLHunxgiBdjXGPu70v90=', 'Content-Length': '0'} 2022-09-09T22:50:23.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000004dded9ef8bf954-00631bc32f-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.751 DEBUG:boto:b'jkcgbj-22999falseobj9KGDz7kz0-0inGbhR-9Nw2HQLGID0YDtrue2022-09-09T22:49:29.443Zrgw-multisite-test-userMultisite Test UserNormalobjwldLOHvZkprMnEge7.vgzLGeDVDkMURfalse2022-09-09T22:49:29.396Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:23.753 DEBUG:boto:path=/jkcgbj-22/ 2022-09-09T22:50:23.754 DEBUG:boto:auth_path=/jkcgbj-22/ 2022-09-09T22:50:23.755 DEBUG:boto:path=/jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:23.755 DEBUG:boto:auth_path=/jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:23.756 DEBUG:boto:Method: GET 2022-09-09T22:50:23.757 DEBUG:boto:Path: /jkcgbj-22/?versions&max-keys=999 2022-09-09T22:50:23.759 DEBUG:boto:Data: 2022-09-09T22:50:23.760 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.761 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:23.762 DEBUG:boto:Port: 8000 2022-09-09T22:50:23.763 DEBUG:boto:Params: {} 2022-09-09T22:50:23.764 DEBUG:boto:Token: None 2022-09-09T22:50:23.765 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/?versions 2022-09-09T22:50:23.766 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:kl5kDiPPLHunxgiBdjXGPu70v90= 2022-09-09T22:50:23.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:kl5kDiPPLHunxgiBdjXGPu70v90=', 'Content-Length': '0'} 2022-09-09T22:50:23.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cebf36001beb008e-00631bc32f-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.774 DEBUG:boto:b'jkcgbj-22999falseobj9KGDz7kz0-0inGbhR-9Nw2HQLGID0YDtrue2022-09-09T22:49:29.443Zrgw-multisite-test-userMultisite Test UserNormalobjwldLOHvZkprMnEge7.vgzLGeDVDkMURfalse2022-09-09T22:49:29.396Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:23.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:50:23.776 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:50:23.777 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:50:23.777 DEBUG:boto:Method: HEAD 2022-09-09T22:50:23.778 DEBUG:boto:Path: /jkcgbj-22/obj 2022-09-09T22:50:23.778 DEBUG:boto:Data: 2022-09-09T22:50:23.779 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.779 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:23.780 DEBUG:boto:Port: 8001 2022-09-09T22:50:23.780 DEBUG:boto:Params: {} 2022-09-09T22:50:23.781 DEBUG:boto:Token: None 2022-09-09T22:50:23.782 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/obj 2022-09-09T22:50:23.782 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TaW1Rcjge4G5lviNxwySw2eT+L4= 2022-09-09T22:50:23.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TaW1Rcjge4G5lviNxwySw2eT+L4=', 'Content-Length': '0'} 2022-09-09T22:50:23.787 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000013e4ccc554ec97a5-00631bc32f-1103-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.788 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:50:23.789 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:50:23.790 DEBUG:boto:Method: HEAD 2022-09-09T22:50:23.791 DEBUG:boto:Path: /jkcgbj-22/obj 2022-09-09T22:50:23.792 DEBUG:boto:Data: 2022-09-09T22:50:23.793 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.794 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:23.795 DEBUG:boto:Port: 8000 2022-09-09T22:50:23.796 DEBUG:boto:Params: {} 2022-09-09T22:50:23.797 DEBUG:boto:Token: None 2022-09-09T22:50:23.798 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/obj 2022-09-09T22:50:23.799 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TaW1Rcjge4G5lviNxwySw2eT+L4= 2022-09-09T22:50:23.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TaW1Rcjge4G5lviNxwySw2eT+L4=', 'Content-Length': '0'} 2022-09-09T22:50:23.805 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000d4e81ca551c46337-00631bc32f-10e8-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:50:23.807 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:50:23.808 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:50:23.809 DEBUG:boto:path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.810 DEBUG:boto:auth_path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.811 DEBUG:boto:Method: GET 2022-09-09T22:50:23.812 DEBUG:boto:Path: /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.813 DEBUG:boto:Data: 2022-09-09T22:50:23.814 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.815 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:23.816 DEBUG:boto:Port: 8001 2022-09-09T22:50:23.817 DEBUG:boto:Params: {} 2022-09-09T22:50:23.818 DEBUG:boto:Token: None 2022-09-09T22:50:23.819 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.820 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:J7HFWjuwpQCvKx219ULDP/tzcPU= 2022-09-09T22:50:23.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:J7HFWjuwpQCvKx219ULDP/tzcPU=', 'Content-Length': '0'} 2022-09-09T22:50:23.824 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'wldLOHvZkprMnEge7.vgzLGeDVDkMUR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000059e2dfb4940f6a78-00631bc32f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.825 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:50:23.826 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:50:23.827 DEBUG:boto:path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.827 DEBUG:boto:auth_path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.828 DEBUG:boto:Method: GET 2022-09-09T22:50:23.828 DEBUG:boto:Path: /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.829 DEBUG:boto:Data: 2022-09-09T22:50:23.831 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.831 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:23.833 DEBUG:boto:Port: 8000 2022-09-09T22:50:23.834 DEBUG:boto:Params: {} 2022-09-09T22:50:23.835 DEBUG:boto:Token: None 2022-09-09T22:50:23.835 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.836 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:J7HFWjuwpQCvKx219ULDP/tzcPU= 2022-09-09T22:50:23.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:J7HFWjuwpQCvKx219ULDP/tzcPU=', 'Content-Length': '0'} 2022-09-09T22:50:23.840 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'wldLOHvZkprMnEge7.vgzLGeDVDkMUR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000060cdb570bf50af8e-00631bc32f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.841 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:50:23.842 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:50:23.842 DEBUG:boto:path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.843 DEBUG:boto:auth_path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.844 DEBUG:boto:Method: HEAD 2022-09-09T22:50:23.845 DEBUG:boto:Path: /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.845 DEBUG:boto:Data: 2022-09-09T22:50:23.846 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.847 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:23.847 DEBUG:boto:Port: 8001 2022-09-09T22:50:23.848 DEBUG:boto:Params: {} 2022-09-09T22:50:23.849 DEBUG:boto:Token: None 2022-09-09T22:50:23.850 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.851 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Sz8rPqIvDl6wBPDO9ly/2E2PRGU= 2022-09-09T22:50:23.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Sz8rPqIvDl6wBPDO9ly/2E2PRGU=', 'Content-Length': '0'} 2022-09-09T22:50:23.854 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'wldLOHvZkprMnEge7.vgzLGeDVDkMUR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d430333b00296a31-00631bc32f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.855 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:50:23.856 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:50:23.856 DEBUG:boto:path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.857 DEBUG:boto:auth_path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.857 DEBUG:boto:Method: HEAD 2022-09-09T22:50:23.858 DEBUG:boto:Path: /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.858 DEBUG:boto:Data: 2022-09-09T22:50:23.859 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.859 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:23.860 DEBUG:boto:Port: 8000 2022-09-09T22:50:23.860 DEBUG:boto:Params: {} 2022-09-09T22:50:23.861 DEBUG:boto:Token: None 2022-09-09T22:50:23.862 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.862 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Sz8rPqIvDl6wBPDO9ly/2E2PRGU= 2022-09-09T22:50:23.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Sz8rPqIvDl6wBPDO9ly/2E2PRGU=', 'Content-Length': '0'} 2022-09-09T22:50:23.866 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'wldLOHvZkprMnEge7.vgzLGeDVDkMUR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007970322304182bec-00631bc32f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:50:23.867 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:50:23.867 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:50:23.868 DEBUG:boto:path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.868 DEBUG:boto:auth_path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.869 DEBUG:boto:Method: GET 2022-09-09T22:50:23.870 DEBUG:boto:Path: /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.870 DEBUG:boto:Data: 2022-09-09T22:50:23.871 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.871 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:23.872 DEBUG:boto:Port: 8001 2022-09-09T22:50:23.872 DEBUG:boto:Params: {} 2022-09-09T22:50:23.873 DEBUG:boto:Token: None 2022-09-09T22:50:23.873 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.874 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:J7HFWjuwpQCvKx219ULDP/tzcPU= 2022-09-09T22:50:23.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:J7HFWjuwpQCvKx219ULDP/tzcPU=', 'Content-Length': '0'} 2022-09-09T22:50:23.878 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'wldLOHvZkprMnEge7.vgzLGeDVDkMUR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000025ef262f57abc0ca-00631bc32f-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.879 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:50:23.879 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:50:23.880 DEBUG:boto:path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.880 DEBUG:boto:auth_path=/jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.881 DEBUG:boto:Method: GET 2022-09-09T22:50:23.881 DEBUG:boto:Path: /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.882 DEBUG:boto:Data: 2022-09-09T22:50:23.882 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.883 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:23.884 DEBUG:boto:Port: 8000 2022-09-09T22:50:23.885 DEBUG:boto:Params: {} 2022-09-09T22:50:23.886 DEBUG:boto:Token: None 2022-09-09T22:50:23.887 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/obj?versionId=wldLOHvZkprMnEge7.vgzLGeDVDkMUR 2022-09-09T22:50:23.888 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:J7HFWjuwpQCvKx219ULDP/tzcPU= 2022-09-09T22:50:23.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:J7HFWjuwpQCvKx219ULDP/tzcPU=', 'Content-Length': '0'} 2022-09-09T22:50:23.893 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'wldLOHvZkprMnEge7.vgzLGeDVDkMUR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e7d15b4a243f014f-00631bc32f-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.895 DEBUG:boto:path=/jkcgbj-22/ 2022-09-09T22:50:23.895 DEBUG:boto:auth_path=/jkcgbj-22/ 2022-09-09T22:50:23.896 DEBUG:boto:path=/jkcgbj-22/?prefix=obj 2022-09-09T22:50:23.896 DEBUG:boto:auth_path=/jkcgbj-22/?prefix=obj 2022-09-09T22:50:23.897 DEBUG:boto:Method: GET 2022-09-09T22:50:23.897 DEBUG:boto:Path: /jkcgbj-22/?prefix=obj 2022-09-09T22:50:23.898 DEBUG:boto:Data: 2022-09-09T22:50:23.898 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.899 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:23.899 DEBUG:boto:Port: 8001 2022-09-09T22:50:23.900 DEBUG:boto:Params: {} 2022-09-09T22:50:23.901 DEBUG:boto:Token: None 2022-09-09T22:50:23.901 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/ 2022-09-09T22:50:23.902 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:s9u1DM1KVhSUVQ/s0S0HVG8ds+k= 2022-09-09T22:50:23.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:s9u1DM1KVhSUVQ/s0S0HVG8ds+k=', 'Content-Length': '0'} 2022-09-09T22:50:23.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003103d1c11f711a5c-00631bc32f-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.950 DEBUG:boto:b'jkcgbj-22obj1000false' 2022-09-09T22:50:23.951 DEBUG:boto:path=/jkcgbj-22/ 2022-09-09T22:50:23.952 DEBUG:boto:auth_path=/jkcgbj-22/ 2022-09-09T22:50:23.953 DEBUG:boto:path=/jkcgbj-22/?prefix=obj 2022-09-09T22:50:23.953 DEBUG:boto:auth_path=/jkcgbj-22/?prefix=obj 2022-09-09T22:50:23.954 DEBUG:boto:Method: GET 2022-09-09T22:50:23.955 DEBUG:boto:Path: /jkcgbj-22/?prefix=obj 2022-09-09T22:50:23.955 DEBUG:boto:Data: 2022-09-09T22:50:23.956 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.956 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:23.958 DEBUG:boto:Port: 8000 2022-09-09T22:50:23.959 DEBUG:boto:Params: {} 2022-09-09T22:50:23.960 DEBUG:boto:Token: None 2022-09-09T22:50:23.960 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/ 2022-09-09T22:50:23.961 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:s9u1DM1KVhSUVQ/s0S0HVG8ds+k= 2022-09-09T22:50:23.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:s9u1DM1KVhSUVQ/s0S0HVG8ds+k=', 'Content-Length': '0'} 2022-09-09T22:50:23.968 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006acde8393d5c0145-00631bc32f-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.969 DEBUG:boto:b'jkcgbj-22obj1000false' 2022-09-09T22:50:23.971 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:50:23.972 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:50:23.973 DEBUG:boto:Method: HEAD 2022-09-09T22:50:23.974 DEBUG:boto:Path: /jkcgbj-22/obj 2022-09-09T22:50:23.975 DEBUG:boto:Data: 2022-09-09T22:50:23.976 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.977 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:23.977 DEBUG:boto:Port: 8001 2022-09-09T22:50:23.978 DEBUG:boto:Params: {} 2022-09-09T22:50:23.980 DEBUG:boto:Token: None 2022-09-09T22:50:23.981 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/obj 2022-09-09T22:50:23.982 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TaW1Rcjge4G5lviNxwySw2eT+L4= 2022-09-09T22:50:23.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TaW1Rcjge4G5lviNxwySw2eT+L4=', 'Content-Length': '0'} 2022-09-09T22:50:23.988 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000038b419e90d5275cf-00631bc32f-1103-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:23.989 DEBUG:boto:path=/jkcgbj-22/obj 2022-09-09T22:50:23.990 DEBUG:boto:auth_path=/jkcgbj-22/obj 2022-09-09T22:50:23.991 DEBUG:boto:Method: HEAD 2022-09-09T22:50:23.993 DEBUG:boto:Path: /jkcgbj-22/obj 2022-09-09T22:50:23.993 DEBUG:boto:Data: 2022-09-09T22:50:23.994 DEBUG:boto:Headers: {} 2022-09-09T22:50:23.995 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:23.996 DEBUG:boto:Port: 8000 2022-09-09T22:50:23.997 DEBUG:boto:Params: {} 2022-09-09T22:50:23.998 DEBUG:boto:Token: None 2022-09-09T22:50:23.999 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:23 GMT /jkcgbj-22/obj 2022-09-09T22:50:23.999 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:TaW1Rcjge4G5lviNxwySw2eT+L4= 2022-09-09T22:50:24.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:23 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:TaW1Rcjge4G5lviNxwySw2eT+L4=', 'Content-Length': '0'} 2022-09-09T22:50:24.005 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000434f0279e31c2f41-00631bc32f-10e8-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:24.005 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-22 zones={test-zone3, test-zone2} 2022-09-09T22:50:24.006 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jkcgbj-23 2022-09-09T22:50:24.006 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:24.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:24.008 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:50:24.062 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:50:24.063 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:50:24.138 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:50:24.131+0000 7fb2f30c4dc0 1 bucket sync caught up with source: 2022-09-09T22:50:24.139 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000002.31.5, , ] 2022-09-09T22:50:24.140 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000002.31.5, , ] 2022-09-09T22:50:24.140 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:50:24.131+0000 7fb2f30c4dc0 0 bucket checkpoint complete 2022-09-09T22:50:24.170 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jkcgbj-23 2022-09-09T22:50:24.170 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:24.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:24.171 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:50:24.229 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:50:24.229 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:50:24.293 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:50:24.287+0000 7ffb621e1dc0 1 bucket sync caught up with source: 2022-09-09T22:50:24.293 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000002.31.5, , ] 2022-09-09T22:50:24.294 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000002.31.5, , ] 2022-09-09T22:50:24.294 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:50:24.287+0000 7ffb621e1dc0 0 bucket checkpoint complete 2022-09-09T22:50:24.320 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jkcgbj-23 2022-09-09T22:50:24.320 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:24.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:24.322 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:50:24.376 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:50:24.377 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:50:24.456 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:50:24.451+0000 7f445c123dc0 1 bucket sync caught up with source: 2022-09-09T22:50:24.457 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000002.31.5, , ] 2022-09-09T22:50:24.457 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000002.31.5, , ] 2022-09-09T22:50:24.458 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:50:24.451+0000 7f445c123dc0 0 bucket checkpoint complete 2022-09-09T22:50:24.489 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jkcgbj-23 2022-09-09T22:50:24.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:24.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:24.491 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:50:24.545 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:50:24.546 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:50:24.614 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:50:24.611+0000 7f573609ddc0 1 bucket sync caught up with source: 2022-09-09T22:50:24.615 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000002.61.5, , ] 2022-09-09T22:50:24.615 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000002.61.5, , ] 2022-09-09T22:50:24.615 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:50:24.611+0000 7f573609ddc0 0 bucket checkpoint complete 2022-09-09T22:50:24.644 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jkcgbj-23 2022-09-09T22:50:24.646 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:24.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:24.647 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:50:24.703 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:50:24.703 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:50:24.773 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:50:24.767+0000 7f2597fb6dc0 1 bucket sync caught up with source: 2022-09-09T22:50:24.773 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000002.61.5, , ] 2022-09-09T22:50:24.773 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000002.61.5, , ] 2022-09-09T22:50:24.774 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:50:24.767+0000 7f2597fb6dc0 0 bucket checkpoint complete 2022-09-09T22:50:24.801 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jkcgbj-23 2022-09-09T22:50:24.802 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jkcgbj-23 2022-09-09T22:50:24.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:24.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:24.803 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:50:24.856 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:50:24.856 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:50:24.929 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:50:24.923+0000 7f4a8ff5ddc0 1 bucket sync caught up with source: 2022-09-09T22:50:24.930 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000003.69.11, , ] 2022-09-09T22:50:24.930 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000003.69.11, , ] 2022-09-09T22:50:24.931 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:50:24.923+0000 7f4a8ff5ddc0 0 bucket checkpoint complete 2022-09-09T22:50:24.952 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jkcgbj-23 2022-09-09T22:50:24.953 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:24.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:24.955 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:50:25.010 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:50:25.011 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:50:25.086 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:50:25.079+0000 7f8142434dc0 1 bucket sync caught up with source: 2022-09-09T22:50:25.086 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000003.69.11, , ] 2022-09-09T22:50:25.086 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000003.69.11, , ] 2022-09-09T22:50:25.087 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:50:25.079+0000 7f8142434dc0 0 bucket checkpoint complete 2022-09-09T22:50:25.115 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jkcgbj-23 2022-09-09T22:50:25.117 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-23 zones={test-zone2, test-zone1} 2022-09-09T22:50:25.117 DEBUG:boto:path=/jkcgbj-23/ 2022-09-09T22:50:25.118 DEBUG:boto:auth_path=/jkcgbj-23/ 2022-09-09T22:50:25.119 DEBUG:boto:Method: HEAD 2022-09-09T22:50:25.120 DEBUG:boto:Path: /jkcgbj-23/ 2022-09-09T22:50:25.120 DEBUG:boto:Data: 2022-09-09T22:50:25.121 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.121 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:25.122 DEBUG:boto:Port: 8000 2022-09-09T22:50:25.122 DEBUG:boto:Params: {} 2022-09-09T22:50:25.123 DEBUG:boto:Token: None 2022-09-09T22:50:25.124 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/ 2022-09-09T22:50:25.125 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:r+snpJdiprmGbmqCkJdWkNdNdbc= 2022-09-09T22:50:25.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:r+snpJdiprmGbmqCkJdWkNdNdbc=', 'Content-Length': '0'} 2022-09-09T22:50:25.130 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000078d717a5ff488c2d-00631bc331-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.132 DEBUG:boto:path=/jkcgbj-23/ 2022-09-09T22:50:25.132 DEBUG:boto:auth_path=/jkcgbj-23/ 2022-09-09T22:50:25.133 DEBUG:boto:Method: HEAD 2022-09-09T22:50:25.133 DEBUG:boto:Path: /jkcgbj-23/ 2022-09-09T22:50:25.134 DEBUG:boto:Data: 2022-09-09T22:50:25.134 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.135 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:25.136 DEBUG:boto:Port: 8000 2022-09-09T22:50:25.136 DEBUG:boto:Params: {} 2022-09-09T22:50:25.137 DEBUG:boto:Token: None 2022-09-09T22:50:25.138 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/ 2022-09-09T22:50:25.138 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:r+snpJdiprmGbmqCkJdWkNdNdbc= 2022-09-09T22:50:25.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:r+snpJdiprmGbmqCkJdWkNdNdbc=', 'Content-Length': '0'} 2022-09-09T22:50:25.144 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f47a59e7476bf85-00631bc331-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.145 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:50:25.145 DEBUG:boto:path=/jkcgbj-23/ 2022-09-09T22:50:25.146 DEBUG:boto:auth_path=/jkcgbj-23/ 2022-09-09T22:50:25.146 DEBUG:boto:path=/jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.147 DEBUG:boto:auth_path=/jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.148 DEBUG:boto:Method: GET 2022-09-09T22:50:25.148 DEBUG:boto:Path: /jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.149 DEBUG:boto:Data: 2022-09-09T22:50:25.149 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.150 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:25.150 DEBUG:boto:Port: 8000 2022-09-09T22:50:25.151 DEBUG:boto:Params: {} 2022-09-09T22:50:25.151 DEBUG:boto:Token: None 2022-09-09T22:50:25.152 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/?versions 2022-09-09T22:50:25.153 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:MqVEbliwUAUBlSy04F37YX430tk= 2022-09-09T22:50:25.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:MqVEbliwUAUBlSy04F37YX430tk=', 'Content-Length': '0'} 2022-09-09T22:50:25.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000955c741e7c279110-00631bc331-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.159 DEBUG:boto:b'jkcgbj-23999falseobj8Z5c4vwfWvpNpRLMwFGV-tguoijURW5true2022-09-09T22:49:29.528Zrgw-multisite-test-userMultisite Test UserNormalobjjqA7MkcivIkZnCalrNyyNmjK3EMR-GFfalse2022-09-09T22:49:29.481Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:25.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:25.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:25.162 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:50:25.162 DEBUG:boto:path=/jkcgbj-23/ 2022-09-09T22:50:25.163 DEBUG:boto:auth_path=/jkcgbj-23/ 2022-09-09T22:50:25.163 DEBUG:boto:path=/jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.164 DEBUG:boto:auth_path=/jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.165 DEBUG:boto:Method: GET 2022-09-09T22:50:25.165 DEBUG:boto:Path: /jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.166 DEBUG:boto:Data: 2022-09-09T22:50:25.166 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.167 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:25.167 DEBUG:boto:Port: 8000 2022-09-09T22:50:25.168 DEBUG:boto:Params: {} 2022-09-09T22:50:25.169 DEBUG:boto:Token: None 2022-09-09T22:50:25.170 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/?versions 2022-09-09T22:50:25.170 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:MqVEbliwUAUBlSy04F37YX430tk= 2022-09-09T22:50:25.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:MqVEbliwUAUBlSy04F37YX430tk=', 'Content-Length': '0'} 2022-09-09T22:50:25.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2631839260cdf3b-00631bc331-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.178 DEBUG:boto:b'jkcgbj-23999falseobj8Z5c4vwfWvpNpRLMwFGV-tguoijURW5true2022-09-09T22:49:29.528Zrgw-multisite-test-userMultisite Test UserNormalobjjqA7MkcivIkZnCalrNyyNmjK3EMR-GFfalse2022-09-09T22:49:29.481Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:25.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:25.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:25.180 DEBUG:boto:path=/jkcgbj-23/ 2022-09-09T22:50:25.181 DEBUG:boto:auth_path=/jkcgbj-23/ 2022-09-09T22:50:25.181 DEBUG:boto:path=/jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.182 DEBUG:boto:auth_path=/jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.183 DEBUG:boto:Method: GET 2022-09-09T22:50:25.184 DEBUG:boto:Path: /jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.185 DEBUG:boto:Data: 2022-09-09T22:50:25.186 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.186 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:25.187 DEBUG:boto:Port: 8000 2022-09-09T22:50:25.187 DEBUG:boto:Params: {} 2022-09-09T22:50:25.188 DEBUG:boto:Token: None 2022-09-09T22:50:25.188 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/?versions 2022-09-09T22:50:25.189 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:MqVEbliwUAUBlSy04F37YX430tk= 2022-09-09T22:50:25.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:MqVEbliwUAUBlSy04F37YX430tk=', 'Content-Length': '0'} 2022-09-09T22:50:25.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015d634aeece11fa2-00631bc331-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.238 DEBUG:boto:b'jkcgbj-23999falseobj8Z5c4vwfWvpNpRLMwFGV-tguoijURW5true2022-09-09T22:49:29.528Zrgw-multisite-test-userMultisite Test UserNormalobjjqA7MkcivIkZnCalrNyyNmjK3EMR-GFfalse2022-09-09T22:49:29.481Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:25.239 DEBUG:boto:path=/jkcgbj-23/ 2022-09-09T22:50:25.240 DEBUG:boto:auth_path=/jkcgbj-23/ 2022-09-09T22:50:25.241 DEBUG:boto:path=/jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.241 DEBUG:boto:auth_path=/jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.242 DEBUG:boto:Method: GET 2022-09-09T22:50:25.243 DEBUG:boto:Path: /jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.243 DEBUG:boto:Data: 2022-09-09T22:50:25.244 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.244 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:25.245 DEBUG:boto:Port: 8000 2022-09-09T22:50:25.245 DEBUG:boto:Params: {} 2022-09-09T22:50:25.247 DEBUG:boto:Token: None 2022-09-09T22:50:25.247 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/?versions 2022-09-09T22:50:25.248 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:MqVEbliwUAUBlSy04F37YX430tk= 2022-09-09T22:50:25.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:MqVEbliwUAUBlSy04F37YX430tk=', 'Content-Length': '0'} 2022-09-09T22:50:25.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000516a52e6f12985e1-00631bc331-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.255 DEBUG:boto:b'jkcgbj-23999falseobj8Z5c4vwfWvpNpRLMwFGV-tguoijURW5true2022-09-09T22:49:29.528Zrgw-multisite-test-userMultisite Test UserNormalobjjqA7MkcivIkZnCalrNyyNmjK3EMR-GFfalse2022-09-09T22:49:29.481Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:25.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:50:25.256 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:50:25.257 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:50:25.258 DEBUG:boto:Method: HEAD 2022-09-09T22:50:25.258 DEBUG:boto:Path: /jkcgbj-23/obj 2022-09-09T22:50:25.259 DEBUG:boto:Data: 2022-09-09T22:50:25.259 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.260 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:25.260 DEBUG:boto:Port: 8000 2022-09-09T22:50:25.261 DEBUG:boto:Params: {} 2022-09-09T22:50:25.261 DEBUG:boto:Token: None 2022-09-09T22:50:25.262 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/obj 2022-09-09T22:50:25.263 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:DB10SY8NRhVFj/ykBMKjp6+HB6Q= 2022-09-09T22:50:25.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:DB10SY8NRhVFj/ykBMKjp6+HB6Q=', 'Content-Length': '0'} 2022-09-09T22:50:25.268 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000ae9ce2f4417f36d3-00631bc331-10e8-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.269 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:50:25.270 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:50:25.271 DEBUG:boto:Method: HEAD 2022-09-09T22:50:25.271 DEBUG:boto:Path: /jkcgbj-23/obj 2022-09-09T22:50:25.272 DEBUG:boto:Data: 2022-09-09T22:50:25.272 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.273 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:25.274 DEBUG:boto:Port: 8000 2022-09-09T22:50:25.274 DEBUG:boto:Params: {} 2022-09-09T22:50:25.275 DEBUG:boto:Token: None 2022-09-09T22:50:25.276 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/obj 2022-09-09T22:50:25.277 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:DB10SY8NRhVFj/ykBMKjp6+HB6Q= 2022-09-09T22:50:25.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:DB10SY8NRhVFj/ykBMKjp6+HB6Q=', 'Content-Length': '0'} 2022-09-09T22:50:25.283 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000009860ad05847db8a-00631bc331-10ff-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:50:25.285 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:50:25.286 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:50:25.287 DEBUG:boto:path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.288 DEBUG:boto:auth_path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.289 DEBUG:boto:Method: GET 2022-09-09T22:50:25.290 DEBUG:boto:Path: /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.291 DEBUG:boto:Data: 2022-09-09T22:50:25.291 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.292 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:25.293 DEBUG:boto:Port: 8000 2022-09-09T22:50:25.293 DEBUG:boto:Params: {} 2022-09-09T22:50:25.294 DEBUG:boto:Token: None 2022-09-09T22:50:25.294 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.295 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CLpJPcq1Pd0MEjpKKhMFjfBUPdc= 2022-09-09T22:50:25.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CLpJPcq1Pd0MEjpKKhMFjfBUPdc=', 'Content-Length': '0'} 2022-09-09T22:50:25.299 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'jqA7MkcivIkZnCalrNyyNmjK3EMR-GF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000088f0dbd5d6202438-00631bc331-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.300 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:50:25.300 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:50:25.301 DEBUG:boto:path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.301 DEBUG:boto:auth_path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.302 DEBUG:boto:Method: GET 2022-09-09T22:50:25.302 DEBUG:boto:Path: /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.303 DEBUG:boto:Data: 2022-09-09T22:50:25.303 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.304 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:25.304 DEBUG:boto:Port: 8000 2022-09-09T22:50:25.305 DEBUG:boto:Params: {} 2022-09-09T22:50:25.306 DEBUG:boto:Token: None 2022-09-09T22:50:25.306 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.307 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CLpJPcq1Pd0MEjpKKhMFjfBUPdc= 2022-09-09T22:50:25.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CLpJPcq1Pd0MEjpKKhMFjfBUPdc=', 'Content-Length': '0'} 2022-09-09T22:50:25.311 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'jqA7MkcivIkZnCalrNyyNmjK3EMR-GF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003e999f33b4e0648a-00631bc331-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.311 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:50:25.312 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:50:25.313 DEBUG:boto:path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.314 DEBUG:boto:auth_path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.315 DEBUG:boto:Method: HEAD 2022-09-09T22:50:25.316 DEBUG:boto:Path: /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.316 DEBUG:boto:Data: 2022-09-09T22:50:25.317 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.318 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:25.319 DEBUG:boto:Port: 8000 2022-09-09T22:50:25.320 DEBUG:boto:Params: {} 2022-09-09T22:50:25.321 DEBUG:boto:Token: None 2022-09-09T22:50:25.322 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.323 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JcZPFLT/xIx0+JOLiKLTKBQcJ+c= 2022-09-09T22:50:25.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JcZPFLT/xIx0+JOLiKLTKBQcJ+c=', 'Content-Length': '0'} 2022-09-09T22:50:25.327 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'jqA7MkcivIkZnCalrNyyNmjK3EMR-GF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000002c918cf35d35d57-00631bc331-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.328 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:50:25.329 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:50:25.329 DEBUG:boto:path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.330 DEBUG:boto:auth_path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.331 DEBUG:boto:Method: HEAD 2022-09-09T22:50:25.332 DEBUG:boto:Path: /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.332 DEBUG:boto:Data: 2022-09-09T22:50:25.333 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.333 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:25.334 DEBUG:boto:Port: 8000 2022-09-09T22:50:25.335 DEBUG:boto:Params: {} 2022-09-09T22:50:25.335 DEBUG:boto:Token: None 2022-09-09T22:50:25.336 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.337 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JcZPFLT/xIx0+JOLiKLTKBQcJ+c= 2022-09-09T22:50:25.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JcZPFLT/xIx0+JOLiKLTKBQcJ+c=', 'Content-Length': '0'} 2022-09-09T22:50:25.340 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'jqA7MkcivIkZnCalrNyyNmjK3EMR-GF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c4824a117993b79d-00631bc331-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:50:25.342 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:50:25.342 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:50:25.343 DEBUG:boto:path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.344 DEBUG:boto:auth_path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.345 DEBUG:boto:Method: GET 2022-09-09T22:50:25.346 DEBUG:boto:Path: /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.347 DEBUG:boto:Data: 2022-09-09T22:50:25.347 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.348 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:25.348 DEBUG:boto:Port: 8000 2022-09-09T22:50:25.349 DEBUG:boto:Params: {} 2022-09-09T22:50:25.349 DEBUG:boto:Token: None 2022-09-09T22:50:25.350 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.351 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CLpJPcq1Pd0MEjpKKhMFjfBUPdc= 2022-09-09T22:50:25.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CLpJPcq1Pd0MEjpKKhMFjfBUPdc=', 'Content-Length': '0'} 2022-09-09T22:50:25.354 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'jqA7MkcivIkZnCalrNyyNmjK3EMR-GF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e8b4992cae99cd31-00631bc331-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.355 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:50:25.356 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:50:25.356 DEBUG:boto:path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.357 DEBUG:boto:auth_path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.357 DEBUG:boto:Method: GET 2022-09-09T22:50:25.358 DEBUG:boto:Path: /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.358 DEBUG:boto:Data: 2022-09-09T22:50:25.359 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.359 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:25.360 DEBUG:boto:Port: 8000 2022-09-09T22:50:25.361 DEBUG:boto:Params: {} 2022-09-09T22:50:25.361 DEBUG:boto:Token: None 2022-09-09T22:50:25.362 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.362 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CLpJPcq1Pd0MEjpKKhMFjfBUPdc= 2022-09-09T22:50:25.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CLpJPcq1Pd0MEjpKKhMFjfBUPdc=', 'Content-Length': '0'} 2022-09-09T22:50:25.366 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'jqA7MkcivIkZnCalrNyyNmjK3EMR-GF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ae8f2773b6944220-00631bc331-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.367 DEBUG:boto:path=/jkcgbj-23/ 2022-09-09T22:50:25.368 DEBUG:boto:auth_path=/jkcgbj-23/ 2022-09-09T22:50:25.368 DEBUG:boto:path=/jkcgbj-23/?prefix=obj 2022-09-09T22:50:25.369 DEBUG:boto:auth_path=/jkcgbj-23/?prefix=obj 2022-09-09T22:50:25.369 DEBUG:boto:Method: GET 2022-09-09T22:50:25.370 DEBUG:boto:Path: /jkcgbj-23/?prefix=obj 2022-09-09T22:50:25.371 DEBUG:boto:Data: 2022-09-09T22:50:25.372 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.373 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:25.373 DEBUG:boto:Port: 8000 2022-09-09T22:50:25.374 DEBUG:boto:Params: {} 2022-09-09T22:50:25.374 DEBUG:boto:Token: None 2022-09-09T22:50:25.375 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/ 2022-09-09T22:50:25.375 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:gAB5vjnTVpmeAOe77boZqVVpjOw= 2022-09-09T22:50:25.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:gAB5vjnTVpmeAOe77boZqVVpjOw=', 'Content-Length': '0'} 2022-09-09T22:50:25.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001095839a4d906cd2-00631bc331-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.422 DEBUG:boto:b'jkcgbj-23obj1000false' 2022-09-09T22:50:25.424 DEBUG:boto:path=/jkcgbj-23/ 2022-09-09T22:50:25.425 DEBUG:boto:auth_path=/jkcgbj-23/ 2022-09-09T22:50:25.425 DEBUG:boto:path=/jkcgbj-23/?prefix=obj 2022-09-09T22:50:25.426 DEBUG:boto:auth_path=/jkcgbj-23/?prefix=obj 2022-09-09T22:50:25.427 DEBUG:boto:Method: GET 2022-09-09T22:50:25.428 DEBUG:boto:Path: /jkcgbj-23/?prefix=obj 2022-09-09T22:50:25.429 DEBUG:boto:Data: 2022-09-09T22:50:25.430 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.431 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:25.432 DEBUG:boto:Port: 8000 2022-09-09T22:50:25.433 DEBUG:boto:Params: {} 2022-09-09T22:50:25.434 DEBUG:boto:Token: None 2022-09-09T22:50:25.434 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/ 2022-09-09T22:50:25.435 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:gAB5vjnTVpmeAOe77boZqVVpjOw= 2022-09-09T22:50:25.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:gAB5vjnTVpmeAOe77boZqVVpjOw=', 'Content-Length': '0'} 2022-09-09T22:50:25.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067d565523689f273-00631bc331-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.443 DEBUG:boto:b'jkcgbj-23obj1000false' 2022-09-09T22:50:25.444 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:50:25.445 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:50:25.446 DEBUG:boto:Method: HEAD 2022-09-09T22:50:25.446 DEBUG:boto:Path: /jkcgbj-23/obj 2022-09-09T22:50:25.447 DEBUG:boto:Data: 2022-09-09T22:50:25.448 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.448 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:25.449 DEBUG:boto:Port: 8000 2022-09-09T22:50:25.449 DEBUG:boto:Params: {} 2022-09-09T22:50:25.450 DEBUG:boto:Token: None 2022-09-09T22:50:25.451 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/obj 2022-09-09T22:50:25.452 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:DB10SY8NRhVFj/ykBMKjp6+HB6Q= 2022-09-09T22:50:25.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:DB10SY8NRhVFj/ykBMKjp6+HB6Q=', 'Content-Length': '0'} 2022-09-09T22:50:25.458 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000d2003465cee2bf61-00631bc331-10e8-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.459 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:50:25.460 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:50:25.461 DEBUG:boto:Method: HEAD 2022-09-09T22:50:25.462 DEBUG:boto:Path: /jkcgbj-23/obj 2022-09-09T22:50:25.463 DEBUG:boto:Data: 2022-09-09T22:50:25.464 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.465 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:25.466 DEBUG:boto:Port: 8000 2022-09-09T22:50:25.467 DEBUG:boto:Params: {} 2022-09-09T22:50:25.469 DEBUG:boto:Token: None 2022-09-09T22:50:25.470 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/obj 2022-09-09T22:50:25.471 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:DB10SY8NRhVFj/ykBMKjp6+HB6Q= 2022-09-09T22:50:25.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:DB10SY8NRhVFj/ykBMKjp6+HB6Q=', 'Content-Length': '0'} 2022-09-09T22:50:25.477 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000006896eca476ba4893-00631bc331-10ff-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.479 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-23 zones={test-zone2, test-zone1} 2022-09-09T22:50:25.480 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-23 zones={test-zone3, test-zone1} 2022-09-09T22:50:25.481 DEBUG:boto:path=/jkcgbj-23/ 2022-09-09T22:50:25.482 DEBUG:boto:auth_path=/jkcgbj-23/ 2022-09-09T22:50:25.483 DEBUG:boto:Method: HEAD 2022-09-09T22:50:25.484 DEBUG:boto:Path: /jkcgbj-23/ 2022-09-09T22:50:25.485 DEBUG:boto:Data: 2022-09-09T22:50:25.486 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.487 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:25.488 DEBUG:boto:Port: 8001 2022-09-09T22:50:25.488 DEBUG:boto:Params: {} 2022-09-09T22:50:25.490 DEBUG:boto:Token: None 2022-09-09T22:50:25.491 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/ 2022-09-09T22:50:25.492 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:r+snpJdiprmGbmqCkJdWkNdNdbc= 2022-09-09T22:50:25.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:r+snpJdiprmGbmqCkJdWkNdNdbc=', 'Content-Length': '0'} 2022-09-09T22:50:25.498 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae75d6f5bb12839a-00631bc331-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.499 DEBUG:boto:path=/jkcgbj-23/ 2022-09-09T22:50:25.499 DEBUG:boto:auth_path=/jkcgbj-23/ 2022-09-09T22:50:25.500 DEBUG:boto:Method: HEAD 2022-09-09T22:50:25.500 DEBUG:boto:Path: /jkcgbj-23/ 2022-09-09T22:50:25.501 DEBUG:boto:Data: 2022-09-09T22:50:25.501 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.502 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:25.503 DEBUG:boto:Port: 8000 2022-09-09T22:50:25.503 DEBUG:boto:Params: {} 2022-09-09T22:50:25.504 DEBUG:boto:Token: None 2022-09-09T22:50:25.504 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/ 2022-09-09T22:50:25.505 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:r+snpJdiprmGbmqCkJdWkNdNdbc= 2022-09-09T22:50:25.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:r+snpJdiprmGbmqCkJdWkNdNdbc=', 'Content-Length': '0'} 2022-09-09T22:50:25.511 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000003051a39fbe58783-00631bc331-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.511 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:50:25.512 DEBUG:boto:path=/jkcgbj-23/ 2022-09-09T22:50:25.512 DEBUG:boto:auth_path=/jkcgbj-23/ 2022-09-09T22:50:25.513 DEBUG:boto:path=/jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.514 DEBUG:boto:auth_path=/jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.514 DEBUG:boto:Method: GET 2022-09-09T22:50:25.515 DEBUG:boto:Path: /jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.515 DEBUG:boto:Data: 2022-09-09T22:50:25.516 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.516 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:25.517 DEBUG:boto:Port: 8001 2022-09-09T22:50:25.517 DEBUG:boto:Params: {} 2022-09-09T22:50:25.518 DEBUG:boto:Token: None 2022-09-09T22:50:25.519 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/?versions 2022-09-09T22:50:25.519 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:MqVEbliwUAUBlSy04F37YX430tk= 2022-09-09T22:50:25.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:MqVEbliwUAUBlSy04F37YX430tk=', 'Content-Length': '0'} 2022-09-09T22:50:25.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f92c1a51183ae022-00631bc331-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.526 DEBUG:boto:b'jkcgbj-23999falseobj8Z5c4vwfWvpNpRLMwFGV-tguoijURW5true2022-09-09T22:49:29.528Zrgw-multisite-test-userMultisite Test UserNormalobjjqA7MkcivIkZnCalrNyyNmjK3EMR-GFfalse2022-09-09T22:49:29.481Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:25.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:25.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:25.528 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:50:25.529 DEBUG:boto:path=/jkcgbj-23/ 2022-09-09T22:50:25.530 DEBUG:boto:auth_path=/jkcgbj-23/ 2022-09-09T22:50:25.530 DEBUG:boto:path=/jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.531 DEBUG:boto:auth_path=/jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.531 DEBUG:boto:Method: GET 2022-09-09T22:50:25.532 DEBUG:boto:Path: /jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.532 DEBUG:boto:Data: 2022-09-09T22:50:25.533 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.533 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:25.534 DEBUG:boto:Port: 8000 2022-09-09T22:50:25.535 DEBUG:boto:Params: {} 2022-09-09T22:50:25.535 DEBUG:boto:Token: None 2022-09-09T22:50:25.536 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/?versions 2022-09-09T22:50:25.536 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:MqVEbliwUAUBlSy04F37YX430tk= 2022-09-09T22:50:25.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:MqVEbliwUAUBlSy04F37YX430tk=', 'Content-Length': '0'} 2022-09-09T22:50:25.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab7361c29a6386c6-00631bc331-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.586 DEBUG:boto:b'jkcgbj-23999falseobj8Z5c4vwfWvpNpRLMwFGV-tguoijURW5true2022-09-09T22:49:29.528Zrgw-multisite-test-userMultisite Test UserNormalobjjqA7MkcivIkZnCalrNyyNmjK3EMR-GFfalse2022-09-09T22:49:29.481Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:25.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:25.589 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:25.590 DEBUG:boto:path=/jkcgbj-23/ 2022-09-09T22:50:25.590 DEBUG:boto:auth_path=/jkcgbj-23/ 2022-09-09T22:50:25.591 DEBUG:boto:path=/jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.591 DEBUG:boto:auth_path=/jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.592 DEBUG:boto:Method: GET 2022-09-09T22:50:25.593 DEBUG:boto:Path: /jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.593 DEBUG:boto:Data: 2022-09-09T22:50:25.594 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.595 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:25.596 DEBUG:boto:Port: 8001 2022-09-09T22:50:25.597 DEBUG:boto:Params: {} 2022-09-09T22:50:25.597 DEBUG:boto:Token: None 2022-09-09T22:50:25.598 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/?versions 2022-09-09T22:50:25.599 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:MqVEbliwUAUBlSy04F37YX430tk= 2022-09-09T22:50:25.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:MqVEbliwUAUBlSy04F37YX430tk=', 'Content-Length': '0'} 2022-09-09T22:50:25.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c86f890c83626132-00631bc331-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.606 DEBUG:boto:b'jkcgbj-23999falseobj8Z5c4vwfWvpNpRLMwFGV-tguoijURW5true2022-09-09T22:49:29.528Zrgw-multisite-test-userMultisite Test UserNormalobjjqA7MkcivIkZnCalrNyyNmjK3EMR-GFfalse2022-09-09T22:49:29.481Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:25.607 DEBUG:boto:path=/jkcgbj-23/ 2022-09-09T22:50:25.608 DEBUG:boto:auth_path=/jkcgbj-23/ 2022-09-09T22:50:25.609 DEBUG:boto:path=/jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.609 DEBUG:boto:auth_path=/jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.610 DEBUG:boto:Method: GET 2022-09-09T22:50:25.610 DEBUG:boto:Path: /jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.611 DEBUG:boto:Data: 2022-09-09T22:50:25.612 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.612 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:25.613 DEBUG:boto:Port: 8000 2022-09-09T22:50:25.614 DEBUG:boto:Params: {} 2022-09-09T22:50:25.615 DEBUG:boto:Token: None 2022-09-09T22:50:25.616 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/?versions 2022-09-09T22:50:25.617 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:MqVEbliwUAUBlSy04F37YX430tk= 2022-09-09T22:50:25.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:MqVEbliwUAUBlSy04F37YX430tk=', 'Content-Length': '0'} 2022-09-09T22:50:25.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000817bb90c3d76f1d3-00631bc331-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.626 DEBUG:boto:b'jkcgbj-23999falseobj8Z5c4vwfWvpNpRLMwFGV-tguoijURW5true2022-09-09T22:49:29.528Zrgw-multisite-test-userMultisite Test UserNormalobjjqA7MkcivIkZnCalrNyyNmjK3EMR-GFfalse2022-09-09T22:49:29.481Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:25.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:50:25.627 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:50:25.628 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:50:25.628 DEBUG:boto:Method: HEAD 2022-09-09T22:50:25.629 DEBUG:boto:Path: /jkcgbj-23/obj 2022-09-09T22:50:25.629 DEBUG:boto:Data: 2022-09-09T22:50:25.630 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.630 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:25.631 DEBUG:boto:Port: 8001 2022-09-09T22:50:25.631 DEBUG:boto:Params: {} 2022-09-09T22:50:25.632 DEBUG:boto:Token: None 2022-09-09T22:50:25.633 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/obj 2022-09-09T22:50:25.633 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:DB10SY8NRhVFj/ykBMKjp6+HB6Q= 2022-09-09T22:50:25.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:DB10SY8NRhVFj/ykBMKjp6+HB6Q=', 'Content-Length': '0'} 2022-09-09T22:50:25.638 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000e528018c365ed3a8-00631bc331-1103-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.639 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:50:25.639 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:50:25.640 DEBUG:boto:Method: HEAD 2022-09-09T22:50:25.640 DEBUG:boto:Path: /jkcgbj-23/obj 2022-09-09T22:50:25.641 DEBUG:boto:Data: 2022-09-09T22:50:25.641 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.642 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:25.642 DEBUG:boto:Port: 8000 2022-09-09T22:50:25.643 DEBUG:boto:Params: {} 2022-09-09T22:50:25.643 DEBUG:boto:Token: None 2022-09-09T22:50:25.644 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/obj 2022-09-09T22:50:25.645 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:DB10SY8NRhVFj/ykBMKjp6+HB6Q= 2022-09-09T22:50:25.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:DB10SY8NRhVFj/ykBMKjp6+HB6Q=', 'Content-Length': '0'} 2022-09-09T22:50:25.651 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000032d4f87dacb8c436-00631bc331-10ff-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:50:25.653 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:50:25.654 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:50:25.655 DEBUG:boto:path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.656 DEBUG:boto:auth_path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.657 DEBUG:boto:Method: GET 2022-09-09T22:50:25.658 DEBUG:boto:Path: /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.659 DEBUG:boto:Data: 2022-09-09T22:50:25.660 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.661 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:25.662 DEBUG:boto:Port: 8001 2022-09-09T22:50:25.663 DEBUG:boto:Params: {} 2022-09-09T22:50:25.664 DEBUG:boto:Token: None 2022-09-09T22:50:25.665 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.666 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CLpJPcq1Pd0MEjpKKhMFjfBUPdc= 2022-09-09T22:50:25.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CLpJPcq1Pd0MEjpKKhMFjfBUPdc=', 'Content-Length': '0'} 2022-09-09T22:50:25.671 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'jqA7MkcivIkZnCalrNyyNmjK3EMR-GF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000426b7195ae2fa070-00631bc331-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.672 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:50:25.673 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:50:25.674 DEBUG:boto:path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.675 DEBUG:boto:auth_path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.676 DEBUG:boto:Method: GET 2022-09-09T22:50:25.677 DEBUG:boto:Path: /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.678 DEBUG:boto:Data: 2022-09-09T22:50:25.679 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.680 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:25.681 DEBUG:boto:Port: 8000 2022-09-09T22:50:25.682 DEBUG:boto:Params: {} 2022-09-09T22:50:25.684 DEBUG:boto:Token: None 2022-09-09T22:50:25.685 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.686 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CLpJPcq1Pd0MEjpKKhMFjfBUPdc= 2022-09-09T22:50:25.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CLpJPcq1Pd0MEjpKKhMFjfBUPdc=', 'Content-Length': '0'} 2022-09-09T22:50:25.691 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'jqA7MkcivIkZnCalrNyyNmjK3EMR-GF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007082daebfbdbc7de-00631bc331-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.692 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:50:25.693 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:50:25.694 DEBUG:boto:path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.695 DEBUG:boto:auth_path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.696 DEBUG:boto:Method: HEAD 2022-09-09T22:50:25.697 DEBUG:boto:Path: /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.698 DEBUG:boto:Data: 2022-09-09T22:50:25.699 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.700 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:25.701 DEBUG:boto:Port: 8001 2022-09-09T22:50:25.701 DEBUG:boto:Params: {} 2022-09-09T22:50:25.702 DEBUG:boto:Token: None 2022-09-09T22:50:25.703 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.704 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JcZPFLT/xIx0+JOLiKLTKBQcJ+c= 2022-09-09T22:50:25.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JcZPFLT/xIx0+JOLiKLTKBQcJ+c=', 'Content-Length': '0'} 2022-09-09T22:50:25.709 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'jqA7MkcivIkZnCalrNyyNmjK3EMR-GF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000730860d409fa3784-00631bc331-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.710 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:50:25.711 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:50:25.711 DEBUG:boto:path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.712 DEBUG:boto:auth_path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.712 DEBUG:boto:Method: HEAD 2022-09-09T22:50:25.713 DEBUG:boto:Path: /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.713 DEBUG:boto:Data: 2022-09-09T22:50:25.714 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.714 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:25.715 DEBUG:boto:Port: 8000 2022-09-09T22:50:25.715 DEBUG:boto:Params: {} 2022-09-09T22:50:25.716 DEBUG:boto:Token: None 2022-09-09T22:50:25.716 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.717 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:JcZPFLT/xIx0+JOLiKLTKBQcJ+c= 2022-09-09T22:50:25.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:JcZPFLT/xIx0+JOLiKLTKBQcJ+c=', 'Content-Length': '0'} 2022-09-09T22:50:25.720 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'jqA7MkcivIkZnCalrNyyNmjK3EMR-GF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003282758e5bde9dcc-00631bc331-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:50:25.721 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:50:25.722 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:50:25.723 DEBUG:boto:path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.723 DEBUG:boto:auth_path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.724 DEBUG:boto:Method: GET 2022-09-09T22:50:25.724 DEBUG:boto:Path: /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.725 DEBUG:boto:Data: 2022-09-09T22:50:25.725 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.726 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:25.726 DEBUG:boto:Port: 8001 2022-09-09T22:50:25.727 DEBUG:boto:Params: {} 2022-09-09T22:50:25.727 DEBUG:boto:Token: None 2022-09-09T22:50:25.728 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.729 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CLpJPcq1Pd0MEjpKKhMFjfBUPdc= 2022-09-09T22:50:25.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CLpJPcq1Pd0MEjpKKhMFjfBUPdc=', 'Content-Length': '0'} 2022-09-09T22:50:25.732 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'jqA7MkcivIkZnCalrNyyNmjK3EMR-GF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000029a621b90d48187e-00631bc331-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.733 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:50:25.734 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:50:25.734 DEBUG:boto:path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.735 DEBUG:boto:auth_path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.735 DEBUG:boto:Method: GET 2022-09-09T22:50:25.736 DEBUG:boto:Path: /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.736 DEBUG:boto:Data: 2022-09-09T22:50:25.737 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.737 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:25.738 DEBUG:boto:Port: 8000 2022-09-09T22:50:25.738 DEBUG:boto:Params: {} 2022-09-09T22:50:25.739 DEBUG:boto:Token: None 2022-09-09T22:50:25.740 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:25.740 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CLpJPcq1Pd0MEjpKKhMFjfBUPdc= 2022-09-09T22:50:25.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CLpJPcq1Pd0MEjpKKhMFjfBUPdc=', 'Content-Length': '0'} 2022-09-09T22:50:25.744 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'jqA7MkcivIkZnCalrNyyNmjK3EMR-GF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007aa977802d2512b5-00631bc331-10ff-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.745 DEBUG:boto:path=/jkcgbj-23/ 2022-09-09T22:50:25.746 DEBUG:boto:auth_path=/jkcgbj-23/ 2022-09-09T22:50:25.746 DEBUG:boto:path=/jkcgbj-23/?prefix=obj 2022-09-09T22:50:25.747 DEBUG:boto:auth_path=/jkcgbj-23/?prefix=obj 2022-09-09T22:50:25.747 DEBUG:boto:Method: GET 2022-09-09T22:50:25.748 DEBUG:boto:Path: /jkcgbj-23/?prefix=obj 2022-09-09T22:50:25.748 DEBUG:boto:Data: 2022-09-09T22:50:25.749 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.749 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:25.750 DEBUG:boto:Port: 8001 2022-09-09T22:50:25.750 DEBUG:boto:Params: {} 2022-09-09T22:50:25.751 DEBUG:boto:Token: None 2022-09-09T22:50:25.752 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/ 2022-09-09T22:50:25.752 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:gAB5vjnTVpmeAOe77boZqVVpjOw= 2022-09-09T22:50:25.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:gAB5vjnTVpmeAOe77boZqVVpjOw=', 'Content-Length': '0'} 2022-09-09T22:50:25.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a085fa5775d793fd-00631bc331-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.802 DEBUG:boto:b'jkcgbj-23obj1000false' 2022-09-09T22:50:25.803 DEBUG:boto:path=/jkcgbj-23/ 2022-09-09T22:50:25.804 DEBUG:boto:auth_path=/jkcgbj-23/ 2022-09-09T22:50:25.805 DEBUG:boto:path=/jkcgbj-23/?prefix=obj 2022-09-09T22:50:25.805 DEBUG:boto:auth_path=/jkcgbj-23/?prefix=obj 2022-09-09T22:50:25.806 DEBUG:boto:Method: GET 2022-09-09T22:50:25.806 DEBUG:boto:Path: /jkcgbj-23/?prefix=obj 2022-09-09T22:50:25.808 DEBUG:boto:Data: 2022-09-09T22:50:25.808 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.809 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:25.810 DEBUG:boto:Port: 8000 2022-09-09T22:50:25.810 DEBUG:boto:Params: {} 2022-09-09T22:50:25.811 DEBUG:boto:Token: None 2022-09-09T22:50:25.811 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/ 2022-09-09T22:50:25.812 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:gAB5vjnTVpmeAOe77boZqVVpjOw= 2022-09-09T22:50:25.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:gAB5vjnTVpmeAOe77boZqVVpjOw=', 'Content-Length': '0'} 2022-09-09T22:50:25.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e3c8da3d9fd96b9-00631bc331-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.820 DEBUG:boto:b'jkcgbj-23obj1000false' 2022-09-09T22:50:25.821 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:50:25.822 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:50:25.823 DEBUG:boto:Method: HEAD 2022-09-09T22:50:25.824 DEBUG:boto:Path: /jkcgbj-23/obj 2022-09-09T22:50:25.825 DEBUG:boto:Data: 2022-09-09T22:50:25.825 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.827 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:25.827 DEBUG:boto:Port: 8001 2022-09-09T22:50:25.828 DEBUG:boto:Params: {} 2022-09-09T22:50:25.829 DEBUG:boto:Token: None 2022-09-09T22:50:25.830 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/obj 2022-09-09T22:50:25.831 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:DB10SY8NRhVFj/ykBMKjp6+HB6Q= 2022-09-09T22:50:25.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:DB10SY8NRhVFj/ykBMKjp6+HB6Q=', 'Content-Length': '0'} 2022-09-09T22:50:25.837 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000769f3ff1c7c313df-00631bc331-1103-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.838 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:50:25.839 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:50:25.840 DEBUG:boto:Method: HEAD 2022-09-09T22:50:25.841 DEBUG:boto:Path: /jkcgbj-23/obj 2022-09-09T22:50:25.842 DEBUG:boto:Data: 2022-09-09T22:50:25.843 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.843 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:25.844 DEBUG:boto:Port: 8000 2022-09-09T22:50:25.844 DEBUG:boto:Params: {} 2022-09-09T22:50:25.845 DEBUG:boto:Token: None 2022-09-09T22:50:25.846 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/obj 2022-09-09T22:50:25.846 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:DB10SY8NRhVFj/ykBMKjp6+HB6Q= 2022-09-09T22:50:25.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:DB10SY8NRhVFj/ykBMKjp6+HB6Q=', 'Content-Length': '0'} 2022-09-09T22:50:25.852 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000004e8b1a9ce2d55f8b-00631bc331-10ff-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.853 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-23 zones={test-zone3, test-zone1} 2022-09-09T22:50:25.853 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-23 zones={test-zone3, test-zone2} 2022-09-09T22:50:25.854 DEBUG:boto:path=/jkcgbj-23/ 2022-09-09T22:50:25.854 DEBUG:boto:auth_path=/jkcgbj-23/ 2022-09-09T22:50:25.855 DEBUG:boto:Method: HEAD 2022-09-09T22:50:25.855 DEBUG:boto:Path: /jkcgbj-23/ 2022-09-09T22:50:25.856 DEBUG:boto:Data: 2022-09-09T22:50:25.856 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.857 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:25.858 DEBUG:boto:Port: 8001 2022-09-09T22:50:25.858 DEBUG:boto:Params: {} 2022-09-09T22:50:25.859 DEBUG:boto:Token: None 2022-09-09T22:50:25.859 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/ 2022-09-09T22:50:25.860 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:r+snpJdiprmGbmqCkJdWkNdNdbc= 2022-09-09T22:50:25.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:r+snpJdiprmGbmqCkJdWkNdNdbc=', 'Content-Length': '0'} 2022-09-09T22:50:25.867 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000764f379f3c5e7434-00631bc331-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.869 DEBUG:boto:path=/jkcgbj-23/ 2022-09-09T22:50:25.869 DEBUG:boto:auth_path=/jkcgbj-23/ 2022-09-09T22:50:25.870 DEBUG:boto:Method: HEAD 2022-09-09T22:50:25.871 DEBUG:boto:Path: /jkcgbj-23/ 2022-09-09T22:50:25.871 DEBUG:boto:Data: 2022-09-09T22:50:25.872 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.872 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:25.873 DEBUG:boto:Port: 8000 2022-09-09T22:50:25.873 DEBUG:boto:Params: {} 2022-09-09T22:50:25.874 DEBUG:boto:Token: None 2022-09-09T22:50:25.875 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/ 2022-09-09T22:50:25.876 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:r+snpJdiprmGbmqCkJdWkNdNdbc= 2022-09-09T22:50:25.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:r+snpJdiprmGbmqCkJdWkNdNdbc=', 'Content-Length': '0'} 2022-09-09T22:50:25.881 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018686604fa626721-00631bc331-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.882 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:50:25.883 DEBUG:boto:path=/jkcgbj-23/ 2022-09-09T22:50:25.884 DEBUG:boto:auth_path=/jkcgbj-23/ 2022-09-09T22:50:25.885 DEBUG:boto:path=/jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.885 DEBUG:boto:auth_path=/jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.886 DEBUG:boto:Method: GET 2022-09-09T22:50:25.887 DEBUG:boto:Path: /jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.887 DEBUG:boto:Data: 2022-09-09T22:50:25.888 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.888 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:25.889 DEBUG:boto:Port: 8001 2022-09-09T22:50:25.889 DEBUG:boto:Params: {} 2022-09-09T22:50:25.890 DEBUG:boto:Token: None 2022-09-09T22:50:25.891 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/?versions 2022-09-09T22:50:25.891 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:MqVEbliwUAUBlSy04F37YX430tk= 2022-09-09T22:50:25.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:MqVEbliwUAUBlSy04F37YX430tk=', 'Content-Length': '0'} 2022-09-09T22:50:25.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec2d2f6f6459891f-00631bc331-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.938 DEBUG:boto:b'jkcgbj-23999falseobj8Z5c4vwfWvpNpRLMwFGV-tguoijURW5true2022-09-09T22:49:29.528Zrgw-multisite-test-userMultisite Test UserNormalobjjqA7MkcivIkZnCalrNyyNmjK3EMR-GFfalse2022-09-09T22:49:29.481Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:25.940 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:25.940 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:25.941 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:50:25.942 DEBUG:boto:path=/jkcgbj-23/ 2022-09-09T22:50:25.943 DEBUG:boto:auth_path=/jkcgbj-23/ 2022-09-09T22:50:25.943 DEBUG:boto:path=/jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.944 DEBUG:boto:auth_path=/jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.944 DEBUG:boto:Method: GET 2022-09-09T22:50:25.945 DEBUG:boto:Path: /jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.945 DEBUG:boto:Data: 2022-09-09T22:50:25.946 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.946 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:25.947 DEBUG:boto:Port: 8000 2022-09-09T22:50:25.947 DEBUG:boto:Params: {} 2022-09-09T22:50:25.948 DEBUG:boto:Token: None 2022-09-09T22:50:25.949 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/?versions 2022-09-09T22:50:25.949 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:MqVEbliwUAUBlSy04F37YX430tk= 2022-09-09T22:50:25.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:MqVEbliwUAUBlSy04F37YX430tk=', 'Content-Length': '0'} 2022-09-09T22:50:25.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fed132acb57504c-00631bc331-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.960 DEBUG:boto:b'jkcgbj-23999falseobj8Z5c4vwfWvpNpRLMwFGV-tguoijURW5true2022-09-09T22:49:29.528Zrgw-multisite-test-userMultisite Test UserNormalobjjqA7MkcivIkZnCalrNyyNmjK3EMR-GFfalse2022-09-09T22:49:29.481Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:25.962 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:25.962 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:50:25.963 DEBUG:boto:path=/jkcgbj-23/ 2022-09-09T22:50:25.964 DEBUG:boto:auth_path=/jkcgbj-23/ 2022-09-09T22:50:25.964 DEBUG:boto:path=/jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.965 DEBUG:boto:auth_path=/jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.965 DEBUG:boto:Method: GET 2022-09-09T22:50:25.966 DEBUG:boto:Path: /jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.966 DEBUG:boto:Data: 2022-09-09T22:50:25.967 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.967 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:25.968 DEBUG:boto:Port: 8001 2022-09-09T22:50:25.968 DEBUG:boto:Params: {} 2022-09-09T22:50:25.969 DEBUG:boto:Token: None 2022-09-09T22:50:25.970 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/?versions 2022-09-09T22:50:25.970 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:MqVEbliwUAUBlSy04F37YX430tk= 2022-09-09T22:50:25.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:MqVEbliwUAUBlSy04F37YX430tk=', 'Content-Length': '0'} 2022-09-09T22:50:25.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c6c76ff183f311a-00631bc331-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:25.978 DEBUG:boto:b'jkcgbj-23999falseobj8Z5c4vwfWvpNpRLMwFGV-tguoijURW5true2022-09-09T22:49:29.528Zrgw-multisite-test-userMultisite Test UserNormalobjjqA7MkcivIkZnCalrNyyNmjK3EMR-GFfalse2022-09-09T22:49:29.481Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:25.980 DEBUG:boto:path=/jkcgbj-23/ 2022-09-09T22:50:25.981 DEBUG:boto:auth_path=/jkcgbj-23/ 2022-09-09T22:50:25.982 DEBUG:boto:path=/jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.983 DEBUG:boto:auth_path=/jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.984 DEBUG:boto:Method: GET 2022-09-09T22:50:25.984 DEBUG:boto:Path: /jkcgbj-23/?versions&max-keys=999 2022-09-09T22:50:25.985 DEBUG:boto:Data: 2022-09-09T22:50:25.986 DEBUG:boto:Headers: {} 2022-09-09T22:50:25.987 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:25.988 DEBUG:boto:Port: 8000 2022-09-09T22:50:25.989 DEBUG:boto:Params: {} 2022-09-09T22:50:25.990 DEBUG:boto:Token: None 2022-09-09T22:50:25.991 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:25 GMT /jkcgbj-23/?versions 2022-09-09T22:50:25.992 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:MqVEbliwUAUBlSy04F37YX430tk= 2022-09-09T22:50:25.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:25 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:MqVEbliwUAUBlSy04F37YX430tk=', 'Content-Length': '0'} 2022-09-09T22:50:25.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000062e10695bb6c156-00631bc331-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:26.000 DEBUG:boto:b'jkcgbj-23999falseobj8Z5c4vwfWvpNpRLMwFGV-tguoijURW5true2022-09-09T22:49:29.528Zrgw-multisite-test-userMultisite Test UserNormalobjjqA7MkcivIkZnCalrNyyNmjK3EMR-GFfalse2022-09-09T22:49:29.481Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:50:26.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:50:26.002 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:50:26.003 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:50:26.004 DEBUG:boto:Method: HEAD 2022-09-09T22:50:26.006 DEBUG:boto:Path: /jkcgbj-23/obj 2022-09-09T22:50:26.007 DEBUG:boto:Data: 2022-09-09T22:50:26.007 DEBUG:boto:Headers: {} 2022-09-09T22:50:26.008 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:26.009 DEBUG:boto:Port: 8001 2022-09-09T22:50:26.010 DEBUG:boto:Params: {} 2022-09-09T22:50:26.011 DEBUG:boto:Token: None 2022-09-09T22:50:26.012 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:26 GMT /jkcgbj-23/obj 2022-09-09T22:50:26.012 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:z4EQv+D822s+E8rMcxM1OYVD3Ck= 2022-09-09T22:50:26.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:z4EQv+D822s+E8rMcxM1OYVD3Ck=', 'Content-Length': '0'} 2022-09-09T22:50:26.017 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000009bddf0cba0626a64-00631bc332-1103-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:26.018 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:50:26.018 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:50:26.019 DEBUG:boto:Method: HEAD 2022-09-09T22:50:26.019 DEBUG:boto:Path: /jkcgbj-23/obj 2022-09-09T22:50:26.020 DEBUG:boto:Data: 2022-09-09T22:50:26.020 DEBUG:boto:Headers: {} 2022-09-09T22:50:26.021 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:26.021 DEBUG:boto:Port: 8000 2022-09-09T22:50:26.022 DEBUG:boto:Params: {} 2022-09-09T22:50:26.022 DEBUG:boto:Token: None 2022-09-09T22:50:26.023 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:26 GMT /jkcgbj-23/obj 2022-09-09T22:50:26.024 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:z4EQv+D822s+E8rMcxM1OYVD3Ck= 2022-09-09T22:50:26.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:z4EQv+D822s+E8rMcxM1OYVD3Ck=', 'Content-Length': '0'} 2022-09-09T22:50:26.029 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000a446f0c6c9b9e292-00631bc332-10e8-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:26.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:50:26.030 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:50:26.031 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:50:26.031 DEBUG:boto:path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:26.032 DEBUG:boto:auth_path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:26.033 DEBUG:boto:Method: GET 2022-09-09T22:50:26.033 DEBUG:boto:Path: /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:26.034 DEBUG:boto:Data: 2022-09-09T22:50:26.034 DEBUG:boto:Headers: {} 2022-09-09T22:50:26.035 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:26.035 DEBUG:boto:Port: 8001 2022-09-09T22:50:26.036 DEBUG:boto:Params: {} 2022-09-09T22:50:26.037 DEBUG:boto:Token: None 2022-09-09T22:50:26.038 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:26 GMT /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:26.039 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rbBkO1FP3BIaPX6eqe5VVzee/o0= 2022-09-09T22:50:26.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rbBkO1FP3BIaPX6eqe5VVzee/o0=', 'Content-Length': '0'} 2022-09-09T22:50:26.042 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'jqA7MkcivIkZnCalrNyyNmjK3EMR-GF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a129d7bb7f800133-00631bc332-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:26.043 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:50:26.044 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:50:26.044 DEBUG:boto:path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:26.045 DEBUG:boto:auth_path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:26.045 DEBUG:boto:Method: GET 2022-09-09T22:50:26.046 DEBUG:boto:Path: /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:26.047 DEBUG:boto:Data: 2022-09-09T22:50:26.047 DEBUG:boto:Headers: {} 2022-09-09T22:50:26.048 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:26.048 DEBUG:boto:Port: 8000 2022-09-09T22:50:26.049 DEBUG:boto:Params: {} 2022-09-09T22:50:26.050 DEBUG:boto:Token: None 2022-09-09T22:50:26.050 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:26 GMT /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:26.051 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rbBkO1FP3BIaPX6eqe5VVzee/o0= 2022-09-09T22:50:26.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rbBkO1FP3BIaPX6eqe5VVzee/o0=', 'Content-Length': '0'} 2022-09-09T22:50:26.055 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'jqA7MkcivIkZnCalrNyyNmjK3EMR-GF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008522441284bf78f3-00631bc332-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:26.056 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:50:26.057 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:50:26.057 DEBUG:boto:path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:26.058 DEBUG:boto:auth_path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:26.058 DEBUG:boto:Method: HEAD 2022-09-09T22:50:26.059 DEBUG:boto:Path: /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:26.059 DEBUG:boto:Data: 2022-09-09T22:50:26.060 DEBUG:boto:Headers: {} 2022-09-09T22:50:26.060 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:26.061 DEBUG:boto:Port: 8001 2022-09-09T22:50:26.061 DEBUG:boto:Params: {} 2022-09-09T22:50:26.062 DEBUG:boto:Token: None 2022-09-09T22:50:26.063 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:26 GMT /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:26.063 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:5HbB0FgqT9h4TsJUWfPqr+ba1gE= 2022-09-09T22:50:26.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:5HbB0FgqT9h4TsJUWfPqr+ba1gE=', 'Content-Length': '0'} 2022-09-09T22:50:26.067 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'jqA7MkcivIkZnCalrNyyNmjK3EMR-GF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000049ba06f10154a6e2-00631bc332-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:26.068 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:50:26.068 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:50:26.069 DEBUG:boto:path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:26.070 DEBUG:boto:auth_path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:26.070 DEBUG:boto:Method: HEAD 2022-09-09T22:50:26.071 DEBUG:boto:Path: /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:26.071 DEBUG:boto:Data: 2022-09-09T22:50:26.072 DEBUG:boto:Headers: {} 2022-09-09T22:50:26.072 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:26.073 DEBUG:boto:Port: 8000 2022-09-09T22:50:26.073 DEBUG:boto:Params: {} 2022-09-09T22:50:26.074 DEBUG:boto:Token: None 2022-09-09T22:50:26.074 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:26 GMT /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:26.075 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:5HbB0FgqT9h4TsJUWfPqr+ba1gE= 2022-09-09T22:50:26.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:5HbB0FgqT9h4TsJUWfPqr+ba1gE=', 'Content-Length': '0'} 2022-09-09T22:50:26.079 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'jqA7MkcivIkZnCalrNyyNmjK3EMR-GF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006b151aa440bd557c-00631bc332-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:26.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:50:26.081 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:50:26.081 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:50:26.082 DEBUG:boto:path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:26.082 DEBUG:boto:auth_path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:26.083 DEBUG:boto:Method: GET 2022-09-09T22:50:26.083 DEBUG:boto:Path: /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:26.084 DEBUG:boto:Data: 2022-09-09T22:50:26.084 DEBUG:boto:Headers: {} 2022-09-09T22:50:26.085 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:26.085 DEBUG:boto:Port: 8001 2022-09-09T22:50:26.086 DEBUG:boto:Params: {} 2022-09-09T22:50:26.087 DEBUG:boto:Token: None 2022-09-09T22:50:26.087 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:26 GMT /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:26.088 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rbBkO1FP3BIaPX6eqe5VVzee/o0= 2022-09-09T22:50:26.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rbBkO1FP3BIaPX6eqe5VVzee/o0=', 'Content-Length': '0'} 2022-09-09T22:50:26.092 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'jqA7MkcivIkZnCalrNyyNmjK3EMR-GF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b5694e7253980bde-00631bc332-1103-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:26.093 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:50:26.095 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:50:26.096 DEBUG:boto:path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:26.097 DEBUG:boto:auth_path=/jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:26.098 DEBUG:boto:Method: GET 2022-09-09T22:50:26.100 DEBUG:boto:Path: /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:26.101 DEBUG:boto:Data: 2022-09-09T22:50:26.101 DEBUG:boto:Headers: {} 2022-09-09T22:50:26.102 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:26.103 DEBUG:boto:Port: 8000 2022-09-09T22:50:26.104 DEBUG:boto:Params: {} 2022-09-09T22:50:26.104 DEBUG:boto:Token: None 2022-09-09T22:50:26.105 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:26 GMT /jkcgbj-23/obj?versionId=jqA7MkcivIkZnCalrNyyNmjK3EMR-GF 2022-09-09T22:50:26.106 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:rbBkO1FP3BIaPX6eqe5VVzee/o0= 2022-09-09T22:50:26.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:rbBkO1FP3BIaPX6eqe5VVzee/o0=', 'Content-Length': '0'} 2022-09-09T22:50:26.110 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 22:49:29 GMT'), ('x-amz-version-id', 'jqA7MkcivIkZnCalrNyyNmjK3EMR-GF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c97df85a3483372e-00631bc332-10e8-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 22:50:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:26.111 DEBUG:boto:path=/jkcgbj-23/ 2022-09-09T22:50:26.112 DEBUG:boto:auth_path=/jkcgbj-23/ 2022-09-09T22:50:26.112 DEBUG:boto:path=/jkcgbj-23/?prefix=obj 2022-09-09T22:50:26.113 DEBUG:boto:auth_path=/jkcgbj-23/?prefix=obj 2022-09-09T22:50:26.113 DEBUG:boto:Method: GET 2022-09-09T22:50:26.114 DEBUG:boto:Path: /jkcgbj-23/?prefix=obj 2022-09-09T22:50:26.114 DEBUG:boto:Data: 2022-09-09T22:50:26.115 DEBUG:boto:Headers: {} 2022-09-09T22:50:26.115 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:26.116 DEBUG:boto:Port: 8001 2022-09-09T22:50:26.117 DEBUG:boto:Params: {} 2022-09-09T22:50:26.117 DEBUG:boto:Token: None 2022-09-09T22:50:26.118 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:26 GMT /jkcgbj-23/ 2022-09-09T22:50:26.120 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:oHcc5uABZU6lqEKDptzV6eDvTDc= 2022-09-09T22:50:26.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:oHcc5uABZU6lqEKDptzV6eDvTDc=', 'Content-Length': '0'} 2022-09-09T22:50:26.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b767505961fdcfaf-00631bc332-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:26.170 DEBUG:boto:b'jkcgbj-23obj1000false' 2022-09-09T22:50:26.171 DEBUG:boto:path=/jkcgbj-23/ 2022-09-09T22:50:26.174 DEBUG:boto:auth_path=/jkcgbj-23/ 2022-09-09T22:50:26.175 DEBUG:boto:path=/jkcgbj-23/?prefix=obj 2022-09-09T22:50:26.176 DEBUG:boto:auth_path=/jkcgbj-23/?prefix=obj 2022-09-09T22:50:26.177 DEBUG:boto:Method: GET 2022-09-09T22:50:26.179 DEBUG:boto:Path: /jkcgbj-23/?prefix=obj 2022-09-09T22:50:26.180 DEBUG:boto:Data: 2022-09-09T22:50:26.181 DEBUG:boto:Headers: {} 2022-09-09T22:50:26.182 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:26.182 DEBUG:boto:Port: 8000 2022-09-09T22:50:26.183 DEBUG:boto:Params: {} 2022-09-09T22:50:26.183 DEBUG:boto:Token: None 2022-09-09T22:50:26.184 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:50:26 GMT /jkcgbj-23/ 2022-09-09T22:50:26.185 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:oHcc5uABZU6lqEKDptzV6eDvTDc= 2022-09-09T22:50:26.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:oHcc5uABZU6lqEKDptzV6eDvTDc=', 'Content-Length': '0'} 2022-09-09T22:50:26.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9274410dd31cf25-00631bc332-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:26.192 DEBUG:boto:b'jkcgbj-23obj1000false' 2022-09-09T22:50:26.193 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:50:26.193 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:50:26.194 DEBUG:boto:Method: HEAD 2022-09-09T22:50:26.194 DEBUG:boto:Path: /jkcgbj-23/obj 2022-09-09T22:50:26.195 DEBUG:boto:Data: 2022-09-09T22:50:26.195 DEBUG:boto:Headers: {} 2022-09-09T22:50:26.196 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:26.196 DEBUG:boto:Port: 8001 2022-09-09T22:50:26.197 DEBUG:boto:Params: {} 2022-09-09T22:50:26.197 DEBUG:boto:Token: None 2022-09-09T22:50:26.198 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:26 GMT /jkcgbj-23/obj 2022-09-09T22:50:26.199 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:z4EQv+D822s+E8rMcxM1OYVD3Ck= 2022-09-09T22:50:26.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:z4EQv+D822s+E8rMcxM1OYVD3Ck=', 'Content-Length': '0'} 2022-09-09T22:50:26.204 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000f9e1e8f7d5881223-00631bc332-1103-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:26.205 DEBUG:boto:path=/jkcgbj-23/obj 2022-09-09T22:50:26.206 DEBUG:boto:auth_path=/jkcgbj-23/obj 2022-09-09T22:50:26.206 DEBUG:boto:Method: HEAD 2022-09-09T22:50:26.207 DEBUG:boto:Path: /jkcgbj-23/obj 2022-09-09T22:50:26.207 DEBUG:boto:Data: 2022-09-09T22:50:26.208 DEBUG:boto:Headers: {} 2022-09-09T22:50:26.209 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:26.209 DEBUG:boto:Port: 8000 2022-09-09T22:50:26.210 DEBUG:boto:Params: {} 2022-09-09T22:50:26.210 DEBUG:boto:Token: None 2022-09-09T22:50:26.211 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:26 GMT /jkcgbj-23/obj 2022-09-09T22:50:26.212 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:z4EQv+D822s+E8rMcxM1OYVD3Ck= 2022-09-09T22:50:26.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:z4EQv+D822s+E8rMcxM1OYVD3Ck=', 'Content-Length': '0'} 2022-09-09T22:50:26.216 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000007ff2a5f22710db15-00631bc332-10e8-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:50:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:26.217 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-23 zones={test-zone3, test-zone2} 2022-09-09T22:50:26.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2022-09-09T22:50:26.219 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jkcgbj-24 2022-09-09T22:50:26.220 DEBUG:boto:path=/jkcgbj-24/ 2022-09-09T22:50:26.220 DEBUG:boto:auth_path=/jkcgbj-24/ 2022-09-09T22:50:26.221 DEBUG:boto:Method: PUT 2022-09-09T22:50:26.221 DEBUG:boto:Path: /jkcgbj-24/ 2022-09-09T22:50:26.223 DEBUG:boto:Data: 2022-09-09T22:50:26.224 DEBUG:boto:Headers: {} 2022-09-09T22:50:26.225 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:26.225 DEBUG:boto:Port: 8000 2022-09-09T22:50:26.226 DEBUG:boto:Params: {} 2022-09-09T22:50:26.228 DEBUG:boto:Token: None 2022-09-09T22:50:26.229 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:50:26 GMT /jkcgbj-24/ 2022-09-09T22:50:26.229 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:RtPLShONSXZQfi9RAA9nfuPvNm8= 2022-09-09T22:50:26.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:RtPLShONSXZQfi9RAA9nfuPvNm8=', 'Content-Length': '0'} 2022-09-09T22:50:26.257 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007354e0494550304a-00631bc332-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:26.258 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jkcgbj-25 2022-09-09T22:50:26.259 DEBUG:boto:path=/jkcgbj-25/ 2022-09-09T22:50:26.259 DEBUG:boto:auth_path=/jkcgbj-25/ 2022-09-09T22:50:26.260 DEBUG:boto:Method: PUT 2022-09-09T22:50:26.260 DEBUG:boto:Path: /jkcgbj-25/ 2022-09-09T22:50:26.261 DEBUG:boto:Data: 2022-09-09T22:50:26.261 DEBUG:boto:Headers: {} 2022-09-09T22:50:26.262 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:26.262 DEBUG:boto:Port: 8000 2022-09-09T22:50:26.263 DEBUG:boto:Params: {} 2022-09-09T22:50:26.263 DEBUG:boto:Token: None 2022-09-09T22:50:26.264 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:50:26 GMT /jkcgbj-25/ 2022-09-09T22:50:26.265 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:NhGDMqXsbGuNGR8TGBPaol0ptkk= 2022-09-09T22:50:26.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:NhGDMqXsbGuNGR8TGBPaol0ptkk=', 'Content-Length': '0'} 2022-09-09T22:50:26.318 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000861e0b28246706ac-00631bc332-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:26.319 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jkcgbj-26 2022-09-09T22:50:26.320 DEBUG:boto:path=/jkcgbj-26/ 2022-09-09T22:50:26.320 DEBUG:boto:auth_path=/jkcgbj-26/ 2022-09-09T22:50:26.321 DEBUG:boto:Method: PUT 2022-09-09T22:50:26.322 DEBUG:boto:Path: /jkcgbj-26/ 2022-09-09T22:50:26.322 DEBUG:boto:Data: 2022-09-09T22:50:26.323 DEBUG:boto:Headers: {} 2022-09-09T22:50:26.324 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:26.324 DEBUG:boto:Port: 8001 2022-09-09T22:50:26.325 DEBUG:boto:Params: {} 2022-09-09T22:50:26.326 DEBUG:boto:Token: None 2022-09-09T22:50:26.327 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:50:26 GMT /jkcgbj-26/ 2022-09-09T22:50:26.328 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OZMzJ12cXqqxgWIixQtvVo8SzAg= 2022-09-09T22:50:26.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OZMzJ12cXqqxgWIixQtvVo8SzAg=', 'Content-Length': '0'} 2022-09-09T22:50:26.376 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000010c1b036f52967bc-00631bc332-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:26.377 DEBUG:boto:path=/jkcgbj-24/ 2022-09-09T22:50:26.377 DEBUG:boto:auth_path=/jkcgbj-24/ 2022-09-09T22:50:26.378 DEBUG:boto:path=/jkcgbj-24/?versioning 2022-09-09T22:50:26.378 DEBUG:boto:auth_path=/jkcgbj-24/?versioning 2022-09-09T22:50:26.379 DEBUG:boto:Method: PUT 2022-09-09T22:50:26.379 DEBUG:boto:Path: /jkcgbj-24/?versioning 2022-09-09T22:50:26.380 DEBUG:boto:Data: Enabled Disabled 2022-09-09T22:50:26.380 DEBUG:boto:Headers: {} 2022-09-09T22:50:26.381 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:26.381 DEBUG:boto:Port: 8000 2022-09-09T22:50:26.382 DEBUG:boto:Params: {} 2022-09-09T22:50:26.382 DEBUG:boto:Token: None 2022-09-09T22:50:26.383 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:50:26 GMT /jkcgbj-24/?versioning 2022-09-09T22:50:26.383 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CyY+nxCnI7zxwyrbOzANJ9U9aSY= 2022-09-09T22:50:26.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CyY+nxCnI7zxwyrbOzANJ9U9aSY=', 'Content-Length': '228'} 2022-09-09T22:50:26.395 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000453c5acd65ea73de-00631bc332-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:26.396 DEBUG:boto:path=/jkcgbj-24/ 2022-09-09T22:50:26.396 DEBUG:boto:auth_path=/jkcgbj-24/ 2022-09-09T22:50:26.397 DEBUG:boto:path=/jkcgbj-24/?versioning 2022-09-09T22:50:26.397 DEBUG:boto:auth_path=/jkcgbj-24/?versioning 2022-09-09T22:50:26.398 DEBUG:boto:Method: PUT 2022-09-09T22:50:26.398 DEBUG:boto:Path: /jkcgbj-24/?versioning 2022-09-09T22:50:26.399 DEBUG:boto:Data: Suspended Disabled 2022-09-09T22:50:26.399 DEBUG:boto:Headers: {} 2022-09-09T22:50:26.400 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:26.400 DEBUG:boto:Port: 8000 2022-09-09T22:50:26.401 DEBUG:boto:Params: {} 2022-09-09T22:50:26.401 DEBUG:boto:Token: None 2022-09-09T22:50:26.402 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:50:26 GMT /jkcgbj-24/?versioning 2022-09-09T22:50:26.402 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CyY+nxCnI7zxwyrbOzANJ9U9aSY= 2022-09-09T22:50:26.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CyY+nxCnI7zxwyrbOzANJ9U9aSY=', 'Content-Length': '230'} 2022-09-09T22:50:26.416 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000092608f88b95efa4a-00631bc332-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:26.417 DEBUG:boto:path=/jkcgbj-25/ 2022-09-09T22:50:26.417 DEBUG:boto:auth_path=/jkcgbj-25/ 2022-09-09T22:50:26.418 DEBUG:boto:path=/jkcgbj-25/?versioning 2022-09-09T22:50:26.418 DEBUG:boto:auth_path=/jkcgbj-25/?versioning 2022-09-09T22:50:26.419 DEBUG:boto:Method: PUT 2022-09-09T22:50:26.419 DEBUG:boto:Path: /jkcgbj-25/?versioning 2022-09-09T22:50:26.420 DEBUG:boto:Data: Enabled Disabled 2022-09-09T22:50:26.420 DEBUG:boto:Headers: {} 2022-09-09T22:50:26.421 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:26.421 DEBUG:boto:Port: 8000 2022-09-09T22:50:26.422 DEBUG:boto:Params: {} 2022-09-09T22:50:26.422 DEBUG:boto:Token: None 2022-09-09T22:50:26.423 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:50:26 GMT /jkcgbj-25/?versioning 2022-09-09T22:50:26.423 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9LBF/YflHViCt3YlxzEJfnByyKk= 2022-09-09T22:50:26.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9LBF/YflHViCt3YlxzEJfnByyKk=', 'Content-Length': '228'} 2022-09-09T22:50:26.445 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004298b79ae61a6d10-00631bc332-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:26.446 DEBUG:boto:path=/jkcgbj-25/ 2022-09-09T22:50:26.446 DEBUG:boto:auth_path=/jkcgbj-25/ 2022-09-09T22:50:26.447 DEBUG:boto:path=/jkcgbj-25/?versioning 2022-09-09T22:50:26.448 DEBUG:boto:auth_path=/jkcgbj-25/?versioning 2022-09-09T22:50:26.448 DEBUG:boto:Method: PUT 2022-09-09T22:50:26.449 DEBUG:boto:Path: /jkcgbj-25/?versioning 2022-09-09T22:50:26.449 DEBUG:boto:Data: Suspended Disabled 2022-09-09T22:50:26.450 DEBUG:boto:Headers: {} 2022-09-09T22:50:26.450 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:26.451 DEBUG:boto:Port: 8000 2022-09-09T22:50:26.451 DEBUG:boto:Params: {} 2022-09-09T22:50:26.452 DEBUG:boto:Token: None 2022-09-09T22:50:26.453 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:50:26 GMT /jkcgbj-25/?versioning 2022-09-09T22:50:26.454 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9LBF/YflHViCt3YlxzEJfnByyKk= 2022-09-09T22:50:26.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9LBF/YflHViCt3YlxzEJfnByyKk=', 'Content-Length': '230'} 2022-09-09T22:50:26.496 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000da665dae038a8ece-00631bc332-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:26.497 DEBUG:boto:path=/jkcgbj-26/ 2022-09-09T22:50:26.498 DEBUG:boto:auth_path=/jkcgbj-26/ 2022-09-09T22:50:26.498 DEBUG:boto:path=/jkcgbj-26/?versioning 2022-09-09T22:50:26.499 DEBUG:boto:auth_path=/jkcgbj-26/?versioning 2022-09-09T22:50:26.501 DEBUG:boto:Method: PUT 2022-09-09T22:50:26.502 DEBUG:boto:Path: /jkcgbj-26/?versioning 2022-09-09T22:50:26.503 DEBUG:boto:Data: Enabled Disabled 2022-09-09T22:50:26.504 DEBUG:boto:Headers: {} 2022-09-09T22:50:26.504 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:26.505 DEBUG:boto:Port: 8001 2022-09-09T22:50:26.506 DEBUG:boto:Params: {} 2022-09-09T22:50:26.507 DEBUG:boto:Token: None 2022-09-09T22:50:26.508 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:50:26 GMT /jkcgbj-26/?versioning 2022-09-09T22:50:26.509 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Fu4OM3tltizWmRIXIRPRuvkrj2o= 2022-09-09T22:50:26.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Fu4OM3tltizWmRIXIRPRuvkrj2o=', 'Content-Length': '228'} 2022-09-09T22:50:26.525 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000051d7ae15066e6b9d-00631bc332-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:26.527 DEBUG:boto:path=/jkcgbj-26/ 2022-09-09T22:50:26.528 DEBUG:boto:auth_path=/jkcgbj-26/ 2022-09-09T22:50:26.528 DEBUG:boto:path=/jkcgbj-26/?versioning 2022-09-09T22:50:26.529 DEBUG:boto:auth_path=/jkcgbj-26/?versioning 2022-09-09T22:50:26.530 DEBUG:boto:Method: PUT 2022-09-09T22:50:26.531 DEBUG:boto:Path: /jkcgbj-26/?versioning 2022-09-09T22:50:26.532 DEBUG:boto:Data: Suspended Disabled 2022-09-09T22:50:26.533 DEBUG:boto:Headers: {} 2022-09-09T22:50:26.534 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:26.535 DEBUG:boto:Port: 8001 2022-09-09T22:50:26.535 DEBUG:boto:Params: {} 2022-09-09T22:50:26.536 DEBUG:boto:Token: None 2022-09-09T22:50:26.537 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:50:26 GMT /jkcgbj-26/?versioning 2022-09-09T22:50:26.539 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Fu4OM3tltizWmRIXIRPRuvkrj2o= 2022-09-09T22:50:26.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:26 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Fu4OM3tltizWmRIXIRPRuvkrj2o=', 'Content-Length': '230'} 2022-09-09T22:50:26.558 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000099e38409685770a3-00631bc332-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:26.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:50:26.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:50:26.561 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:50:26.612 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:50:26.613 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:50:26.670 INFO:teuthology.orchestra.run.smithi005.stderr:No --period given, using current period=503b1d61-4513-48c8-a308-17b76219610c 2022-09-09T22:50:26.688 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T22:50:26.689 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:50:26.689 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:50:26.690 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:28:06.146147Z" 2022-09-09T22:50:26.691 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:50:26.691 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:50:26.692 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:50:26.693 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:50:26.694 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:50:26.694 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:50:26.695 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:50:26.695 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:30:37.757014Z" 2022-09-09T22:50:26.696 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:50:26.697 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:50:26.697 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662763826.550710_156.1", 2022-09-09T22:50:26.698 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:50:26.550710Z" 2022-09-09T22:50:26.698 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:50:26.699 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T22:50:26.701 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763826.550710_156.1'} 2022-09-09T22:50:26.701 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-09T22:50:26.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:50:26.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:50:26.703 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:50:26.744 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:50:26.744 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:50:26.799 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:50:26.791+0000 7fbe394f9dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:50:26.802 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:50:26.795+0000 7fbe394f9dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:50:26.822 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:50:26.822 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:50:26.822 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:50:26.823 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:50:26.823 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:50:26.823 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:50:26.824 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:50:26.824 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:50:26.824 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:50:26.824 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:50:26.825 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:50:26.825 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:50:26.825 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:50:26.825 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:50:26.826 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:50:26.826 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:50:26.826 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:50:26.826 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:50:26.827 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:50:26.827 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:50:26.827 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:50:26.827 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:50:26.828 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:50:26.828 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:50:26.828 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:50:26.828 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:50:26.829 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:50:26.829 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:50:26.829 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:50:26.829 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:50:26.830 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:50:26.830 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:50:26.830 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:50:26.830 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:50:26.831 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:50:26.831 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:50:26.831 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:50:26.831 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:50:26.832 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:50:26.832 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:50:26.832 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:50:26.832 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:50:26.833 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:50:26.833 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:50:26.833 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:50:26.833 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:50:26.834 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:50:26.834 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:50:26.834 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:50:26.834 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763826.282622_137.1", 2022-09-09T22:50:26.835 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:50:26.835 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:50:26.835 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:50:26.835 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.282622Z", 2022-09-09T22:50:26.836 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:50:26.836 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:50:26.836 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:50:26.836 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:50:26.836 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:50:26.837 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:50:26.837 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:50:26.837 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:50:26.837 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:50:26.838 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:50:26Z" 2022-09-09T22:50:26.838 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:50:26.839 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763826.282622_137.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:50:26.282622Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:50:26Z" } 2022-09-09T22:50:26.840 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763826.282622_137.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:50:26.282622Z', 'realm_epoch': 2}}] 2022-09-09T22:50:26.840 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763826.550710_156.1'} 2022-09-09T22:50:26.841 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763826.282622_137.1'} 2022-09-09T22:50:26.841 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763826.550710_156.1 target=1_1662763826.282622_137.1 2022-09-09T22:50:31.843 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:50:31.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:50:31.845 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:50:31.894 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:50:31.895 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:50:31.946 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:50:31.939+0000 7f962a64adc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:50:31.947 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:50:31.943+0000 7f962a64adc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:50:31.972 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:50:31.972 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:50:31.973 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:50:31.973 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:50:31.973 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:50:31.973 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:50:31.974 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:50:31.974 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:50:31.974 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:50:31.975 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:50:31.975 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:50:31.975 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:50:31.975 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:50:31.976 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:50:31.976 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:50:31.976 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:50:31.976 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:50:31.977 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:50:31.977 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:50:31.977 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:50:31.977 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:50:31.978 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:50:31.978 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:50:31.978 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:50:31.978 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:50:31.979 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:50:31.979 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:50:31.979 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:50:31.979 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:50:31.980 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:50:31.980 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:50:31.980 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:50:31.981 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:50:31.981 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:50:31.981 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:50:31.981 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:50:31.982 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:50:31.982 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:50:31.982 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:50:31.983 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:50:31.983 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:50:31.983 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:50:31.984 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:50:31.984 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:50:31.984 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:50:31.984 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:50:31.985 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:50:31.985 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:50:31.985 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:50:31.985 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763826.550710_156.1", 2022-09-09T22:50:31.986 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:50:31.986 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:50:31.987 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:50:31.987 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.550710Z", 2022-09-09T22:50:31.987 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:50:31.988 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:50:31.988 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:50:31.988 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:50:31.988 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:50:31.989 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:50:31.989 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:50:31.989 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:50:31.989 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:50:31.990 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:50:31Z" 2022-09-09T22:50:31.990 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:50:31.992 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763826.550710_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:50:26.550710Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:50:31Z" } 2022-09-09T22:50:31.993 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763826.550710_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:50:26.550710Z', 'realm_epoch': 2}}] 2022-09-09T22:50:31.993 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763826.550710_156.1'} 2022-09-09T22:50:31.994 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763826.550710_156.1'} 2022-09-09T22:50:31.994 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-09T22:50:31.995 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-09T22:50:31.995 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:50:31.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:50:31.996 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:50:32.047 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:50:32.047 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:50:32.101 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:50:32.095+0000 7febaf315dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:50:32.103 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:50:32.099+0000 7febaf315dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:50:32.130 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:50:32.131 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T22:50:32.131 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T22:50:32.131 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T22:50:32.131 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T22:50:32.132 INFO:teuthology.orchestra.run.smithi005.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:50:32.132 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:50:32.132 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:50:32.133 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T22:50:32.133 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:50:32.133 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T22:50:32.134 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:50:32.134 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:50:32.135 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:50:32.135 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:50:32.135 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 1, 2022-09-09T22:50:32.136 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:50:32.136 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:50:32.137 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:50:32.137 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:50:32.138 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:50:32.138 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:50:32.138 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T22:50:32.139 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:50:32.139 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:50:32.139 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:50:32.140 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:50:32.140 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:50:32.140 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:50:32.140 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:50:32.141 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:50:32.141 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:50:32.142 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:50:32.142 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:50:32.142 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T22:50:32.143 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:50:32.143 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:50:32.143 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:50:32.143 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:50:32.144 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:50:32.144 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:50:32.144 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:50:32.145 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:50:32.145 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:50:32.145 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:50:32.146 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:50:32.146 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T22:50:32.146 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:50:32.147 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:50:32.147 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662763826.550710_156.1", 2022-09-09T22:50:32.147 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:50:32.147 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:50:32.148 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:50:32.148 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.550710Z", 2022-09-09T22:50:32.148 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:50:32.149 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:50:32.149 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:50:32.149 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:50:32.149 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:50:32.150 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T22:50:32.150 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 1, 2022-09-09T22:50:32.150 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 1 2022-09-09T22:50:32.150 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:50:32.151 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T22:50:32Z" 2022-09-09T22:50:32.151 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:50:32.154 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763826.550710_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:50:26.550710Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:50:32Z" } 2022-09-09T22:50:32.155 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763826.550710_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:50:26.550710Z', 'realm_epoch': 2}}] 2022-09-09T22:50:32.155 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763826.550710_156.1'} 2022-09-09T22:50:32.156 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763826.550710_156.1'} 2022-09-09T22:50:32.156 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-09T22:50:32.157 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-09T22:50:32.158 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:50:32.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:50:32.160 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:50:32.205 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:50:32.206 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:50:32.264 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:50:32.251+0000 7f007c079dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:50:32.265 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:50:32.255+0000 7f007c079dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:50:32.287 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:50:32.288 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:50:32.289 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:50:32.290 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:50:32.290 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:50:32.291 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:50:32.291 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:50:32.291 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:50:32.292 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:50:32.292 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:50:32.293 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:50:32.293 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:50:32.293 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:50:32.294 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:50:32.294 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:50:32.295 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:50:32.295 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:50:32.296 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:50:32.296 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:50:32.296 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:50:32.297 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:50:32.297 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:50:32.298 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:50:32.298 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:50:32.298 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:50:32.299 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:50:32.299 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:50:32.299 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:50:32.300 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:50:32.300 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:50:32.301 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:50:32.301 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:50:32.301 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:50:32.302 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:50:32.302 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:50:32.303 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:50:32.303 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:50:32.303 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:50:32.304 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:50:32.304 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:50:32.305 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:50:32.305 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:50:32.305 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:50:32.306 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:50:32.306 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:50:32.307 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:50:32.307 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:50:32.308 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:50:32.308 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:50:32.308 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763826.550710_156.1", 2022-09-09T22:50:32.309 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:50:32.309 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:50:32.310 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:50:32.310 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.550710Z", 2022-09-09T22:50:32.311 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:50:32.311 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:50:32.312 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:50:32.312 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:50:32.312 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:50:32.313 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:50:32.313 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:50:32.314 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:50:32.314 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:50:32.314 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:50:32Z" 2022-09-09T22:50:32.315 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:50:32.317 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763826.550710_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:50:26.550710Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:50:32Z" } 2022-09-09T22:50:32.317 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763826.550710_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:50:26.550710Z', 'realm_epoch': 2}}] 2022-09-09T22:50:32.318 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763826.550710_156.1'} 2022-09-09T22:50:32.318 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763826.550710_156.1'} 2022-09-09T22:50:32.319 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-09T22:50:32.319 DEBUG:boto:path=/jkcgbj-24/ 2022-09-09T22:50:32.320 DEBUG:boto:auth_path=/jkcgbj-24/ 2022-09-09T22:50:32.321 DEBUG:boto:Method: HEAD 2022-09-09T22:50:32.321 DEBUG:boto:Path: /jkcgbj-24/ 2022-09-09T22:50:32.322 DEBUG:boto:Data: 2022-09-09T22:50:32.322 DEBUG:boto:Headers: {} 2022-09-09T22:50:32.323 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:32.323 DEBUG:boto:Port: 8000 2022-09-09T22:50:32.324 DEBUG:boto:Params: {} 2022-09-09T22:50:32.324 DEBUG:boto:Token: None 2022-09-09T22:50:32.325 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:32 GMT /jkcgbj-24/ 2022-09-09T22:50:32.326 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Q0dnACqS8p0Q4KYTd0YSWYvbQm0= 2022-09-09T22:50:32.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Q0dnACqS8p0Q4KYTd0YSWYvbQm0=', 'Content-Length': '0'} 2022-09-09T22:50:32.332 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000de15ef1debb6ecf6-00631bc338-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:32.333 DEBUG:boto:path=/jkcgbj-24/obj 2022-09-09T22:50:32.334 DEBUG:boto:auth_path=/jkcgbj-24/obj 2022-09-09T22:50:32.335 DEBUG:boto:Method: PUT 2022-09-09T22:50:32.336 DEBUG:boto:Path: /jkcgbj-24/obj 2022-09-09T22:50:32.336 DEBUG:boto:Data: 2022-09-09T22:50:32.337 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-09T22:50:32.338 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:32.338 DEBUG:boto:Port: 8000 2022-09-09T22:50:32.339 DEBUG:boto:Params: {} 2022-09-09T22:50:32.339 DEBUG:boto:Token: None 2022-09-09T22:50:32.340 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 09 Sep 2022 22:50:32 GMT /jkcgbj-24/obj 2022-09-09T22:50:32.340 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:gPHdo7W3KMTo46RxG/7qQ2FBbuk= 2022-09-09T22:50:32.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:50:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:gPHdo7W3KMTo46RxG/7qQ2FBbuk='} 2022-09-09T22:50:32.386 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f0c329e576f7777-00631bc338-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:50:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:32.387 DEBUG:boto:path=/jkcgbj-24/ 2022-09-09T22:50:32.388 DEBUG:boto:auth_path=/jkcgbj-24/ 2022-09-09T22:50:32.388 DEBUG:boto:Method: HEAD 2022-09-09T22:50:32.389 DEBUG:boto:Path: /jkcgbj-24/ 2022-09-09T22:50:32.389 DEBUG:boto:Data: 2022-09-09T22:50:32.390 DEBUG:boto:Headers: {} 2022-09-09T22:50:32.390 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:32.392 DEBUG:boto:Port: 8000 2022-09-09T22:50:32.393 DEBUG:boto:Params: {} 2022-09-09T22:50:32.394 DEBUG:boto:Token: None 2022-09-09T22:50:32.395 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:32 GMT /jkcgbj-24/ 2022-09-09T22:50:32.396 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Q0dnACqS8p0Q4KYTd0YSWYvbQm0= 2022-09-09T22:50:32.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Q0dnACqS8p0Q4KYTd0YSWYvbQm0=', 'Content-Length': '0'} 2022-09-09T22:50:32.403 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005c89c4ae3f798b75-00631bc338-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:32.404 DEBUG:boto:path=/jkcgbj-24/obj 2022-09-09T22:50:32.405 DEBUG:boto:auth_path=/jkcgbj-24/obj 2022-09-09T22:50:32.406 DEBUG:boto:Method: DELETE 2022-09-09T22:50:32.407 DEBUG:boto:Path: /jkcgbj-24/obj 2022-09-09T22:50:32.408 DEBUG:boto:Data: 2022-09-09T22:50:32.408 DEBUG:boto:Headers: {} 2022-09-09T22:50:32.409 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:50:32.410 DEBUG:boto:Port: 8000 2022-09-09T22:50:32.411 DEBUG:boto:Params: {} 2022-09-09T22:50:32.411 DEBUG:boto:Token: None 2022-09-09T22:50:32.412 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:50:32 GMT /jkcgbj-24/obj 2022-09-09T22:50:32.413 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:QsQ21h6TG0m9h/RKDIjSW+DomY4= 2022-09-09T22:50:32.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:QsQ21h6TG0m9h/RKDIjSW+DomY4=', 'Content-Length': '0'} 2022-09-09T22:50:32.439 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000009c894e6282cf7c14-00631bc338-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:50:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:32.440 DEBUG:boto:path=/jkcgbj-25/ 2022-09-09T22:50:32.441 DEBUG:boto:auth_path=/jkcgbj-25/ 2022-09-09T22:50:32.441 DEBUG:boto:Method: HEAD 2022-09-09T22:50:32.442 DEBUG:boto:Path: /jkcgbj-25/ 2022-09-09T22:50:32.442 DEBUG:boto:Data: 2022-09-09T22:50:32.443 DEBUG:boto:Headers: {} 2022-09-09T22:50:32.443 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:32.444 DEBUG:boto:Port: 8000 2022-09-09T22:50:32.444 DEBUG:boto:Params: {} 2022-09-09T22:50:32.445 DEBUG:boto:Token: None 2022-09-09T22:50:32.445 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:32 GMT /jkcgbj-25/ 2022-09-09T22:50:32.446 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:kST3JkGEUHnTIDbCmcQfC1D03lA= 2022-09-09T22:50:32.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:kST3JkGEUHnTIDbCmcQfC1D03lA=', 'Content-Length': '0'} 2022-09-09T22:50:32.451 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f53807ba8e3ae29-00631bc338-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:32.452 DEBUG:boto:path=/jkcgbj-25/obj 2022-09-09T22:50:32.452 DEBUG:boto:auth_path=/jkcgbj-25/obj 2022-09-09T22:50:32.453 DEBUG:boto:Method: PUT 2022-09-09T22:50:32.454 DEBUG:boto:Path: /jkcgbj-25/obj 2022-09-09T22:50:32.454 DEBUG:boto:Data: 2022-09-09T22:50:32.455 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-09T22:50:32.455 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:32.456 DEBUG:boto:Port: 8000 2022-09-09T22:50:32.456 DEBUG:boto:Params: {} 2022-09-09T22:50:32.457 DEBUG:boto:Token: None 2022-09-09T22:50:32.458 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 09 Sep 2022 22:50:32 GMT /jkcgbj-25/obj 2022-09-09T22:50:32.458 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ewmIPM9rKO6dkwhCvmp0ar2w4sc= 2022-09-09T22:50:32.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:50:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ewmIPM9rKO6dkwhCvmp0ar2w4sc='} 2022-09-09T22:50:32.508 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008de4e12d23657b18-00631bc338-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:50:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:32.509 DEBUG:boto:path=/jkcgbj-25/ 2022-09-09T22:50:32.509 DEBUG:boto:auth_path=/jkcgbj-25/ 2022-09-09T22:50:32.510 DEBUG:boto:Method: HEAD 2022-09-09T22:50:32.510 DEBUG:boto:Path: /jkcgbj-25/ 2022-09-09T22:50:32.511 DEBUG:boto:Data: 2022-09-09T22:50:32.511 DEBUG:boto:Headers: {} 2022-09-09T22:50:32.512 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:32.512 DEBUG:boto:Port: 8000 2022-09-09T22:50:32.512 DEBUG:boto:Params: {} 2022-09-09T22:50:32.513 DEBUG:boto:Token: None 2022-09-09T22:50:32.514 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:32 GMT /jkcgbj-25/ 2022-09-09T22:50:32.514 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:kST3JkGEUHnTIDbCmcQfC1D03lA= 2022-09-09T22:50:32.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:kST3JkGEUHnTIDbCmcQfC1D03lA=', 'Content-Length': '0'} 2022-09-09T22:50:32.519 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f6e4fa3a8a463b5-00631bc338-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:32.520 DEBUG:boto:path=/jkcgbj-25/obj 2022-09-09T22:50:32.521 DEBUG:boto:auth_path=/jkcgbj-25/obj 2022-09-09T22:50:32.521 DEBUG:boto:Method: DELETE 2022-09-09T22:50:32.522 DEBUG:boto:Path: /jkcgbj-25/obj 2022-09-09T22:50:32.522 DEBUG:boto:Data: 2022-09-09T22:50:32.523 DEBUG:boto:Headers: {} 2022-09-09T22:50:32.523 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:50:32.524 DEBUG:boto:Port: 8000 2022-09-09T22:50:32.524 DEBUG:boto:Params: {} 2022-09-09T22:50:32.525 DEBUG:boto:Token: None 2022-09-09T22:50:32.526 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:50:32 GMT /jkcgbj-25/obj 2022-09-09T22:50:32.527 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Z/AKo45mLbpjezov59vQbLr8c5I= 2022-09-09T22:50:32.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Z/AKo45mLbpjezov59vQbLr8c5I=', 'Content-Length': '0'} 2022-09-09T22:50:32.554 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000a4ed90573d8af482-00631bc338-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:50:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:32.555 DEBUG:boto:path=/jkcgbj-26/ 2022-09-09T22:50:32.556 DEBUG:boto:auth_path=/jkcgbj-26/ 2022-09-09T22:50:32.556 DEBUG:boto:Method: HEAD 2022-09-09T22:50:32.557 DEBUG:boto:Path: /jkcgbj-26/ 2022-09-09T22:50:32.557 DEBUG:boto:Data: 2022-09-09T22:50:32.558 DEBUG:boto:Headers: {} 2022-09-09T22:50:32.558 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:32.559 DEBUG:boto:Port: 8001 2022-09-09T22:50:32.559 DEBUG:boto:Params: {} 2022-09-09T22:50:32.560 DEBUG:boto:Token: None 2022-09-09T22:50:32.561 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:32 GMT /jkcgbj-26/ 2022-09-09T22:50:32.563 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:clquVAFuUmcAAyRqjE5qM9ZKI6k= 2022-09-09T22:50:32.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:clquVAFuUmcAAyRqjE5qM9ZKI6k=', 'Content-Length': '0'} 2022-09-09T22:50:32.569 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bf7a15bfd26e5b18-00631bc338-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:32.570 DEBUG:boto:path=/jkcgbj-26/obj 2022-09-09T22:50:32.571 DEBUG:boto:auth_path=/jkcgbj-26/obj 2022-09-09T22:50:32.572 DEBUG:boto:Method: PUT 2022-09-09T22:50:32.574 DEBUG:boto:Path: /jkcgbj-26/obj 2022-09-09T22:50:32.575 DEBUG:boto:Data: 2022-09-09T22:50:32.576 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-09T22:50:32.577 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:32.577 DEBUG:boto:Port: 8001 2022-09-09T22:50:32.578 DEBUG:boto:Params: {} 2022-09-09T22:50:32.579 DEBUG:boto:Token: None 2022-09-09T22:50:32.580 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 09 Sep 2022 22:50:32 GMT /jkcgbj-26/obj 2022-09-09T22:50:32.580 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:l5LfrDurzzFDFAZziP0Ps4Pmmc4= 2022-09-09T22:50:32.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:50:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:l5LfrDurzzFDFAZziP0Ps4Pmmc4='} 2022-09-09T22:50:32.629 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ff74d75c21fe6e68-00631bc338-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:50:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:32.630 DEBUG:boto:path=/jkcgbj-26/ 2022-09-09T22:50:32.630 DEBUG:boto:auth_path=/jkcgbj-26/ 2022-09-09T22:50:32.631 DEBUG:boto:Method: HEAD 2022-09-09T22:50:32.631 DEBUG:boto:Path: /jkcgbj-26/ 2022-09-09T22:50:32.632 DEBUG:boto:Data: 2022-09-09T22:50:32.632 DEBUG:boto:Headers: {} 2022-09-09T22:50:32.633 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:32.633 DEBUG:boto:Port: 8001 2022-09-09T22:50:32.634 DEBUG:boto:Params: {} 2022-09-09T22:50:32.634 DEBUG:boto:Token: None 2022-09-09T22:50:32.635 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:50:32 GMT /jkcgbj-26/ 2022-09-09T22:50:32.635 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:clquVAFuUmcAAyRqjE5qM9ZKI6k= 2022-09-09T22:50:32.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:clquVAFuUmcAAyRqjE5qM9ZKI6k=', 'Content-Length': '0'} 2022-09-09T22:50:32.640 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007f8b90d9497b03a-00631bc338-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:50:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:32.641 DEBUG:boto:path=/jkcgbj-26/obj 2022-09-09T22:50:32.641 DEBUG:boto:auth_path=/jkcgbj-26/obj 2022-09-09T22:50:32.642 DEBUG:boto:Method: DELETE 2022-09-09T22:50:32.642 DEBUG:boto:Path: /jkcgbj-26/obj 2022-09-09T22:50:32.643 DEBUG:boto:Data: 2022-09-09T22:50:32.643 DEBUG:boto:Headers: {} 2022-09-09T22:50:32.644 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:50:32.644 DEBUG:boto:Port: 8001 2022-09-09T22:50:32.645 DEBUG:boto:Params: {} 2022-09-09T22:50:32.645 DEBUG:boto:Token: None 2022-09-09T22:50:32.646 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:50:32 GMT /jkcgbj-26/obj 2022-09-09T22:50:32.647 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:g+7o5YN2JMQ2TasE0X3yWlqY4ao= 2022-09-09T22:50:32.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:50:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:g+7o5YN2JMQ2TasE0X3yWlqY4ao=', 'Content-Length': '0'} 2022-09-09T22:50:32.671 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000e1f44aad52d7d6e9-00631bc338-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:50:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:50:32.673 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jkcgbj-24 2022-09-09T22:50:32.674 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:32.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:32.675 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:50:32.729 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:50:32.730 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:50:32.810 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:50:32.803+0000 7f6a89412dc0 1 waiting to reach incremental sync.. 2022-09-09T22:50:37.811 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:50:37.803+0000 7f6a89412dc0 1 waiting to reach incremental sync.. 2022-09-09T22:50:42.815 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:50:42.807+0000 7f6a89412dc0 1 bucket sync caught up with source: 2022-09-09T22:50:42.815 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000003.16.6, , ] 2022-09-09T22:50:42.816 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000003.16.6, , ] 2022-09-09T22:50:42.816 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:50:42.807+0000 7f6a89412dc0 0 bucket checkpoint complete 2022-09-09T22:50:42.844 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jkcgbj-24 2022-09-09T22:50:42.845 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:42.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:42.846 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:50:42.901 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:50:42.901 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:50:42.970 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:50:42.967+0000 7fa13e0f3dc0 1 bucket sync caught up with source: 2022-09-09T22:50:42.970 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000003.16.6, , ] 2022-09-09T22:50:42.971 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000003.16.6, , ] 2022-09-09T22:50:42.971 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:50:42.967+0000 7fa13e0f3dc0 0 bucket checkpoint complete 2022-09-09T22:50:42.995 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jkcgbj-24 2022-09-09T22:50:42.995 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:42.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:42.997 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:50:43.052 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:50:43.053 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:50:43.133 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:50:43.127+0000 7f774bf49dc0 1 waiting to reach incremental sync.. 2022-09-09T22:50:48.137 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:50:48.135+0000 7f774bf49dc0 1 bucket sync caught up with source: 2022-09-09T22:50:48.137 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000003.16.6, , ] 2022-09-09T22:50:48.138 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000003.16.6, , ] 2022-09-09T22:50:48.138 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:50:48.135+0000 7f774bf49dc0 0 bucket checkpoint complete 2022-09-09T22:50:48.166 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jkcgbj-24 2022-09-09T22:50:48.167 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:48.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:48.170 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:50:48.225 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:50:48.225 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:50:48.294 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:50:48.291+0000 7fb54f16cdc0 1 bucket sync caught up with source: 2022-09-09T22:50:48.295 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000001.27.12, , ] 2022-09-09T22:50:48.295 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000001.27.12, , ] 2022-09-09T22:50:48.296 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:50:48.291+0000 7fb54f16cdc0 0 bucket checkpoint complete 2022-09-09T22:50:48.321 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jkcgbj-24 2022-09-09T22:50:48.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:48.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:50:48.323 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:50:48.376 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:50:48.377 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:50:48.442 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:50:48.439+0000 7fad0884bdc0 1 waiting to reach incremental sync.. 2022-09-09T22:50:53.443 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:50:53.438+0000 7fad0884bdc0 1 waiting to reach incremental sync.. 2022-09-09T22:50:58.444 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:50:58.438+0000 7fad0884bdc0 1 waiting to reach incremental sync.. 2022-09-09T22:51:03.445 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:51:03.438+0000 7fad0884bdc0 1 waiting to reach incremental sync.. 2022-09-09T22:51:08.446 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:51:08.442+0000 7fad0884bdc0 1 waiting to reach incremental sync.. 2022-09-09T22:51:13.447 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:51:13.442+0000 7fad0884bdc0 1 waiting to reach incremental sync.. 2022-09-09T22:51:18.448 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:51:18.442+0000 7fad0884bdc0 1 waiting to reach incremental sync.. 2022-09-09T22:51:23.449 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:51:23.446+0000 7fad0884bdc0 1 waiting to reach incremental sync.. 2022-09-09T22:51:28.450 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:51:28.446+0000 7fad0884bdc0 1 waiting to reach incremental sync.. 2022-09-09T22:51:33.451 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:51:33.450+0000 7fad0884bdc0 1 waiting to reach incremental sync.. 2022-09-09T22:51:38.452 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:51:38.450+0000 7fad0884bdc0 1 waiting to reach incremental sync.. 2022-09-09T22:51:43.453 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:51:43.446+0000 7fad0884bdc0 1 waiting to reach incremental sync.. 2022-09-09T22:51:48.455 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:51:48.449+0000 7fad0884bdc0 1 waiting to reach incremental sync.. 2022-09-09T22:51:53.456 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:51:53.449+0000 7fad0884bdc0 1 waiting to reach incremental sync.. 2022-09-09T22:51:58.457 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:51:58.453+0000 7fad0884bdc0 1 waiting to reach incremental sync.. 2022-09-09T22:52:03.458 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:52:03.457+0000 7fad0884bdc0 1 waiting to reach incremental sync.. 2022-09-09T22:52:08.459 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:52:08.457+0000 7fad0884bdc0 1 waiting to reach incremental sync.. 2022-09-09T22:52:13.460 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:52:13.457+0000 7fad0884bdc0 1 waiting to reach incremental sync.. 2022-09-09T22:52:18.461 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:52:18.457+0000 7fad0884bdc0 1 waiting to reach incremental sync.. 2022-09-09T22:52:23.462 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:52:23.461+0000 7fad0884bdc0 1 waiting to reach incremental sync.. 2022-09-09T22:52:28.463 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:52:28.461+0000 7fad0884bdc0 1 waiting to reach incremental sync.. 2022-09-09T22:52:33.464 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:52:33.461+0000 7fad0884bdc0 1 waiting to reach incremental sync.. 2022-09-09T22:52:38.465 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:52:38.460+0000 7fad0884bdc0 1 waiting to reach incremental sync.. 2022-09-09T22:52:43.466 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:52:43.464+0000 7fad0884bdc0 1 waiting to reach incremental sync.. 2022-09-09T22:52:48.467 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:52:48.464+0000 7fad0884bdc0 1 waiting to reach incremental sync.. 2022-09-09T22:52:53.468 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:52:53.464+0000 7fad0884bdc0 1 waiting to reach incremental sync.. 2022-09-09T22:52:58.469 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:52:58.464+0000 7fad0884bdc0 1 waiting to reach incremental sync.. 2022-09-09T22:53:03.474 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:53:03.472+0000 7fad0884bdc0 1 bucket sync caught up with source: 2022-09-09T22:53:03.474 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000001.27.12, , ] 2022-09-09T22:53:03.475 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000001.27.12, , ] 2022-09-09T22:53:03.475 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:53:03.472+0000 7fad0884bdc0 0 bucket checkpoint complete 2022-09-09T22:53:03.501 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jkcgbj-24 2022-09-09T22:53:03.503 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jkcgbj-24 2022-09-09T22:53:03.503 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:03.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:03.505 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:53:03.559 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:53:03.560 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:53:03.625 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:53:03.620+0000 7f5e8d68bdc0 1 bucket sync caught up with source: 2022-09-09T22:53:03.625 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000001.22.12, , ] 2022-09-09T22:53:03.626 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000001.22.12, , ] 2022-09-09T22:53:03.626 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:53:03.620+0000 7f5e8d68bdc0 0 bucket checkpoint complete 2022-09-09T22:53:03.653 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jkcgbj-24 2022-09-09T22:53:03.654 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:03.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:03.655 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:53:03.707 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:53:03.708 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:53:03.783 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:03.780+0000 7f5d6f46edc0 1 bucket sync caught up with source: 2022-09-09T22:53:03.784 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000001.22.12, , ] 2022-09-09T22:53:03.784 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000001.22.12, , ] 2022-09-09T22:53:03.785 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:03.780+0000 7f5d6f46edc0 0 bucket checkpoint complete 2022-09-09T22:53:03.813 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jkcgbj-24 2022-09-09T22:53:03.814 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-24 zones={test-zone2, test-zone1} 2022-09-09T22:53:03.816 DEBUG:boto:path=/jkcgbj-24/ 2022-09-09T22:53:03.817 DEBUG:boto:auth_path=/jkcgbj-24/ 2022-09-09T22:53:03.818 DEBUG:boto:Method: HEAD 2022-09-09T22:53:03.819 DEBUG:boto:Path: /jkcgbj-24/ 2022-09-09T22:53:03.820 DEBUG:boto:Data: 2022-09-09T22:53:03.821 DEBUG:boto:Headers: {} 2022-09-09T22:53:03.821 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:03.822 DEBUG:boto:Port: 8000 2022-09-09T22:53:03.823 DEBUG:boto:Params: {} 2022-09-09T22:53:03.825 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T22:53:03.826 DEBUG:boto:Token: None 2022-09-09T22:53:03.826 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:03 GMT /jkcgbj-24/ 2022-09-09T22:53:03.827 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:PeW993wdA/YJkUWnC0FcZHVlpyQ= 2022-09-09T22:53:03.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:PeW993wdA/YJkUWnC0FcZHVlpyQ=', 'Content-Length': '0'} 2022-09-09T22:53:03.836 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005aaa56145f4ac305-00631bc3cf-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:03.837 DEBUG:boto:path=/jkcgbj-24/ 2022-09-09T22:53:03.838 DEBUG:boto:auth_path=/jkcgbj-24/ 2022-09-09T22:53:03.839 DEBUG:boto:Method: HEAD 2022-09-09T22:53:03.840 DEBUG:boto:Path: /jkcgbj-24/ 2022-09-09T22:53:03.840 DEBUG:boto:Data: 2022-09-09T22:53:03.841 DEBUG:boto:Headers: {} 2022-09-09T22:53:03.842 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:03.842 DEBUG:boto:Port: 8000 2022-09-09T22:53:03.843 DEBUG:boto:Params: {} 2022-09-09T22:53:03.845 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T22:53:03.845 DEBUG:boto:Token: None 2022-09-09T22:53:03.846 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:03 GMT /jkcgbj-24/ 2022-09-09T22:53:03.848 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:PeW993wdA/YJkUWnC0FcZHVlpyQ= 2022-09-09T22:53:03.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:PeW993wdA/YJkUWnC0FcZHVlpyQ=', 'Content-Length': '0'} 2022-09-09T22:53:03.855 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000067920edeadd5da4b-00631bc3cf-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:03.856 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:53:03.857 DEBUG:boto:path=/jkcgbj-24/ 2022-09-09T22:53:03.857 DEBUG:boto:auth_path=/jkcgbj-24/ 2022-09-09T22:53:03.861 DEBUG:boto:path=/jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:03.862 DEBUG:boto:auth_path=/jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:03.864 DEBUG:boto:Method: GET 2022-09-09T22:53:03.865 DEBUG:boto:Path: /jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:03.865 DEBUG:boto:Data: 2022-09-09T22:53:03.866 DEBUG:boto:Headers: {} 2022-09-09T22:53:03.866 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:03.867 DEBUG:boto:Port: 8000 2022-09-09T22:53:03.867 DEBUG:boto:Params: {} 2022-09-09T22:53:03.868 DEBUG:boto:Token: None 2022-09-09T22:53:03.869 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:03 GMT /jkcgbj-24/?versions 2022-09-09T22:53:03.869 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nky5CBqqrls0Vk6p10tCouZwR+w= 2022-09-09T22:53:03.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nky5CBqqrls0Vk6p10tCouZwR+w=', 'Content-Length': '0'} 2022-09-09T22:53:03.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000981104a72a4f7671-00631bc3cf-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:03.876 DEBUG:boto:b'jkcgbj-24999falseobjnulltrue2022-09-09T22:50:32.417Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:03.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:53:03.878 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:53:03.879 DEBUG:boto:path=/jkcgbj-24/ 2022-09-09T22:53:03.879 DEBUG:boto:auth_path=/jkcgbj-24/ 2022-09-09T22:53:03.880 DEBUG:boto:path=/jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:03.880 DEBUG:boto:auth_path=/jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:03.881 DEBUG:boto:Method: GET 2022-09-09T22:53:03.881 DEBUG:boto:Path: /jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:03.882 DEBUG:boto:Data: 2022-09-09T22:53:03.883 DEBUG:boto:Headers: {} 2022-09-09T22:53:03.883 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:03.884 DEBUG:boto:Port: 8000 2022-09-09T22:53:03.884 DEBUG:boto:Params: {} 2022-09-09T22:53:03.885 DEBUG:boto:Token: None 2022-09-09T22:53:03.885 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:03 GMT /jkcgbj-24/?versions 2022-09-09T22:53:03.886 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nky5CBqqrls0Vk6p10tCouZwR+w= 2022-09-09T22:53:03.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nky5CBqqrls0Vk6p10tCouZwR+w=', 'Content-Length': '0'} 2022-09-09T22:53:03.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2d35c56795c4cd5-00631bc3cf-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:03.893 DEBUG:boto:b'jkcgbj-24999falseobjnulltrue2022-09-09T22:50:32.417Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:03.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:53:03.895 DEBUG:boto:path=/jkcgbj-24/ 2022-09-09T22:53:03.895 DEBUG:boto:auth_path=/jkcgbj-24/ 2022-09-09T22:53:03.896 DEBUG:boto:path=/jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:03.896 DEBUG:boto:auth_path=/jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:03.897 DEBUG:boto:Method: GET 2022-09-09T22:53:03.897 DEBUG:boto:Path: /jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:03.898 DEBUG:boto:Data: 2022-09-09T22:53:03.899 DEBUG:boto:Headers: {} 2022-09-09T22:53:03.899 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:03.900 DEBUG:boto:Port: 8000 2022-09-09T22:53:03.900 DEBUG:boto:Params: {} 2022-09-09T22:53:03.901 DEBUG:boto:Token: None 2022-09-09T22:53:03.902 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:03 GMT /jkcgbj-24/?versions 2022-09-09T22:53:03.902 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nky5CBqqrls0Vk6p10tCouZwR+w= 2022-09-09T22:53:03.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nky5CBqqrls0Vk6p10tCouZwR+w=', 'Content-Length': '0'} 2022-09-09T22:53:03.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009dcfa78c60c5c187-00631bc3cf-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:03.909 DEBUG:boto:b'jkcgbj-24999falseobjnulltrue2022-09-09T22:50:32.417Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:03.910 DEBUG:boto:path=/jkcgbj-24/ 2022-09-09T22:53:03.910 DEBUG:boto:auth_path=/jkcgbj-24/ 2022-09-09T22:53:03.911 DEBUG:boto:path=/jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:03.911 DEBUG:boto:auth_path=/jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:03.912 DEBUG:boto:Method: GET 2022-09-09T22:53:03.912 DEBUG:boto:Path: /jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:03.913 DEBUG:boto:Data: 2022-09-09T22:53:03.913 DEBUG:boto:Headers: {} 2022-09-09T22:53:03.914 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:03.914 DEBUG:boto:Port: 8000 2022-09-09T22:53:03.915 DEBUG:boto:Params: {} 2022-09-09T22:53:03.915 DEBUG:boto:Token: None 2022-09-09T22:53:03.916 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:03 GMT /jkcgbj-24/?versions 2022-09-09T22:53:03.917 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nky5CBqqrls0Vk6p10tCouZwR+w= 2022-09-09T22:53:03.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nky5CBqqrls0Vk6p10tCouZwR+w=', 'Content-Length': '0'} 2022-09-09T22:53:03.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000662a8f2d4f931b0b-00631bc3cf-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:03.923 DEBUG:boto:b'jkcgbj-24999falseobjnulltrue2022-09-09T22:50:32.417Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:03.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:53:03.924 DEBUG:boto:path=/jkcgbj-24/obj 2022-09-09T22:53:03.925 DEBUG:boto:auth_path=/jkcgbj-24/obj 2022-09-09T22:53:03.926 DEBUG:boto:Method: HEAD 2022-09-09T22:53:03.926 DEBUG:boto:Path: /jkcgbj-24/obj 2022-09-09T22:53:03.927 DEBUG:boto:Data: 2022-09-09T22:53:03.927 DEBUG:boto:Headers: {} 2022-09-09T22:53:03.928 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:03.928 DEBUG:boto:Port: 8000 2022-09-09T22:53:03.929 DEBUG:boto:Params: {} 2022-09-09T22:53:03.929 DEBUG:boto:Token: None 2022-09-09T22:53:03.930 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:03 GMT /jkcgbj-24/obj 2022-09-09T22:53:03.930 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UzAJXY8tCEEnLeTB3lJFESyqyOI= 2022-09-09T22:53:03.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UzAJXY8tCEEnLeTB3lJFESyqyOI=', 'Content-Length': '0'} 2022-09-09T22:53:03.934 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000c9a6f6b8ba871a29-00631bc3cf-10e8-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:03.934 DEBUG:boto:path=/jkcgbj-24/obj 2022-09-09T22:53:03.935 DEBUG:boto:auth_path=/jkcgbj-24/obj 2022-09-09T22:53:03.935 DEBUG:boto:Method: HEAD 2022-09-09T22:53:03.936 DEBUG:boto:Path: /jkcgbj-24/obj 2022-09-09T22:53:03.936 DEBUG:boto:Data: 2022-09-09T22:53:03.937 DEBUG:boto:Headers: {} 2022-09-09T22:53:03.937 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:03.938 DEBUG:boto:Port: 8000 2022-09-09T22:53:03.938 DEBUG:boto:Params: {} 2022-09-09T22:53:03.939 DEBUG:boto:Token: None 2022-09-09T22:53:03.940 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:03 GMT /jkcgbj-24/obj 2022-09-09T22:53:03.940 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UzAJXY8tCEEnLeTB3lJFESyqyOI= 2022-09-09T22:53:03.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UzAJXY8tCEEnLeTB3lJFESyqyOI=', 'Content-Length': '0'} 2022-09-09T22:53:03.944 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000004d8a4d652f966453-00631bc3cf-10ff-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:03.944 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-24 zones={test-zone2, test-zone1} 2022-09-09T22:53:03.945 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-24 zones={test-zone3, test-zone1} 2022-09-09T22:53:03.945 DEBUG:boto:path=/jkcgbj-24/ 2022-09-09T22:53:03.946 DEBUG:boto:auth_path=/jkcgbj-24/ 2022-09-09T22:53:03.946 DEBUG:boto:Method: HEAD 2022-09-09T22:53:03.947 DEBUG:boto:Path: /jkcgbj-24/ 2022-09-09T22:53:03.948 DEBUG:boto:Data: 2022-09-09T22:53:03.948 DEBUG:boto:Headers: {} 2022-09-09T22:53:03.949 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:03.949 DEBUG:boto:Port: 8001 2022-09-09T22:53:03.950 DEBUG:boto:Params: {} 2022-09-09T22:53:03.950 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-09T22:53:03.951 DEBUG:boto:Token: None 2022-09-09T22:53:03.952 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:03 GMT /jkcgbj-24/ 2022-09-09T22:53:03.952 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:PeW993wdA/YJkUWnC0FcZHVlpyQ= 2022-09-09T22:53:03.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:PeW993wdA/YJkUWnC0FcZHVlpyQ=', 'Content-Length': '0'} 2022-09-09T22:53:03.959 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd130ae2e6161dbf-00631bc3cf-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:03.960 DEBUG:boto:path=/jkcgbj-24/ 2022-09-09T22:53:03.960 DEBUG:boto:auth_path=/jkcgbj-24/ 2022-09-09T22:53:03.961 DEBUG:boto:Method: HEAD 2022-09-09T22:53:03.961 DEBUG:boto:Path: /jkcgbj-24/ 2022-09-09T22:53:03.962 DEBUG:boto:Data: 2022-09-09T22:53:03.962 DEBUG:boto:Headers: {} 2022-09-09T22:53:03.963 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:03.963 DEBUG:boto:Port: 8000 2022-09-09T22:53:03.964 DEBUG:boto:Params: {} 2022-09-09T22:53:03.964 DEBUG:boto:Token: None 2022-09-09T22:53:03.965 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:03 GMT /jkcgbj-24/ 2022-09-09T22:53:03.965 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:PeW993wdA/YJkUWnC0FcZHVlpyQ= 2022-09-09T22:53:03.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:PeW993wdA/YJkUWnC0FcZHVlpyQ=', 'Content-Length': '0'} 2022-09-09T22:53:03.971 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f4df36eb01bf5e9-00631bc3cf-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:03.971 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:53:03.972 DEBUG:boto:path=/jkcgbj-24/ 2022-09-09T22:53:03.973 DEBUG:boto:auth_path=/jkcgbj-24/ 2022-09-09T22:53:03.973 DEBUG:boto:path=/jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:03.974 DEBUG:boto:auth_path=/jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:03.974 DEBUG:boto:Method: GET 2022-09-09T22:53:03.975 DEBUG:boto:Path: /jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:03.975 DEBUG:boto:Data: 2022-09-09T22:53:03.976 DEBUG:boto:Headers: {} 2022-09-09T22:53:03.976 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:03.977 DEBUG:boto:Port: 8001 2022-09-09T22:53:03.977 DEBUG:boto:Params: {} 2022-09-09T22:53:03.978 DEBUG:boto:Token: None 2022-09-09T22:53:03.978 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:03 GMT /jkcgbj-24/?versions 2022-09-09T22:53:03.979 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nky5CBqqrls0Vk6p10tCouZwR+w= 2022-09-09T22:53:03.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nky5CBqqrls0Vk6p10tCouZwR+w=', 'Content-Length': '0'} 2022-09-09T22:53:03.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7c2a3f9182f90d6-00631bc3cf-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:03.987 DEBUG:boto:b'jkcgbj-24999falseobjnulltrue2022-09-09T22:50:32.417Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:03.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:53:03.988 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:53:03.988 DEBUG:boto:path=/jkcgbj-24/ 2022-09-09T22:53:03.989 DEBUG:boto:auth_path=/jkcgbj-24/ 2022-09-09T22:53:03.989 DEBUG:boto:path=/jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:03.990 DEBUG:boto:auth_path=/jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:03.991 DEBUG:boto:Method: GET 2022-09-09T22:53:03.991 DEBUG:boto:Path: /jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:03.992 DEBUG:boto:Data: 2022-09-09T22:53:03.992 DEBUG:boto:Headers: {} 2022-09-09T22:53:03.993 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:03.993 DEBUG:boto:Port: 8000 2022-09-09T22:53:03.994 DEBUG:boto:Params: {} 2022-09-09T22:53:03.995 DEBUG:boto:Token: None 2022-09-09T22:53:03.996 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:03 GMT /jkcgbj-24/?versions 2022-09-09T22:53:03.996 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nky5CBqqrls0Vk6p10tCouZwR+w= 2022-09-09T22:53:03.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:03 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nky5CBqqrls0Vk6p10tCouZwR+w=', 'Content-Length': '0'} 2022-09-09T22:53:04.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000743e2c536aa94a1d-00631bc3cf-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:04.050 DEBUG:boto:b'jkcgbj-24999falseobjnulltrue2022-09-09T22:50:32.417Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:04.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:53:04.052 DEBUG:boto:path=/jkcgbj-24/ 2022-09-09T22:53:04.053 DEBUG:boto:auth_path=/jkcgbj-24/ 2022-09-09T22:53:04.053 DEBUG:boto:path=/jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:04.054 DEBUG:boto:auth_path=/jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:04.055 DEBUG:boto:Method: GET 2022-09-09T22:53:04.056 DEBUG:boto:Path: /jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:04.057 DEBUG:boto:Data: 2022-09-09T22:53:04.057 DEBUG:boto:Headers: {} 2022-09-09T22:53:04.058 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:04.059 DEBUG:boto:Port: 8001 2022-09-09T22:53:04.059 DEBUG:boto:Params: {} 2022-09-09T22:53:04.060 DEBUG:boto:Token: None 2022-09-09T22:53:04.061 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:04 GMT /jkcgbj-24/?versions 2022-09-09T22:53:04.061 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R9HNUMOazjZf6ty6vpmzyCdtnAs= 2022-09-09T22:53:04.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R9HNUMOazjZf6ty6vpmzyCdtnAs=', 'Content-Length': '0'} 2022-09-09T22:53:04.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cddc4d613d4a64af-00631bc3d0-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:04.068 DEBUG:boto:b'jkcgbj-24999falseobjnulltrue2022-09-09T22:50:32.417Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:04.069 DEBUG:boto:path=/jkcgbj-24/ 2022-09-09T22:53:04.070 DEBUG:boto:auth_path=/jkcgbj-24/ 2022-09-09T22:53:04.070 DEBUG:boto:path=/jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:04.071 DEBUG:boto:auth_path=/jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:04.071 DEBUG:boto:Method: GET 2022-09-09T22:53:04.072 DEBUG:boto:Path: /jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:04.072 DEBUG:boto:Data: 2022-09-09T22:53:04.073 DEBUG:boto:Headers: {} 2022-09-09T22:53:04.073 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:04.074 DEBUG:boto:Port: 8000 2022-09-09T22:53:04.074 DEBUG:boto:Params: {} 2022-09-09T22:53:04.075 DEBUG:boto:Token: None 2022-09-09T22:53:04.076 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:04 GMT /jkcgbj-24/?versions 2022-09-09T22:53:04.076 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R9HNUMOazjZf6ty6vpmzyCdtnAs= 2022-09-09T22:53:04.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R9HNUMOazjZf6ty6vpmzyCdtnAs=', 'Content-Length': '0'} 2022-09-09T22:53:04.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d631a9d75447cafd-00631bc3d0-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:04.084 DEBUG:boto:b'jkcgbj-24999falseobjnulltrue2022-09-09T22:50:32.417Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:04.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:53:04.085 DEBUG:boto:path=/jkcgbj-24/obj 2022-09-09T22:53:04.086 DEBUG:boto:auth_path=/jkcgbj-24/obj 2022-09-09T22:53:04.087 DEBUG:boto:Method: HEAD 2022-09-09T22:53:04.087 DEBUG:boto:Path: /jkcgbj-24/obj 2022-09-09T22:53:04.088 DEBUG:boto:Data: 2022-09-09T22:53:04.088 DEBUG:boto:Headers: {} 2022-09-09T22:53:04.089 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:04.089 DEBUG:boto:Port: 8001 2022-09-09T22:53:04.090 DEBUG:boto:Params: {} 2022-09-09T22:53:04.090 DEBUG:boto:Token: None 2022-09-09T22:53:04.091 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:04 GMT /jkcgbj-24/obj 2022-09-09T22:53:04.091 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ONMzpULxJZJTZ0cK4iku+fyc9J4= 2022-09-09T22:53:04.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ONMzpULxJZJTZ0cK4iku+fyc9J4=', 'Content-Length': '0'} 2022-09-09T22:53:04.095 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000000b2396cc6e7489b0-00631bc3d0-1103-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:04.096 DEBUG:boto:path=/jkcgbj-24/obj 2022-09-09T22:53:04.096 DEBUG:boto:auth_path=/jkcgbj-24/obj 2022-09-09T22:53:04.097 DEBUG:boto:Method: HEAD 2022-09-09T22:53:04.098 DEBUG:boto:Path: /jkcgbj-24/obj 2022-09-09T22:53:04.098 DEBUG:boto:Data: 2022-09-09T22:53:04.099 DEBUG:boto:Headers: {} 2022-09-09T22:53:04.099 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:04.100 DEBUG:boto:Port: 8000 2022-09-09T22:53:04.100 DEBUG:boto:Params: {} 2022-09-09T22:53:04.101 DEBUG:boto:Token: None 2022-09-09T22:53:04.101 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:04 GMT /jkcgbj-24/obj 2022-09-09T22:53:04.102 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ONMzpULxJZJTZ0cK4iku+fyc9J4= 2022-09-09T22:53:04.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ONMzpULxJZJTZ0cK4iku+fyc9J4=', 'Content-Length': '0'} 2022-09-09T22:53:04.105 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000ad7506b68646135d-00631bc3d0-10ff-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:04.106 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-24 zones={test-zone3, test-zone1} 2022-09-09T22:53:04.107 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-24 zones={test-zone3, test-zone2} 2022-09-09T22:53:04.107 DEBUG:boto:path=/jkcgbj-24/ 2022-09-09T22:53:04.108 DEBUG:boto:auth_path=/jkcgbj-24/ 2022-09-09T22:53:04.108 DEBUG:boto:Method: HEAD 2022-09-09T22:53:04.109 DEBUG:boto:Path: /jkcgbj-24/ 2022-09-09T22:53:04.109 DEBUG:boto:Data: 2022-09-09T22:53:04.110 DEBUG:boto:Headers: {} 2022-09-09T22:53:04.110 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:04.111 DEBUG:boto:Port: 8001 2022-09-09T22:53:04.111 DEBUG:boto:Params: {} 2022-09-09T22:53:04.112 DEBUG:boto:Token: None 2022-09-09T22:53:04.112 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:04 GMT /jkcgbj-24/ 2022-09-09T22:53:04.113 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:izeFIVmMoUldcW5NnTTumHj0DAg= 2022-09-09T22:53:04.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:izeFIVmMoUldcW5NnTTumHj0DAg=', 'Content-Length': '0'} 2022-09-09T22:53:04.120 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000de3c2779059acf0-00631bc3d0-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:04.121 DEBUG:boto:path=/jkcgbj-24/ 2022-09-09T22:53:04.122 DEBUG:boto:auth_path=/jkcgbj-24/ 2022-09-09T22:53:04.123 DEBUG:boto:Method: HEAD 2022-09-09T22:53:04.124 DEBUG:boto:Path: /jkcgbj-24/ 2022-09-09T22:53:04.125 DEBUG:boto:Data: 2022-09-09T22:53:04.126 DEBUG:boto:Headers: {} 2022-09-09T22:53:04.126 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:04.127 DEBUG:boto:Port: 8000 2022-09-09T22:53:04.127 DEBUG:boto:Params: {} 2022-09-09T22:53:04.128 DEBUG:boto:Token: None 2022-09-09T22:53:04.128 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:04 GMT /jkcgbj-24/ 2022-09-09T22:53:04.129 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:izeFIVmMoUldcW5NnTTumHj0DAg= 2022-09-09T22:53:04.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:izeFIVmMoUldcW5NnTTumHj0DAg=', 'Content-Length': '0'} 2022-09-09T22:53:04.135 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a0036471fd8b0ad-00631bc3d0-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:04.136 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:53:04.137 DEBUG:boto:path=/jkcgbj-24/ 2022-09-09T22:53:04.138 DEBUG:boto:auth_path=/jkcgbj-24/ 2022-09-09T22:53:04.139 DEBUG:boto:path=/jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:04.141 DEBUG:boto:auth_path=/jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:04.142 DEBUG:boto:Method: GET 2022-09-09T22:53:04.143 DEBUG:boto:Path: /jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:04.144 DEBUG:boto:Data: 2022-09-09T22:53:04.145 DEBUG:boto:Headers: {} 2022-09-09T22:53:04.146 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:04.147 DEBUG:boto:Port: 8001 2022-09-09T22:53:04.148 DEBUG:boto:Params: {} 2022-09-09T22:53:04.149 DEBUG:boto:Token: None 2022-09-09T22:53:04.150 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:04 GMT /jkcgbj-24/?versions 2022-09-09T22:53:04.151 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R9HNUMOazjZf6ty6vpmzyCdtnAs= 2022-09-09T22:53:04.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R9HNUMOazjZf6ty6vpmzyCdtnAs=', 'Content-Length': '0'} 2022-09-09T22:53:04.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f608ad680f422690-00631bc3d0-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:04.202 DEBUG:boto:b'jkcgbj-24999falseobjnulltrue2022-09-09T22:50:32.417Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:04.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:53:04.204 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:53:04.204 DEBUG:boto:path=/jkcgbj-24/ 2022-09-09T22:53:04.205 DEBUG:boto:auth_path=/jkcgbj-24/ 2022-09-09T22:53:04.206 DEBUG:boto:path=/jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:04.207 DEBUG:boto:auth_path=/jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:04.207 DEBUG:boto:Method: GET 2022-09-09T22:53:04.208 DEBUG:boto:Path: /jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:04.209 DEBUG:boto:Data: 2022-09-09T22:53:04.209 DEBUG:boto:Headers: {} 2022-09-09T22:53:04.210 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:04.211 DEBUG:boto:Port: 8000 2022-09-09T22:53:04.211 DEBUG:boto:Params: {} 2022-09-09T22:53:04.212 DEBUG:boto:Token: None 2022-09-09T22:53:04.212 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:04 GMT /jkcgbj-24/?versions 2022-09-09T22:53:04.213 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R9HNUMOazjZf6ty6vpmzyCdtnAs= 2022-09-09T22:53:04.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R9HNUMOazjZf6ty6vpmzyCdtnAs=', 'Content-Length': '0'} 2022-09-09T22:53:04.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc5b201e93c9f45b-00631bc3d0-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:04.226 DEBUG:boto:b'jkcgbj-24999falseobjnulltrue2022-09-09T22:50:32.417Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:04.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:53:04.228 DEBUG:boto:path=/jkcgbj-24/ 2022-09-09T22:53:04.229 DEBUG:boto:auth_path=/jkcgbj-24/ 2022-09-09T22:53:04.229 DEBUG:boto:path=/jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:04.230 DEBUG:boto:auth_path=/jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:04.231 DEBUG:boto:Method: GET 2022-09-09T22:53:04.231 DEBUG:boto:Path: /jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:04.232 DEBUG:boto:Data: 2022-09-09T22:53:04.232 DEBUG:boto:Headers: {} 2022-09-09T22:53:04.233 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:04.233 DEBUG:boto:Port: 8001 2022-09-09T22:53:04.234 DEBUG:boto:Params: {} 2022-09-09T22:53:04.235 DEBUG:boto:Token: None 2022-09-09T22:53:04.235 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:04 GMT /jkcgbj-24/?versions 2022-09-09T22:53:04.236 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R9HNUMOazjZf6ty6vpmzyCdtnAs= 2022-09-09T22:53:04.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R9HNUMOazjZf6ty6vpmzyCdtnAs=', 'Content-Length': '0'} 2022-09-09T22:53:04.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009669d7d6240d4baf-00631bc3d0-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:04.245 DEBUG:boto:b'jkcgbj-24999falseobjnulltrue2022-09-09T22:50:32.417Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:04.246 DEBUG:boto:path=/jkcgbj-24/ 2022-09-09T22:53:04.247 DEBUG:boto:auth_path=/jkcgbj-24/ 2022-09-09T22:53:04.247 DEBUG:boto:path=/jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:04.248 DEBUG:boto:auth_path=/jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:04.248 DEBUG:boto:Method: GET 2022-09-09T22:53:04.249 DEBUG:boto:Path: /jkcgbj-24/?versions&max-keys=999 2022-09-09T22:53:04.249 DEBUG:boto:Data: 2022-09-09T22:53:04.250 DEBUG:boto:Headers: {} 2022-09-09T22:53:04.250 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:04.251 DEBUG:boto:Port: 8000 2022-09-09T22:53:04.251 DEBUG:boto:Params: {} 2022-09-09T22:53:04.253 DEBUG:boto:Token: None 2022-09-09T22:53:04.254 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:04 GMT /jkcgbj-24/?versions 2022-09-09T22:53:04.254 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R9HNUMOazjZf6ty6vpmzyCdtnAs= 2022-09-09T22:53:04.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R9HNUMOazjZf6ty6vpmzyCdtnAs=', 'Content-Length': '0'} 2022-09-09T22:53:04.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f3d4ed0670fe8a6-00631bc3d0-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:04.261 DEBUG:boto:b'jkcgbj-24999falseobjnulltrue2022-09-09T22:50:32.417Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:04.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:53:04.262 DEBUG:boto:path=/jkcgbj-24/obj 2022-09-09T22:53:04.263 DEBUG:boto:auth_path=/jkcgbj-24/obj 2022-09-09T22:53:04.263 DEBUG:boto:Method: HEAD 2022-09-09T22:53:04.264 DEBUG:boto:Path: /jkcgbj-24/obj 2022-09-09T22:53:04.264 DEBUG:boto:Data: 2022-09-09T22:53:04.265 DEBUG:boto:Headers: {} 2022-09-09T22:53:04.265 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:04.266 DEBUG:boto:Port: 8001 2022-09-09T22:53:04.266 DEBUG:boto:Params: {} 2022-09-09T22:53:04.267 DEBUG:boto:Token: None 2022-09-09T22:53:04.267 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:04 GMT /jkcgbj-24/obj 2022-09-09T22:53:04.268 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ONMzpULxJZJTZ0cK4iku+fyc9J4= 2022-09-09T22:53:04.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ONMzpULxJZJTZ0cK4iku+fyc9J4=', 'Content-Length': '0'} 2022-09-09T22:53:04.272 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000001e6c45e0b30eb6e5-00631bc3d0-1103-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:04.273 DEBUG:boto:path=/jkcgbj-24/obj 2022-09-09T22:53:04.274 DEBUG:boto:auth_path=/jkcgbj-24/obj 2022-09-09T22:53:04.274 DEBUG:boto:Method: HEAD 2022-09-09T22:53:04.275 DEBUG:boto:Path: /jkcgbj-24/obj 2022-09-09T22:53:04.275 DEBUG:boto:Data: 2022-09-09T22:53:04.276 DEBUG:boto:Headers: {} 2022-09-09T22:53:04.276 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:04.277 DEBUG:boto:Port: 8000 2022-09-09T22:53:04.277 DEBUG:boto:Params: {} 2022-09-09T22:53:04.278 DEBUG:boto:Token: None 2022-09-09T22:53:04.278 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:04 GMT /jkcgbj-24/obj 2022-09-09T22:53:04.279 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ONMzpULxJZJTZ0cK4iku+fyc9J4= 2022-09-09T22:53:04.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:04 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ONMzpULxJZJTZ0cK4iku+fyc9J4=', 'Content-Length': '0'} 2022-09-09T22:53:04.283 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000007b17654b8d9ab0d3-00631bc3d0-10e8-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:04.284 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-24 zones={test-zone3, test-zone2} 2022-09-09T22:53:04.284 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jkcgbj-25 2022-09-09T22:53:04.285 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:04.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:04.287 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:53:04.342 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:53:04.343 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:53:04.416 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:04.412+0000 7fcd94be3dc0 1 bucket sync caught up with source: 2022-09-09T22:53:04.417 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000001.8.12, , ] 2022-09-09T22:53:04.417 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000001.8.12, , ] 2022-09-09T22:53:04.417 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:04.412+0000 7fcd94be3dc0 0 bucket checkpoint complete 2022-09-09T22:53:04.451 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jkcgbj-25 2022-09-09T22:53:04.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:04.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:04.452 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:53:04.509 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:53:04.509 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:53:04.578 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:53:04.576+0000 7faed61cbdc0 1 bucket sync caught up with source: 2022-09-09T22:53:04.579 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000001.8.12, , ] 2022-09-09T22:53:04.579 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000001.8.12, , ] 2022-09-09T22:53:04.580 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:53:04.576+0000 7faed61cbdc0 0 bucket checkpoint complete 2022-09-09T22:53:04.605 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jkcgbj-25 2022-09-09T22:53:04.606 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:04.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:04.608 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:53:04.660 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:53:04.661 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:53:04.744 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:04.740+0000 7fc83259adc0 1 bucket sync caught up with source: 2022-09-09T22:53:04.744 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000001.8.12, , ] 2022-09-09T22:53:04.745 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000001.8.12, , ] 2022-09-09T22:53:04.745 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:04.740+0000 7fc83259adc0 0 bucket checkpoint complete 2022-09-09T22:53:04.775 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jkcgbj-25 2022-09-09T22:53:04.775 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:04.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:04.777 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:53:04.833 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:53:04.833 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:53:04.900 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:53:04.896+0000 7fb149b70dc0 1 bucket sync caught up with source: 2022-09-09T22:53:04.901 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000003.69.6, , ] 2022-09-09T22:53:04.901 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000003.69.6, , ] 2022-09-09T22:53:04.902 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:53:04.896+0000 7fb149b70dc0 0 bucket checkpoint complete 2022-09-09T22:53:04.931 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jkcgbj-25 2022-09-09T22:53:04.932 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:04.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:04.933 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:53:04.987 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:53:04.988 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:53:05.055 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:53:05.052+0000 7fac251d3dc0 1 bucket sync caught up with source: 2022-09-09T22:53:05.055 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000003.69.6, , ] 2022-09-09T22:53:05.056 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000003.69.6, , ] 2022-09-09T22:53:05.056 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:53:05.052+0000 7fac251d3dc0 0 bucket checkpoint complete 2022-09-09T22:53:05.082 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jkcgbj-25 2022-09-09T22:53:05.083 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jkcgbj-25 2022-09-09T22:53:05.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:05.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:05.085 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:53:05.136 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:53:05.137 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:53:05.204 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:53:05.200+0000 7ff417ce2dc0 1 bucket sync caught up with source: 2022-09-09T22:53:05.205 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000001.47.12, , ] 2022-09-09T22:53:05.205 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000001.47.12, , ] 2022-09-09T22:53:05.206 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:53:05.200+0000 7ff417ce2dc0 0 bucket checkpoint complete 2022-09-09T22:53:05.230 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jkcgbj-25 2022-09-09T22:53:05.231 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:05.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:05.232 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:53:05.285 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:53:05.286 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:53:05.353 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:05.348+0000 7fdeef7b7dc0 1 bucket sync caught up with source: 2022-09-09T22:53:05.353 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000001.47.12, , ] 2022-09-09T22:53:05.354 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000001.47.12, , ] 2022-09-09T22:53:05.354 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:05.348+0000 7fdeef7b7dc0 0 bucket checkpoint complete 2022-09-09T22:53:05.378 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jkcgbj-25 2022-09-09T22:53:05.379 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-25 zones={test-zone2, test-zone1} 2022-09-09T22:53:05.380 DEBUG:boto:path=/jkcgbj-25/ 2022-09-09T22:53:05.380 DEBUG:boto:auth_path=/jkcgbj-25/ 2022-09-09T22:53:05.381 DEBUG:boto:Method: HEAD 2022-09-09T22:53:05.382 DEBUG:boto:Path: /jkcgbj-25/ 2022-09-09T22:53:05.383 DEBUG:boto:Data: 2022-09-09T22:53:05.383 DEBUG:boto:Headers: {} 2022-09-09T22:53:05.384 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:05.384 DEBUG:boto:Port: 8000 2022-09-09T22:53:05.385 DEBUG:boto:Params: {} 2022-09-09T22:53:05.386 DEBUG:boto:Token: None 2022-09-09T22:53:05.387 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:05 GMT /jkcgbj-25/ 2022-09-09T22:53:05.388 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Uv2okMZEIszxWriEleii/+MC4b8= 2022-09-09T22:53:05.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Uv2okMZEIszxWriEleii/+MC4b8=', 'Content-Length': '0'} 2022-09-09T22:53:05.392 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fe6a0690cc05f003-00631bc3d1-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:05.393 DEBUG:boto:path=/jkcgbj-25/ 2022-09-09T22:53:05.394 DEBUG:boto:auth_path=/jkcgbj-25/ 2022-09-09T22:53:05.395 DEBUG:boto:Method: HEAD 2022-09-09T22:53:05.395 DEBUG:boto:Path: /jkcgbj-25/ 2022-09-09T22:53:05.396 DEBUG:boto:Data: 2022-09-09T22:53:05.397 DEBUG:boto:Headers: {} 2022-09-09T22:53:05.397 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:05.398 DEBUG:boto:Port: 8000 2022-09-09T22:53:05.398 DEBUG:boto:Params: {} 2022-09-09T22:53:05.399 DEBUG:boto:Token: None 2022-09-09T22:53:05.399 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:05 GMT /jkcgbj-25/ 2022-09-09T22:53:05.400 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Uv2okMZEIszxWriEleii/+MC4b8= 2022-09-09T22:53:05.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Uv2okMZEIszxWriEleii/+MC4b8=', 'Content-Length': '0'} 2022-09-09T22:53:05.406 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000059f50104e0593d50-00631bc3d1-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:05.407 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:53:05.408 DEBUG:boto:path=/jkcgbj-25/ 2022-09-09T22:53:05.408 DEBUG:boto:auth_path=/jkcgbj-25/ 2022-09-09T22:53:05.409 DEBUG:boto:path=/jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.409 DEBUG:boto:auth_path=/jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.410 DEBUG:boto:Method: GET 2022-09-09T22:53:05.410 DEBUG:boto:Path: /jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.411 DEBUG:boto:Data: 2022-09-09T22:53:05.412 DEBUG:boto:Headers: {} 2022-09-09T22:53:05.413 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:05.413 DEBUG:boto:Port: 8000 2022-09-09T22:53:05.414 DEBUG:boto:Params: {} 2022-09-09T22:53:05.414 DEBUG:boto:Token: None 2022-09-09T22:53:05.415 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:05 GMT /jkcgbj-25/?versions 2022-09-09T22:53:05.415 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:VY37JvCgcJkVQzBk7CZntuGxtds= 2022-09-09T22:53:05.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:VY37JvCgcJkVQzBk7CZntuGxtds=', 'Content-Length': '0'} 2022-09-09T22:53:05.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a63ea5aa1cf608f-00631bc3d1-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:05.467 DEBUG:boto:b'jkcgbj-25999falseobjnulltrue2022-09-09T22:50:32.532Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:05.468 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:53:05.469 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:53:05.469 DEBUG:boto:path=/jkcgbj-25/ 2022-09-09T22:53:05.470 DEBUG:boto:auth_path=/jkcgbj-25/ 2022-09-09T22:53:05.471 DEBUG:boto:path=/jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.471 DEBUG:boto:auth_path=/jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.472 DEBUG:boto:Method: GET 2022-09-09T22:53:05.473 DEBUG:boto:Path: /jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.476 DEBUG:boto:Data: 2022-09-09T22:53:05.477 DEBUG:boto:Headers: {} 2022-09-09T22:53:05.478 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:05.479 DEBUG:boto:Port: 8000 2022-09-09T22:53:05.480 DEBUG:boto:Params: {} 2022-09-09T22:53:05.481 DEBUG:boto:Token: None 2022-09-09T22:53:05.482 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:05 GMT /jkcgbj-25/?versions 2022-09-09T22:53:05.483 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:VY37JvCgcJkVQzBk7CZntuGxtds= 2022-09-09T22:53:05.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:VY37JvCgcJkVQzBk7CZntuGxtds=', 'Content-Length': '0'} 2022-09-09T22:53:05.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c20262894d367f2-00631bc3d1-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:05.490 DEBUG:boto:b'jkcgbj-25999falseobjnulltrue2022-09-09T22:50:32.532Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:05.491 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:53:05.492 DEBUG:boto:path=/jkcgbj-25/ 2022-09-09T22:53:05.492 DEBUG:boto:auth_path=/jkcgbj-25/ 2022-09-09T22:53:05.493 DEBUG:boto:path=/jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.494 DEBUG:boto:auth_path=/jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.494 DEBUG:boto:Method: GET 2022-09-09T22:53:05.495 DEBUG:boto:Path: /jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.495 DEBUG:boto:Data: 2022-09-09T22:53:05.496 DEBUG:boto:Headers: {} 2022-09-09T22:53:05.497 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:05.497 DEBUG:boto:Port: 8000 2022-09-09T22:53:05.498 DEBUG:boto:Params: {} 2022-09-09T22:53:05.498 DEBUG:boto:Token: None 2022-09-09T22:53:05.499 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:05 GMT /jkcgbj-25/?versions 2022-09-09T22:53:05.500 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:VY37JvCgcJkVQzBk7CZntuGxtds= 2022-09-09T22:53:05.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:VY37JvCgcJkVQzBk7CZntuGxtds=', 'Content-Length': '0'} 2022-09-09T22:53:05.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f18822f47b4b43f-00631bc3d1-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:05.508 DEBUG:boto:b'jkcgbj-25999falseobjnulltrue2022-09-09T22:50:32.532Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:05.508 DEBUG:boto:path=/jkcgbj-25/ 2022-09-09T22:53:05.509 DEBUG:boto:auth_path=/jkcgbj-25/ 2022-09-09T22:53:05.510 DEBUG:boto:path=/jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.510 DEBUG:boto:auth_path=/jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.511 DEBUG:boto:Method: GET 2022-09-09T22:53:05.511 DEBUG:boto:Path: /jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.512 DEBUG:boto:Data: 2022-09-09T22:53:05.512 DEBUG:boto:Headers: {} 2022-09-09T22:53:05.513 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:05.513 DEBUG:boto:Port: 8000 2022-09-09T22:53:05.514 DEBUG:boto:Params: {} 2022-09-09T22:53:05.514 DEBUG:boto:Token: None 2022-09-09T22:53:05.515 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:05 GMT /jkcgbj-25/?versions 2022-09-09T22:53:05.516 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:VY37JvCgcJkVQzBk7CZntuGxtds= 2022-09-09T22:53:05.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:VY37JvCgcJkVQzBk7CZntuGxtds=', 'Content-Length': '0'} 2022-09-09T22:53:05.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f4bc158afd78503-00631bc3d1-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:05.566 DEBUG:boto:b'jkcgbj-25999falseobjnulltrue2022-09-09T22:50:32.532Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:05.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:53:05.568 DEBUG:boto:path=/jkcgbj-25/obj 2022-09-09T22:53:05.568 DEBUG:boto:auth_path=/jkcgbj-25/obj 2022-09-09T22:53:05.569 DEBUG:boto:Method: HEAD 2022-09-09T22:53:05.569 DEBUG:boto:Path: /jkcgbj-25/obj 2022-09-09T22:53:05.570 DEBUG:boto:Data: 2022-09-09T22:53:05.570 DEBUG:boto:Headers: {} 2022-09-09T22:53:05.571 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:05.571 DEBUG:boto:Port: 8000 2022-09-09T22:53:05.572 DEBUG:boto:Params: {} 2022-09-09T22:53:05.572 DEBUG:boto:Token: None 2022-09-09T22:53:05.573 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:05 GMT /jkcgbj-25/obj 2022-09-09T22:53:05.574 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6bfQPlqtx+DBANcOlnPbLzI3jp8= 2022-09-09T22:53:05.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6bfQPlqtx+DBANcOlnPbLzI3jp8=', 'Content-Length': '0'} 2022-09-09T22:53:05.578 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000e1200fd4fb6209fa-00631bc3d1-10e8-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:05.579 DEBUG:boto:path=/jkcgbj-25/obj 2022-09-09T22:53:05.579 DEBUG:boto:auth_path=/jkcgbj-25/obj 2022-09-09T22:53:05.580 DEBUG:boto:Method: HEAD 2022-09-09T22:53:05.580 DEBUG:boto:Path: /jkcgbj-25/obj 2022-09-09T22:53:05.581 DEBUG:boto:Data: 2022-09-09T22:53:05.581 DEBUG:boto:Headers: {} 2022-09-09T22:53:05.582 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:05.582 DEBUG:boto:Port: 8000 2022-09-09T22:53:05.584 DEBUG:boto:Params: {} 2022-09-09T22:53:05.584 DEBUG:boto:Token: None 2022-09-09T22:53:05.585 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:05 GMT /jkcgbj-25/obj 2022-09-09T22:53:05.586 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6bfQPlqtx+DBANcOlnPbLzI3jp8= 2022-09-09T22:53:05.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6bfQPlqtx+DBANcOlnPbLzI3jp8=', 'Content-Length': '0'} 2022-09-09T22:53:05.590 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000086487f8cea3c5b0a-00631bc3d1-10ff-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:05.590 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-25 zones={test-zone2, test-zone1} 2022-09-09T22:53:05.591 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-25 zones={test-zone3, test-zone1} 2022-09-09T22:53:05.591 DEBUG:boto:path=/jkcgbj-25/ 2022-09-09T22:53:05.592 DEBUG:boto:auth_path=/jkcgbj-25/ 2022-09-09T22:53:05.592 DEBUG:boto:Method: HEAD 2022-09-09T22:53:05.593 DEBUG:boto:Path: /jkcgbj-25/ 2022-09-09T22:53:05.594 DEBUG:boto:Data: 2022-09-09T22:53:05.594 DEBUG:boto:Headers: {} 2022-09-09T22:53:05.595 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:05.596 DEBUG:boto:Port: 8001 2022-09-09T22:53:05.596 DEBUG:boto:Params: {} 2022-09-09T22:53:05.597 DEBUG:boto:Token: None 2022-09-09T22:53:05.598 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:05 GMT /jkcgbj-25/ 2022-09-09T22:53:05.598 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Uv2okMZEIszxWriEleii/+MC4b8= 2022-09-09T22:53:05.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Uv2okMZEIszxWriEleii/+MC4b8=', 'Content-Length': '0'} 2022-09-09T22:53:05.605 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c23781410b32f77e-00631bc3d1-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:05.606 DEBUG:boto:path=/jkcgbj-25/ 2022-09-09T22:53:05.607 DEBUG:boto:auth_path=/jkcgbj-25/ 2022-09-09T22:53:05.607 DEBUG:boto:Method: HEAD 2022-09-09T22:53:05.608 DEBUG:boto:Path: /jkcgbj-25/ 2022-09-09T22:53:05.608 DEBUG:boto:Data: 2022-09-09T22:53:05.609 DEBUG:boto:Headers: {} 2022-09-09T22:53:05.609 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:05.610 DEBUG:boto:Port: 8000 2022-09-09T22:53:05.610 DEBUG:boto:Params: {} 2022-09-09T22:53:05.611 DEBUG:boto:Token: None 2022-09-09T22:53:05.612 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:05 GMT /jkcgbj-25/ 2022-09-09T22:53:05.612 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Uv2okMZEIszxWriEleii/+MC4b8= 2022-09-09T22:53:05.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Uv2okMZEIszxWriEleii/+MC4b8=', 'Content-Length': '0'} 2022-09-09T22:53:05.618 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000075d779275349fd09-00631bc3d1-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:05.619 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:53:05.620 DEBUG:boto:path=/jkcgbj-25/ 2022-09-09T22:53:05.621 DEBUG:boto:auth_path=/jkcgbj-25/ 2022-09-09T22:53:05.622 DEBUG:boto:path=/jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.623 DEBUG:boto:auth_path=/jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.624 DEBUG:boto:Method: GET 2022-09-09T22:53:05.625 DEBUG:boto:Path: /jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.626 DEBUG:boto:Data: 2022-09-09T22:53:05.627 DEBUG:boto:Headers: {} 2022-09-09T22:53:05.628 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:05.629 DEBUG:boto:Port: 8001 2022-09-09T22:53:05.629 DEBUG:boto:Params: {} 2022-09-09T22:53:05.630 DEBUG:boto:Token: None 2022-09-09T22:53:05.631 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:05 GMT /jkcgbj-25/?versions 2022-09-09T22:53:05.632 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:VY37JvCgcJkVQzBk7CZntuGxtds= 2022-09-09T22:53:05.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:VY37JvCgcJkVQzBk7CZntuGxtds=', 'Content-Length': '0'} 2022-09-09T22:53:05.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3bb30a0be023dae-00631bc3d1-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:05.682 DEBUG:boto:b'jkcgbj-25999falseobjnulltrue2022-09-09T22:50:32.532Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:05.684 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:53:05.685 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:53:05.686 DEBUG:boto:path=/jkcgbj-25/ 2022-09-09T22:53:05.687 DEBUG:boto:auth_path=/jkcgbj-25/ 2022-09-09T22:53:05.687 DEBUG:boto:path=/jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.688 DEBUG:boto:auth_path=/jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.688 DEBUG:boto:Method: GET 2022-09-09T22:53:05.689 DEBUG:boto:Path: /jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.689 DEBUG:boto:Data: 2022-09-09T22:53:05.690 DEBUG:boto:Headers: {} 2022-09-09T22:53:05.690 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:05.691 DEBUG:boto:Port: 8000 2022-09-09T22:53:05.691 DEBUG:boto:Params: {} 2022-09-09T22:53:05.692 DEBUG:boto:Token: None 2022-09-09T22:53:05.693 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:05 GMT /jkcgbj-25/?versions 2022-09-09T22:53:05.693 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:VY37JvCgcJkVQzBk7CZntuGxtds= 2022-09-09T22:53:05.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:VY37JvCgcJkVQzBk7CZntuGxtds=', 'Content-Length': '0'} 2022-09-09T22:53:05.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052881f1203408c74-00631bc3d1-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:05.701 DEBUG:boto:b'jkcgbj-25999falseobjnulltrue2022-09-09T22:50:32.532Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:05.702 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:53:05.702 DEBUG:boto:path=/jkcgbj-25/ 2022-09-09T22:53:05.703 DEBUG:boto:auth_path=/jkcgbj-25/ 2022-09-09T22:53:05.703 DEBUG:boto:path=/jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.704 DEBUG:boto:auth_path=/jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.704 DEBUG:boto:Method: GET 2022-09-09T22:53:05.705 DEBUG:boto:Path: /jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.705 DEBUG:boto:Data: 2022-09-09T22:53:05.706 DEBUG:boto:Headers: {} 2022-09-09T22:53:05.706 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:05.707 DEBUG:boto:Port: 8001 2022-09-09T22:53:05.707 DEBUG:boto:Params: {} 2022-09-09T22:53:05.708 DEBUG:boto:Token: None 2022-09-09T22:53:05.709 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:05 GMT /jkcgbj-25/?versions 2022-09-09T22:53:05.709 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:VY37JvCgcJkVQzBk7CZntuGxtds= 2022-09-09T22:53:05.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:VY37JvCgcJkVQzBk7CZntuGxtds=', 'Content-Length': '0'} 2022-09-09T22:53:05.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000320074daafee4d3e-00631bc3d1-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:05.717 DEBUG:boto:b'jkcgbj-25999falseobjnulltrue2022-09-09T22:50:32.532Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:05.718 DEBUG:boto:path=/jkcgbj-25/ 2022-09-09T22:53:05.719 DEBUG:boto:auth_path=/jkcgbj-25/ 2022-09-09T22:53:05.720 DEBUG:boto:path=/jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.721 DEBUG:boto:auth_path=/jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.722 DEBUG:boto:Method: GET 2022-09-09T22:53:05.723 DEBUG:boto:Path: /jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.724 DEBUG:boto:Data: 2022-09-09T22:53:05.725 DEBUG:boto:Headers: {} 2022-09-09T22:53:05.726 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:05.727 DEBUG:boto:Port: 8000 2022-09-09T22:53:05.728 DEBUG:boto:Params: {} 2022-09-09T22:53:05.729 DEBUG:boto:Token: None 2022-09-09T22:53:05.730 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:05 GMT /jkcgbj-25/?versions 2022-09-09T22:53:05.731 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:VY37JvCgcJkVQzBk7CZntuGxtds= 2022-09-09T22:53:05.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:VY37JvCgcJkVQzBk7CZntuGxtds=', 'Content-Length': '0'} 2022-09-09T22:53:05.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fceb61b58f20ee8-00631bc3d1-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:05.738 DEBUG:boto:b'jkcgbj-25999falseobjnulltrue2022-09-09T22:50:32.532Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:05.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:53:05.740 DEBUG:boto:path=/jkcgbj-25/obj 2022-09-09T22:53:05.741 DEBUG:boto:auth_path=/jkcgbj-25/obj 2022-09-09T22:53:05.743 DEBUG:boto:Method: HEAD 2022-09-09T22:53:05.745 DEBUG:boto:Path: /jkcgbj-25/obj 2022-09-09T22:53:05.746 DEBUG:boto:Data: 2022-09-09T22:53:05.746 DEBUG:boto:Headers: {} 2022-09-09T22:53:05.747 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:05.747 DEBUG:boto:Port: 8001 2022-09-09T22:53:05.748 DEBUG:boto:Params: {} 2022-09-09T22:53:05.748 DEBUG:boto:Token: None 2022-09-09T22:53:05.749 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:05 GMT /jkcgbj-25/obj 2022-09-09T22:53:05.750 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6bfQPlqtx+DBANcOlnPbLzI3jp8= 2022-09-09T22:53:05.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6bfQPlqtx+DBANcOlnPbLzI3jp8=', 'Content-Length': '0'} 2022-09-09T22:53:05.754 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000008206cc5e908dbfbb-00631bc3d1-1103-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:05.756 DEBUG:boto:path=/jkcgbj-25/obj 2022-09-09T22:53:05.756 DEBUG:boto:auth_path=/jkcgbj-25/obj 2022-09-09T22:53:05.757 DEBUG:boto:Method: HEAD 2022-09-09T22:53:05.758 DEBUG:boto:Path: /jkcgbj-25/obj 2022-09-09T22:53:05.759 DEBUG:boto:Data: 2022-09-09T22:53:05.760 DEBUG:boto:Headers: {} 2022-09-09T22:53:05.761 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:05.762 DEBUG:boto:Port: 8000 2022-09-09T22:53:05.763 DEBUG:boto:Params: {} 2022-09-09T22:53:05.764 DEBUG:boto:Token: None 2022-09-09T22:53:05.764 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:05 GMT /jkcgbj-25/obj 2022-09-09T22:53:05.765 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6bfQPlqtx+DBANcOlnPbLzI3jp8= 2022-09-09T22:53:05.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6bfQPlqtx+DBANcOlnPbLzI3jp8=', 'Content-Length': '0'} 2022-09-09T22:53:05.769 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000089b2492f08fe8d5e-00631bc3d1-10ff-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:05.769 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-25 zones={test-zone3, test-zone1} 2022-09-09T22:53:05.770 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-25 zones={test-zone3, test-zone2} 2022-09-09T22:53:05.770 DEBUG:boto:path=/jkcgbj-25/ 2022-09-09T22:53:05.771 DEBUG:boto:auth_path=/jkcgbj-25/ 2022-09-09T22:53:05.771 DEBUG:boto:Method: HEAD 2022-09-09T22:53:05.772 DEBUG:boto:Path: /jkcgbj-25/ 2022-09-09T22:53:05.772 DEBUG:boto:Data: 2022-09-09T22:53:05.773 DEBUG:boto:Headers: {} 2022-09-09T22:53:05.773 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:05.774 DEBUG:boto:Port: 8001 2022-09-09T22:53:05.774 DEBUG:boto:Params: {} 2022-09-09T22:53:05.775 DEBUG:boto:Token: None 2022-09-09T22:53:05.776 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:05 GMT /jkcgbj-25/ 2022-09-09T22:53:05.777 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Uv2okMZEIszxWriEleii/+MC4b8= 2022-09-09T22:53:05.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Uv2okMZEIszxWriEleii/+MC4b8=', 'Content-Length': '0'} 2022-09-09T22:53:05.785 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f5a99fc5cfce4f5b-00631bc3d1-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:05.786 DEBUG:boto:path=/jkcgbj-25/ 2022-09-09T22:53:05.787 DEBUG:boto:auth_path=/jkcgbj-25/ 2022-09-09T22:53:05.788 DEBUG:boto:Method: HEAD 2022-09-09T22:53:05.788 DEBUG:boto:Path: /jkcgbj-25/ 2022-09-09T22:53:05.789 DEBUG:boto:Data: 2022-09-09T22:53:05.790 DEBUG:boto:Headers: {} 2022-09-09T22:53:05.790 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:05.791 DEBUG:boto:Port: 8000 2022-09-09T22:53:05.791 DEBUG:boto:Params: {} 2022-09-09T22:53:05.792 DEBUG:boto:Token: None 2022-09-09T22:53:05.792 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:05 GMT /jkcgbj-25/ 2022-09-09T22:53:05.793 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Uv2okMZEIszxWriEleii/+MC4b8= 2022-09-09T22:53:05.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Uv2okMZEIszxWriEleii/+MC4b8=', 'Content-Length': '0'} 2022-09-09T22:53:05.798 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003d9a3aacba9c3f7e-00631bc3d1-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:05.799 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:53:05.799 DEBUG:boto:path=/jkcgbj-25/ 2022-09-09T22:53:05.800 DEBUG:boto:auth_path=/jkcgbj-25/ 2022-09-09T22:53:05.800 DEBUG:boto:path=/jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.801 DEBUG:boto:auth_path=/jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.801 DEBUG:boto:Method: GET 2022-09-09T22:53:05.802 DEBUG:boto:Path: /jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.802 DEBUG:boto:Data: 2022-09-09T22:53:05.803 DEBUG:boto:Headers: {} 2022-09-09T22:53:05.803 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:05.804 DEBUG:boto:Port: 8001 2022-09-09T22:53:05.804 DEBUG:boto:Params: {} 2022-09-09T22:53:05.805 DEBUG:boto:Token: None 2022-09-09T22:53:05.805 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:05 GMT /jkcgbj-25/?versions 2022-09-09T22:53:05.806 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:VY37JvCgcJkVQzBk7CZntuGxtds= 2022-09-09T22:53:05.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:VY37JvCgcJkVQzBk7CZntuGxtds=', 'Content-Length': '0'} 2022-09-09T22:53:05.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b7a3353399750f1-00631bc3d1-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:05.858 DEBUG:boto:b'jkcgbj-25999falseobjnulltrue2022-09-09T22:50:32.532Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:05.859 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:53:05.860 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:53:05.861 DEBUG:boto:path=/jkcgbj-25/ 2022-09-09T22:53:05.862 DEBUG:boto:auth_path=/jkcgbj-25/ 2022-09-09T22:53:05.864 DEBUG:boto:path=/jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.865 DEBUG:boto:auth_path=/jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.865 DEBUG:boto:Method: GET 2022-09-09T22:53:05.866 DEBUG:boto:Path: /jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.867 DEBUG:boto:Data: 2022-09-09T22:53:05.868 DEBUG:boto:Headers: {} 2022-09-09T22:53:05.869 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:05.870 DEBUG:boto:Port: 8000 2022-09-09T22:53:05.871 DEBUG:boto:Params: {} 2022-09-09T22:53:05.871 DEBUG:boto:Token: None 2022-09-09T22:53:05.872 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:05 GMT /jkcgbj-25/?versions 2022-09-09T22:53:05.873 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:VY37JvCgcJkVQzBk7CZntuGxtds= 2022-09-09T22:53:05.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:VY37JvCgcJkVQzBk7CZntuGxtds=', 'Content-Length': '0'} 2022-09-09T22:53:05.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8a845eb5c6e311d-00631bc3d1-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:05.880 DEBUG:boto:b'jkcgbj-25999falseobjnulltrue2022-09-09T22:50:32.532Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:05.881 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:53:05.882 DEBUG:boto:path=/jkcgbj-25/ 2022-09-09T22:53:05.882 DEBUG:boto:auth_path=/jkcgbj-25/ 2022-09-09T22:53:05.883 DEBUG:boto:path=/jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.884 DEBUG:boto:auth_path=/jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.884 DEBUG:boto:Method: GET 2022-09-09T22:53:05.885 DEBUG:boto:Path: /jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.885 DEBUG:boto:Data: 2022-09-09T22:53:05.886 DEBUG:boto:Headers: {} 2022-09-09T22:53:05.887 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:05.889 DEBUG:boto:Port: 8001 2022-09-09T22:53:05.890 DEBUG:boto:Params: {} 2022-09-09T22:53:05.891 DEBUG:boto:Token: None 2022-09-09T22:53:05.892 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:05 GMT /jkcgbj-25/?versions 2022-09-09T22:53:05.893 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:VY37JvCgcJkVQzBk7CZntuGxtds= 2022-09-09T22:53:05.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:VY37JvCgcJkVQzBk7CZntuGxtds=', 'Content-Length': '0'} 2022-09-09T22:53:05.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b9b022dcced8fdb-00631bc3d1-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:05.901 DEBUG:boto:b'jkcgbj-25999falseobjnulltrue2022-09-09T22:50:32.532Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:05.902 DEBUG:boto:path=/jkcgbj-25/ 2022-09-09T22:53:05.903 DEBUG:boto:auth_path=/jkcgbj-25/ 2022-09-09T22:53:05.903 DEBUG:boto:path=/jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.904 DEBUG:boto:auth_path=/jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.905 DEBUG:boto:Method: GET 2022-09-09T22:53:05.906 DEBUG:boto:Path: /jkcgbj-25/?versions&max-keys=999 2022-09-09T22:53:05.907 DEBUG:boto:Data: 2022-09-09T22:53:05.907 DEBUG:boto:Headers: {} 2022-09-09T22:53:05.908 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:05.909 DEBUG:boto:Port: 8000 2022-09-09T22:53:05.910 DEBUG:boto:Params: {} 2022-09-09T22:53:05.911 DEBUG:boto:Token: None 2022-09-09T22:53:05.912 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:05 GMT /jkcgbj-25/?versions 2022-09-09T22:53:05.913 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:VY37JvCgcJkVQzBk7CZntuGxtds= 2022-09-09T22:53:05.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:VY37JvCgcJkVQzBk7CZntuGxtds=', 'Content-Length': '0'} 2022-09-09T22:53:05.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b666327a85620f31-00631bc3d1-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:05.920 DEBUG:boto:b'jkcgbj-25999falseobjnulltrue2022-09-09T22:50:32.532Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:05.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:53:05.922 DEBUG:boto:path=/jkcgbj-25/obj 2022-09-09T22:53:05.923 DEBUG:boto:auth_path=/jkcgbj-25/obj 2022-09-09T22:53:05.924 DEBUG:boto:Method: HEAD 2022-09-09T22:53:05.925 DEBUG:boto:Path: /jkcgbj-25/obj 2022-09-09T22:53:05.926 DEBUG:boto:Data: 2022-09-09T22:53:05.927 DEBUG:boto:Headers: {} 2022-09-09T22:53:05.928 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:05.929 DEBUG:boto:Port: 8001 2022-09-09T22:53:05.931 DEBUG:boto:Params: {} 2022-09-09T22:53:05.931 DEBUG:boto:Token: None 2022-09-09T22:53:05.932 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:05 GMT /jkcgbj-25/obj 2022-09-09T22:53:05.932 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6bfQPlqtx+DBANcOlnPbLzI3jp8= 2022-09-09T22:53:05.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6bfQPlqtx+DBANcOlnPbLzI3jp8=', 'Content-Length': '0'} 2022-09-09T22:53:05.937 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000005e1cb92b67103034-00631bc3d1-1103-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:05.938 DEBUG:boto:path=/jkcgbj-25/obj 2022-09-09T22:53:05.938 DEBUG:boto:auth_path=/jkcgbj-25/obj 2022-09-09T22:53:05.939 DEBUG:boto:Method: HEAD 2022-09-09T22:53:05.939 DEBUG:boto:Path: /jkcgbj-25/obj 2022-09-09T22:53:05.940 DEBUG:boto:Data: 2022-09-09T22:53:05.940 DEBUG:boto:Headers: {} 2022-09-09T22:53:05.941 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:05.941 DEBUG:boto:Port: 8000 2022-09-09T22:53:05.942 DEBUG:boto:Params: {} 2022-09-09T22:53:05.942 DEBUG:boto:Token: None 2022-09-09T22:53:05.943 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:05 GMT /jkcgbj-25/obj 2022-09-09T22:53:05.943 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6bfQPlqtx+DBANcOlnPbLzI3jp8= 2022-09-09T22:53:05.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:05 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6bfQPlqtx+DBANcOlnPbLzI3jp8=', 'Content-Length': '0'} 2022-09-09T22:53:05.947 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000064c90aca82a46da8-00631bc3d1-10e8-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:05.948 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-25 zones={test-zone3, test-zone2} 2022-09-09T22:53:05.948 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jkcgbj-26 2022-09-09T22:53:05.949 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:05.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:05.950 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:53:06.002 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:53:06.003 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:53:06.074 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:06.068+0000 7effa2c31dc0 1 bucket sync caught up with source: 2022-09-09T22:53:06.074 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000001.34.12, , ] 2022-09-09T22:53:06.074 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000001.34.12, , ] 2022-09-09T22:53:06.075 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:06.068+0000 7effa2c31dc0 0 bucket checkpoint complete 2022-09-09T22:53:06.102 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jkcgbj-26 2022-09-09T22:53:06.103 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:06.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:06.104 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:53:06.160 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:53:06.160 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:53:06.229 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:53:06.224+0000 7f07bc0f0dc0 1 bucket sync caught up with source: 2022-09-09T22:53:06.230 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000001.34.12, , ] 2022-09-09T22:53:06.230 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000001.34.12, , ] 2022-09-09T22:53:06.230 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:53:06.224+0000 7f07bc0f0dc0 0 bucket checkpoint complete 2022-09-09T22:53:06.254 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jkcgbj-26 2022-09-09T22:53:06.255 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:06.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:06.256 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:53:06.309 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:53:06.310 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:53:06.395 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:06.392+0000 7f8b691c7dc0 1 bucket sync caught up with source: 2022-09-09T22:53:06.395 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000001.34.12, , ] 2022-09-09T22:53:06.395 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000001.34.12, , ] 2022-09-09T22:53:06.396 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:06.392+0000 7f8b691c7dc0 0 bucket checkpoint complete 2022-09-09T22:53:06.420 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jkcgbj-26 2022-09-09T22:53:06.421 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:06.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:06.422 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:53:06.477 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:53:06.478 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:53:06.543 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:53:06.540+0000 7f9e73cefdc0 1 bucket sync caught up with source: 2022-09-09T22:53:06.544 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000001.65.12, , ] 2022-09-09T22:53:06.544 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000001.65.12, , ] 2022-09-09T22:53:06.544 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:53:06.540+0000 7f9e73cefdc0 0 bucket checkpoint complete 2022-09-09T22:53:06.572 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jkcgbj-26 2022-09-09T22:53:06.573 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:06.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:06.574 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:53:06.626 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:53:06.627 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:53:06.695 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:53:06.692+0000 7fb6f5274dc0 1 bucket sync caught up with source: 2022-09-09T22:53:06.696 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000001.65.12, , ] 2022-09-09T22:53:06.696 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000001.65.12, , ] 2022-09-09T22:53:06.696 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:53:06.692+0000 7fb6f5274dc0 0 bucket checkpoint complete 2022-09-09T22:53:06.723 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jkcgbj-26 2022-09-09T22:53:06.723 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jkcgbj-26 2022-09-09T22:53:06.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:06.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:06.725 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jkcgbj-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-09T22:53:06.781 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:53:06.782 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:53:06.848 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:53:06.844+0000 7f8031cabdc0 1 bucket sync caught up with source: 2022-09-09T22:53:06.849 INFO:teuthology.orchestra.run.smithi005.stderr: local status: [, , , , , , , , 00000000003.80.6, , ] 2022-09-09T22:53:06.849 INFO:teuthology.orchestra.run.smithi005.stderr: remote markers: [, , , , , , , , 00000000003.80.6, , ] 2022-09-09T22:53:06.849 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:53:06.844+0000 7f8031cabdc0 0 bucket checkpoint complete 2022-09-09T22:53:06.875 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jkcgbj-26 2022-09-09T22:53:06.876 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:06.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jkcgbj-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-09T22:53:06.878 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jkcgbj-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-09T22:53:06.933 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:53:06.934 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:53:07.016 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:07.012+0000 7fdea396fdc0 1 bucket sync caught up with source: 2022-09-09T22:53:07.016 INFO:teuthology.orchestra.run.smithi080.stderr: local status: [, , , , , , , , 00000000003.80.6, , ] 2022-09-09T22:53:07.016 INFO:teuthology.orchestra.run.smithi080.stderr: remote markers: [, , , , , , , , 00000000003.80.6, , ] 2022-09-09T22:53:07.016 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:07.012+0000 7fdea396fdc0 0 bucket checkpoint complete 2022-09-09T22:53:07.049 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jkcgbj-26 2022-09-09T22:53:07.050 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-26 zones={test-zone2, test-zone1} 2022-09-09T22:53:07.051 DEBUG:boto:path=/jkcgbj-26/ 2022-09-09T22:53:07.051 DEBUG:boto:auth_path=/jkcgbj-26/ 2022-09-09T22:53:07.052 DEBUG:boto:Method: HEAD 2022-09-09T22:53:07.052 DEBUG:boto:Path: /jkcgbj-26/ 2022-09-09T22:53:07.053 DEBUG:boto:Data: 2022-09-09T22:53:07.053 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.054 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:07.054 DEBUG:boto:Port: 8000 2022-09-09T22:53:07.055 DEBUG:boto:Params: {} 2022-09-09T22:53:07.055 DEBUG:boto:Token: None 2022-09-09T22:53:07.056 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-26/ 2022-09-09T22:53:07.057 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ktBvAnrbvkLStpKRHy58/CaI9dM= 2022-09-09T22:53:07.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ktBvAnrbvkLStpKRHy58/CaI9dM=', 'Content-Length': '0'} 2022-09-09T22:53:07.062 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000291f4fe668f0b99b-00631bc3d3-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.063 DEBUG:boto:path=/jkcgbj-26/ 2022-09-09T22:53:07.063 DEBUG:boto:auth_path=/jkcgbj-26/ 2022-09-09T22:53:07.064 DEBUG:boto:Method: HEAD 2022-09-09T22:53:07.064 DEBUG:boto:Path: /jkcgbj-26/ 2022-09-09T22:53:07.065 DEBUG:boto:Data: 2022-09-09T22:53:07.065 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.066 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:07.066 DEBUG:boto:Port: 8000 2022-09-09T22:53:07.067 DEBUG:boto:Params: {} 2022-09-09T22:53:07.068 DEBUG:boto:Token: None 2022-09-09T22:53:07.069 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-26/ 2022-09-09T22:53:07.070 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ktBvAnrbvkLStpKRHy58/CaI9dM= 2022-09-09T22:53:07.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ktBvAnrbvkLStpKRHy58/CaI9dM=', 'Content-Length': '0'} 2022-09-09T22:53:07.075 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006937a8cae5785fd9-00631bc3d3-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.076 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:53:07.077 DEBUG:boto:path=/jkcgbj-26/ 2022-09-09T22:53:07.078 DEBUG:boto:auth_path=/jkcgbj-26/ 2022-09-09T22:53:07.079 DEBUG:boto:path=/jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.080 DEBUG:boto:auth_path=/jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.080 DEBUG:boto:Method: GET 2022-09-09T22:53:07.081 DEBUG:boto:Path: /jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.081 DEBUG:boto:Data: 2022-09-09T22:53:07.083 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.083 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:07.084 DEBUG:boto:Port: 8000 2022-09-09T22:53:07.084 DEBUG:boto:Params: {} 2022-09-09T22:53:07.085 DEBUG:boto:Token: None 2022-09-09T22:53:07.085 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-26/?versions 2022-09-09T22:53:07.086 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ua6U4ii0uDEIvZh77BfVpMWEe24= 2022-09-09T22:53:07.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ua6U4ii0uDEIvZh77BfVpMWEe24=', 'Content-Length': '0'} 2022-09-09T22:53:07.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a5c6c41ca1e224d-00631bc3d3-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.094 DEBUG:boto:b'jkcgbj-26999falseobjnulltrue2022-09-09T22:50:32.650Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:07.096 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:53:07.097 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:53:07.098 DEBUG:boto:path=/jkcgbj-26/ 2022-09-09T22:53:07.099 DEBUG:boto:auth_path=/jkcgbj-26/ 2022-09-09T22:53:07.099 DEBUG:boto:path=/jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.100 DEBUG:boto:auth_path=/jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.101 DEBUG:boto:Method: GET 2022-09-09T22:53:07.102 DEBUG:boto:Path: /jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.102 DEBUG:boto:Data: 2022-09-09T22:53:07.103 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.103 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:07.104 DEBUG:boto:Port: 8000 2022-09-09T22:53:07.104 DEBUG:boto:Params: {} 2022-09-09T22:53:07.105 DEBUG:boto:Token: None 2022-09-09T22:53:07.106 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-26/?versions 2022-09-09T22:53:07.106 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ua6U4ii0uDEIvZh77BfVpMWEe24= 2022-09-09T22:53:07.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ua6U4ii0uDEIvZh77BfVpMWEe24=', 'Content-Length': '0'} 2022-09-09T22:53:07.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060f7966239874a08-00631bc3d3-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.114 DEBUG:boto:b'jkcgbj-26999falseobjnulltrue2022-09-09T22:50:32.650Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:07.115 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:53:07.116 DEBUG:boto:path=/jkcgbj-26/ 2022-09-09T22:53:07.117 DEBUG:boto:auth_path=/jkcgbj-26/ 2022-09-09T22:53:07.118 DEBUG:boto:path=/jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.118 DEBUG:boto:auth_path=/jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.119 DEBUG:boto:Method: GET 2022-09-09T22:53:07.120 DEBUG:boto:Path: /jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.121 DEBUG:boto:Data: 2022-09-09T22:53:07.122 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.123 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:07.123 DEBUG:boto:Port: 8000 2022-09-09T22:53:07.124 DEBUG:boto:Params: {} 2022-09-09T22:53:07.125 DEBUG:boto:Token: None 2022-09-09T22:53:07.126 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-26/?versions 2022-09-09T22:53:07.127 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ua6U4ii0uDEIvZh77BfVpMWEe24= 2022-09-09T22:53:07.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ua6U4ii0uDEIvZh77BfVpMWEe24=', 'Content-Length': '0'} 2022-09-09T22:53:07.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f757003c0448952f-00631bc3d3-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.135 DEBUG:boto:b'jkcgbj-26999falseobjnulltrue2022-09-09T22:50:32.650Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:07.137 DEBUG:boto:path=/jkcgbj-26/ 2022-09-09T22:53:07.138 DEBUG:boto:auth_path=/jkcgbj-26/ 2022-09-09T22:53:07.139 DEBUG:boto:path=/jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.139 DEBUG:boto:auth_path=/jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.140 DEBUG:boto:Method: GET 2022-09-09T22:53:07.141 DEBUG:boto:Path: /jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.142 DEBUG:boto:Data: 2022-09-09T22:53:07.144 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.145 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:07.146 DEBUG:boto:Port: 8000 2022-09-09T22:53:07.146 DEBUG:boto:Params: {} 2022-09-09T22:53:07.147 DEBUG:boto:Token: None 2022-09-09T22:53:07.147 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-26/?versions 2022-09-09T22:53:07.148 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ua6U4ii0uDEIvZh77BfVpMWEe24= 2022-09-09T22:53:07.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ua6U4ii0uDEIvZh77BfVpMWEe24=', 'Content-Length': '0'} 2022-09-09T22:53:07.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d605167c7822d08-00631bc3d3-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.155 DEBUG:boto:b'jkcgbj-26999falseobjnulltrue2022-09-09T22:50:32.650Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:07.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:53:07.156 DEBUG:boto:path=/jkcgbj-26/obj 2022-09-09T22:53:07.157 DEBUG:boto:auth_path=/jkcgbj-26/obj 2022-09-09T22:53:07.157 DEBUG:boto:Method: HEAD 2022-09-09T22:53:07.158 DEBUG:boto:Path: /jkcgbj-26/obj 2022-09-09T22:53:07.159 DEBUG:boto:Data: 2022-09-09T22:53:07.159 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.160 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:07.160 DEBUG:boto:Port: 8000 2022-09-09T22:53:07.161 DEBUG:boto:Params: {} 2022-09-09T22:53:07.161 DEBUG:boto:Token: None 2022-09-09T22:53:07.162 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-26/obj 2022-09-09T22:53:07.163 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+Gk3aqSLjyg8Oai96crYhuiQtzM= 2022-09-09T22:53:07.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+Gk3aqSLjyg8Oai96crYhuiQtzM=', 'Content-Length': '0'} 2022-09-09T22:53:07.166 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000006fccecd0ae6ce0b0-00631bc3d3-10e8-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.167 DEBUG:boto:path=/jkcgbj-26/obj 2022-09-09T22:53:07.167 DEBUG:boto:auth_path=/jkcgbj-26/obj 2022-09-09T22:53:07.168 DEBUG:boto:Method: HEAD 2022-09-09T22:53:07.168 DEBUG:boto:Path: /jkcgbj-26/obj 2022-09-09T22:53:07.169 DEBUG:boto:Data: 2022-09-09T22:53:07.169 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.170 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:07.170 DEBUG:boto:Port: 8000 2022-09-09T22:53:07.171 DEBUG:boto:Params: {} 2022-09-09T22:53:07.172 DEBUG:boto:Token: None 2022-09-09T22:53:07.173 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-26/obj 2022-09-09T22:53:07.174 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+Gk3aqSLjyg8Oai96crYhuiQtzM= 2022-09-09T22:53:07.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+Gk3aqSLjyg8Oai96crYhuiQtzM=', 'Content-Length': '0'} 2022-09-09T22:53:07.179 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000977226c5b0d8ee1f-00631bc3d3-10ff-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.180 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-26 zones={test-zone2, test-zone1} 2022-09-09T22:53:07.181 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-26 zones={test-zone3, test-zone1} 2022-09-09T22:53:07.182 DEBUG:boto:path=/jkcgbj-26/ 2022-09-09T22:53:07.183 DEBUG:boto:auth_path=/jkcgbj-26/ 2022-09-09T22:53:07.184 DEBUG:boto:Method: HEAD 2022-09-09T22:53:07.184 DEBUG:boto:Path: /jkcgbj-26/ 2022-09-09T22:53:07.185 DEBUG:boto:Data: 2022-09-09T22:53:07.186 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.187 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:07.188 DEBUG:boto:Port: 8001 2022-09-09T22:53:07.189 DEBUG:boto:Params: {} 2022-09-09T22:53:07.190 DEBUG:boto:Token: None 2022-09-09T22:53:07.191 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-26/ 2022-09-09T22:53:07.192 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ktBvAnrbvkLStpKRHy58/CaI9dM= 2022-09-09T22:53:07.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ktBvAnrbvkLStpKRHy58/CaI9dM=', 'Content-Length': '0'} 2022-09-09T22:53:07.199 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000087b1a946a2fc589b-00631bc3d3-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.200 DEBUG:boto:path=/jkcgbj-26/ 2022-09-09T22:53:07.201 DEBUG:boto:auth_path=/jkcgbj-26/ 2022-09-09T22:53:07.202 DEBUG:boto:Method: HEAD 2022-09-09T22:53:07.203 DEBUG:boto:Path: /jkcgbj-26/ 2022-09-09T22:53:07.204 DEBUG:boto:Data: 2022-09-09T22:53:07.205 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.206 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:07.206 DEBUG:boto:Port: 8000 2022-09-09T22:53:07.207 DEBUG:boto:Params: {} 2022-09-09T22:53:07.208 DEBUG:boto:Token: None 2022-09-09T22:53:07.208 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-26/ 2022-09-09T22:53:07.209 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ktBvAnrbvkLStpKRHy58/CaI9dM= 2022-09-09T22:53:07.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ktBvAnrbvkLStpKRHy58/CaI9dM=', 'Content-Length': '0'} 2022-09-09T22:53:07.214 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050675bbd21d394b3-00631bc3d3-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.215 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:53:07.215 DEBUG:boto:path=/jkcgbj-26/ 2022-09-09T22:53:07.216 DEBUG:boto:auth_path=/jkcgbj-26/ 2022-09-09T22:53:07.216 DEBUG:boto:path=/jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.217 DEBUG:boto:auth_path=/jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.217 DEBUG:boto:Method: GET 2022-09-09T22:53:07.218 DEBUG:boto:Path: /jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.218 DEBUG:boto:Data: 2022-09-09T22:53:07.219 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.219 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:07.220 DEBUG:boto:Port: 8001 2022-09-09T22:53:07.221 DEBUG:boto:Params: {} 2022-09-09T22:53:07.222 DEBUG:boto:Token: None 2022-09-09T22:53:07.222 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-26/?versions 2022-09-09T22:53:07.223 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ua6U4ii0uDEIvZh77BfVpMWEe24= 2022-09-09T22:53:07.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ua6U4ii0uDEIvZh77BfVpMWEe24=', 'Content-Length': '0'} 2022-09-09T22:53:07.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075436bdfe5b28f46-00631bc3d3-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.274 DEBUG:boto:b'jkcgbj-26999falseobjnulltrue2022-09-09T22:50:32.650Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:07.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:53:07.276 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:53:07.276 DEBUG:boto:path=/jkcgbj-26/ 2022-09-09T22:53:07.277 DEBUG:boto:auth_path=/jkcgbj-26/ 2022-09-09T22:53:07.277 DEBUG:boto:path=/jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.278 DEBUG:boto:auth_path=/jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.279 DEBUG:boto:Method: GET 2022-09-09T22:53:07.279 DEBUG:boto:Path: /jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.280 DEBUG:boto:Data: 2022-09-09T22:53:07.281 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.281 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:07.282 DEBUG:boto:Port: 8000 2022-09-09T22:53:07.283 DEBUG:boto:Params: {} 2022-09-09T22:53:07.283 DEBUG:boto:Token: None 2022-09-09T22:53:07.284 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-26/?versions 2022-09-09T22:53:07.285 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ua6U4ii0uDEIvZh77BfVpMWEe24= 2022-09-09T22:53:07.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ua6U4ii0uDEIvZh77BfVpMWEe24=', 'Content-Length': '0'} 2022-09-09T22:53:07.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9030f4cabb9f049-00631bc3d3-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.292 DEBUG:boto:b'jkcgbj-26999falseobjnulltrue2022-09-09T22:50:32.650Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:07.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:53:07.295 DEBUG:boto:path=/jkcgbj-26/ 2022-09-09T22:53:07.296 DEBUG:boto:auth_path=/jkcgbj-26/ 2022-09-09T22:53:07.297 DEBUG:boto:path=/jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.298 DEBUG:boto:auth_path=/jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.299 DEBUG:boto:Method: GET 2022-09-09T22:53:07.300 DEBUG:boto:Path: /jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.301 DEBUG:boto:Data: 2022-09-09T22:53:07.302 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.303 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:07.303 DEBUG:boto:Port: 8001 2022-09-09T22:53:07.304 DEBUG:boto:Params: {} 2022-09-09T22:53:07.304 DEBUG:boto:Token: None 2022-09-09T22:53:07.305 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-26/?versions 2022-09-09T22:53:07.306 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ua6U4ii0uDEIvZh77BfVpMWEe24= 2022-09-09T22:53:07.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ua6U4ii0uDEIvZh77BfVpMWEe24=', 'Content-Length': '0'} 2022-09-09T22:53:07.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b77930680ce2762-00631bc3d3-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.315 DEBUG:boto:b'jkcgbj-26999falseobjnulltrue2022-09-09T22:50:32.650Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:07.316 DEBUG:boto:path=/jkcgbj-26/ 2022-09-09T22:53:07.317 DEBUG:boto:auth_path=/jkcgbj-26/ 2022-09-09T22:53:07.318 DEBUG:boto:path=/jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.320 DEBUG:boto:auth_path=/jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.321 DEBUG:boto:Method: GET 2022-09-09T22:53:07.322 DEBUG:boto:Path: /jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.323 DEBUG:boto:Data: 2022-09-09T22:53:07.324 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.325 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:07.326 DEBUG:boto:Port: 8000 2022-09-09T22:53:07.326 DEBUG:boto:Params: {} 2022-09-09T22:53:07.327 DEBUG:boto:Token: None 2022-09-09T22:53:07.328 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-26/?versions 2022-09-09T22:53:07.328 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ua6U4ii0uDEIvZh77BfVpMWEe24= 2022-09-09T22:53:07.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ua6U4ii0uDEIvZh77BfVpMWEe24=', 'Content-Length': '0'} 2022-09-09T22:53:07.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e45aeea08a806fe-00631bc3d3-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.336 DEBUG:boto:b'jkcgbj-26999falseobjnulltrue2022-09-09T22:50:32.650Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:07.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:53:07.337 DEBUG:boto:path=/jkcgbj-26/obj 2022-09-09T22:53:07.338 DEBUG:boto:auth_path=/jkcgbj-26/obj 2022-09-09T22:53:07.339 DEBUG:boto:Method: HEAD 2022-09-09T22:53:07.339 DEBUG:boto:Path: /jkcgbj-26/obj 2022-09-09T22:53:07.340 DEBUG:boto:Data: 2022-09-09T22:53:07.341 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.341 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:07.342 DEBUG:boto:Port: 8001 2022-09-09T22:53:07.343 DEBUG:boto:Params: {} 2022-09-09T22:53:07.344 DEBUG:boto:Token: None 2022-09-09T22:53:07.346 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-26/obj 2022-09-09T22:53:07.347 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+Gk3aqSLjyg8Oai96crYhuiQtzM= 2022-09-09T22:53:07.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+Gk3aqSLjyg8Oai96crYhuiQtzM=', 'Content-Length': '0'} 2022-09-09T22:53:07.353 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000020ba2f0fbbe72dd8-00631bc3d3-1103-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.355 DEBUG:boto:path=/jkcgbj-26/obj 2022-09-09T22:53:07.356 DEBUG:boto:auth_path=/jkcgbj-26/obj 2022-09-09T22:53:07.361 DEBUG:boto:Method: HEAD 2022-09-09T22:53:07.362 DEBUG:boto:Path: /jkcgbj-26/obj 2022-09-09T22:53:07.363 DEBUG:boto:Data: 2022-09-09T22:53:07.364 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.366 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:07.367 DEBUG:boto:Port: 8000 2022-09-09T22:53:07.367 DEBUG:boto:Params: {} 2022-09-09T22:53:07.368 DEBUG:boto:Token: None 2022-09-09T22:53:07.370 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-26/obj 2022-09-09T22:53:07.371 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+Gk3aqSLjyg8Oai96crYhuiQtzM= 2022-09-09T22:53:07.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+Gk3aqSLjyg8Oai96crYhuiQtzM=', 'Content-Length': '0'} 2022-09-09T22:53:07.375 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000106cb7a161650bae-00631bc3d3-10ff-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.376 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-26 zones={test-zone3, test-zone1} 2022-09-09T22:53:07.377 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-26 zones={test-zone3, test-zone2} 2022-09-09T22:53:07.377 DEBUG:boto:path=/jkcgbj-26/ 2022-09-09T22:53:07.378 DEBUG:boto:auth_path=/jkcgbj-26/ 2022-09-09T22:53:07.378 DEBUG:boto:Method: HEAD 2022-09-09T22:53:07.379 DEBUG:boto:Path: /jkcgbj-26/ 2022-09-09T22:53:07.380 DEBUG:boto:Data: 2022-09-09T22:53:07.381 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.382 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:07.383 DEBUG:boto:Port: 8001 2022-09-09T22:53:07.383 DEBUG:boto:Params: {} 2022-09-09T22:53:07.384 DEBUG:boto:Token: None 2022-09-09T22:53:07.385 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-26/ 2022-09-09T22:53:07.386 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ktBvAnrbvkLStpKRHy58/CaI9dM= 2022-09-09T22:53:07.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ktBvAnrbvkLStpKRHy58/CaI9dM=', 'Content-Length': '0'} 2022-09-09T22:53:07.392 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c4d0b009ae70ff8-00631bc3d3-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.393 DEBUG:boto:path=/jkcgbj-26/ 2022-09-09T22:53:07.393 DEBUG:boto:auth_path=/jkcgbj-26/ 2022-09-09T22:53:07.394 DEBUG:boto:Method: HEAD 2022-09-09T22:53:07.394 DEBUG:boto:Path: /jkcgbj-26/ 2022-09-09T22:53:07.395 DEBUG:boto:Data: 2022-09-09T22:53:07.395 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.396 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:07.396 DEBUG:boto:Port: 8000 2022-09-09T22:53:07.397 DEBUG:boto:Params: {} 2022-09-09T22:53:07.397 DEBUG:boto:Token: None 2022-09-09T22:53:07.398 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-26/ 2022-09-09T22:53:07.399 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ktBvAnrbvkLStpKRHy58/CaI9dM= 2022-09-09T22:53:07.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ktBvAnrbvkLStpKRHy58/CaI9dM=', 'Content-Length': '0'} 2022-09-09T22:53:07.404 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000097784d15d83af145-00631bc3d3-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.405 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:53:07.405 DEBUG:boto:path=/jkcgbj-26/ 2022-09-09T22:53:07.406 DEBUG:boto:auth_path=/jkcgbj-26/ 2022-09-09T22:53:07.407 DEBUG:boto:path=/jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.407 DEBUG:boto:auth_path=/jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.408 DEBUG:boto:Method: GET 2022-09-09T22:53:07.408 DEBUG:boto:Path: /jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.409 DEBUG:boto:Data: 2022-09-09T22:53:07.409 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.410 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:07.410 DEBUG:boto:Port: 8001 2022-09-09T22:53:07.411 DEBUG:boto:Params: {} 2022-09-09T22:53:07.411 DEBUG:boto:Token: None 2022-09-09T22:53:07.412 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-26/?versions 2022-09-09T22:53:07.413 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ua6U4ii0uDEIvZh77BfVpMWEe24= 2022-09-09T22:53:07.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ua6U4ii0uDEIvZh77BfVpMWEe24=', 'Content-Length': '0'} 2022-09-09T22:53:07.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0fcfe1a7f2cbc13-00631bc3d3-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.462 DEBUG:boto:b'jkcgbj-26999falseobjnulltrue2022-09-09T22:50:32.650Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:07.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:53:07.464 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:53:07.465 DEBUG:boto:path=/jkcgbj-26/ 2022-09-09T22:53:07.466 DEBUG:boto:auth_path=/jkcgbj-26/ 2022-09-09T22:53:07.467 DEBUG:boto:path=/jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.467 DEBUG:boto:auth_path=/jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.468 DEBUG:boto:Method: GET 2022-09-09T22:53:07.468 DEBUG:boto:Path: /jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.469 DEBUG:boto:Data: 2022-09-09T22:53:07.469 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.470 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:07.470 DEBUG:boto:Port: 8000 2022-09-09T22:53:07.471 DEBUG:boto:Params: {} 2022-09-09T22:53:07.472 DEBUG:boto:Token: None 2022-09-09T22:53:07.472 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-26/?versions 2022-09-09T22:53:07.473 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ua6U4ii0uDEIvZh77BfVpMWEe24= 2022-09-09T22:53:07.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ua6U4ii0uDEIvZh77BfVpMWEe24=', 'Content-Length': '0'} 2022-09-09T22:53:07.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048efd7b650b7ce04-00631bc3d3-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.490 DEBUG:boto:b'jkcgbj-26999falseobjnulltrue2022-09-09T22:50:32.650Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:07.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-09T22:53:07.493 DEBUG:boto:path=/jkcgbj-26/ 2022-09-09T22:53:07.494 DEBUG:boto:auth_path=/jkcgbj-26/ 2022-09-09T22:53:07.495 DEBUG:boto:path=/jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.496 DEBUG:boto:auth_path=/jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.497 DEBUG:boto:Method: GET 2022-09-09T22:53:07.498 DEBUG:boto:Path: /jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.500 DEBUG:boto:Data: 2022-09-09T22:53:07.501 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.502 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:07.502 DEBUG:boto:Port: 8001 2022-09-09T22:53:07.503 DEBUG:boto:Params: {} 2022-09-09T22:53:07.504 DEBUG:boto:Token: None 2022-09-09T22:53:07.504 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-26/?versions 2022-09-09T22:53:07.505 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ua6U4ii0uDEIvZh77BfVpMWEe24= 2022-09-09T22:53:07.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ua6U4ii0uDEIvZh77BfVpMWEe24=', 'Content-Length': '0'} 2022-09-09T22:53:07.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5c1713386dadb02-00631bc3d3-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.512 DEBUG:boto:b'jkcgbj-26999falseobjnulltrue2022-09-09T22:50:32.650Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:07.513 DEBUG:boto:path=/jkcgbj-26/ 2022-09-09T22:53:07.514 DEBUG:boto:auth_path=/jkcgbj-26/ 2022-09-09T22:53:07.515 DEBUG:boto:path=/jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.517 DEBUG:boto:auth_path=/jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.517 DEBUG:boto:Method: GET 2022-09-09T22:53:07.518 DEBUG:boto:Path: /jkcgbj-26/?versions&max-keys=999 2022-09-09T22:53:07.520 DEBUG:boto:Data: 2022-09-09T22:53:07.521 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.521 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:07.522 DEBUG:boto:Port: 8000 2022-09-09T22:53:07.523 DEBUG:boto:Params: {} 2022-09-09T22:53:07.524 DEBUG:boto:Token: None 2022-09-09T22:53:07.525 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-26/?versions 2022-09-09T22:53:07.526 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ua6U4ii0uDEIvZh77BfVpMWEe24= 2022-09-09T22:53:07.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ua6U4ii0uDEIvZh77BfVpMWEe24=', 'Content-Length': '0'} 2022-09-09T22:53:07.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c37da1384bd39b32-00631bc3d3-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.535 DEBUG:boto:b'jkcgbj-26999falseobjnulltrue2022-09-09T22:50:32.650Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-09T22:53:07.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-09T22:53:07.537 DEBUG:boto:path=/jkcgbj-26/obj 2022-09-09T22:53:07.538 DEBUG:boto:auth_path=/jkcgbj-26/obj 2022-09-09T22:53:07.539 DEBUG:boto:Method: HEAD 2022-09-09T22:53:07.539 DEBUG:boto:Path: /jkcgbj-26/obj 2022-09-09T22:53:07.540 DEBUG:boto:Data: 2022-09-09T22:53:07.541 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.541 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:07.542 DEBUG:boto:Port: 8001 2022-09-09T22:53:07.542 DEBUG:boto:Params: {} 2022-09-09T22:53:07.543 DEBUG:boto:Token: None 2022-09-09T22:53:07.544 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-26/obj 2022-09-09T22:53:07.545 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+Gk3aqSLjyg8Oai96crYhuiQtzM= 2022-09-09T22:53:07.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+Gk3aqSLjyg8Oai96crYhuiQtzM=', 'Content-Length': '0'} 2022-09-09T22:53:07.550 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000f762ee71a2a112d8-00631bc3d3-1103-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.551 DEBUG:boto:path=/jkcgbj-26/obj 2022-09-09T22:53:07.551 DEBUG:boto:auth_path=/jkcgbj-26/obj 2022-09-09T22:53:07.552 DEBUG:boto:Method: HEAD 2022-09-09T22:53:07.553 DEBUG:boto:Path: /jkcgbj-26/obj 2022-09-09T22:53:07.553 DEBUG:boto:Data: 2022-09-09T22:53:07.554 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.554 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:07.555 DEBUG:boto:Port: 8000 2022-09-09T22:53:07.555 DEBUG:boto:Params: {} 2022-09-09T22:53:07.556 DEBUG:boto:Token: None 2022-09-09T22:53:07.557 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-26/obj 2022-09-09T22:53:07.558 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+Gk3aqSLjyg8Oai96crYhuiQtzM= 2022-09-09T22:53:07.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+Gk3aqSLjyg8Oai96crYhuiQtzM=', 'Content-Length': '0'} 2022-09-09T22:53:07.561 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000122d28fadd069dd9-00631bc3d3-10e8-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.562 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-26 zones={test-zone3, test-zone2} 2022-09-09T22:53:07.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2022-09-09T22:53:07.564 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jkcgbj-27 2022-09-09T22:53:07.565 DEBUG:boto:path=/jkcgbj-27/ 2022-09-09T22:53:07.566 DEBUG:boto:auth_path=/jkcgbj-27/ 2022-09-09T22:53:07.566 DEBUG:boto:Method: PUT 2022-09-09T22:53:07.567 DEBUG:boto:Path: /jkcgbj-27/ 2022-09-09T22:53:07.567 DEBUG:boto:Data: 2022-09-09T22:53:07.568 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.568 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:07.568 DEBUG:boto:Port: 8000 2022-09-09T22:53:07.569 DEBUG:boto:Params: {} 2022-09-09T22:53:07.570 DEBUG:boto:Token: None 2022-09-09T22:53:07.570 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-27/ 2022-09-09T22:53:07.571 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ti2z+DLHwMkPEnrfd7aQGwNbRTo= 2022-09-09T22:53:07.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ti2z+DLHwMkPEnrfd7aQGwNbRTo=', 'Content-Length': '0'} 2022-09-09T22:53:07.600 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000093487f7e2c34b709-00631bc3d3-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.601 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jkcgbj-28 2022-09-09T22:53:07.601 DEBUG:boto:path=/jkcgbj-28/ 2022-09-09T22:53:07.602 DEBUG:boto:auth_path=/jkcgbj-28/ 2022-09-09T22:53:07.602 DEBUG:boto:Method: PUT 2022-09-09T22:53:07.603 DEBUG:boto:Path: /jkcgbj-28/ 2022-09-09T22:53:07.604 DEBUG:boto:Data: 2022-09-09T22:53:07.604 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.605 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:07.605 DEBUG:boto:Port: 8000 2022-09-09T22:53:07.606 DEBUG:boto:Params: {} 2022-09-09T22:53:07.606 DEBUG:boto:Token: None 2022-09-09T22:53:07.607 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-28/ 2022-09-09T22:53:07.608 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:iyH7/0W31rp9mi9HyQRZRbGD3C8= 2022-09-09T22:53:07.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:iyH7/0W31rp9mi9HyQRZRbGD3C8=', 'Content-Length': '0'} 2022-09-09T22:53:07.662 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c1ac0b240571fb8d-00631bc3d3-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.663 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jkcgbj-29 2022-09-09T22:53:07.663 DEBUG:boto:path=/jkcgbj-29/ 2022-09-09T22:53:07.664 DEBUG:boto:auth_path=/jkcgbj-29/ 2022-09-09T22:53:07.664 DEBUG:boto:Method: PUT 2022-09-09T22:53:07.665 DEBUG:boto:Path: /jkcgbj-29/ 2022-09-09T22:53:07.665 DEBUG:boto:Data: 2022-09-09T22:53:07.666 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.667 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:07.667 DEBUG:boto:Port: 8001 2022-09-09T22:53:07.668 DEBUG:boto:Params: {} 2022-09-09T22:53:07.668 DEBUG:boto:Token: None 2022-09-09T22:53:07.669 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-29/ 2022-09-09T22:53:07.669 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wSeS7ncBaOSiBV5mc2dwiwBFge8= 2022-09-09T22:53:07.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wSeS7ncBaOSiBV5mc2dwiwBFge8=', 'Content-Length': '0'} 2022-09-09T22:53:07.718 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004632cd0f499f41ef-00631bc3d3-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.719 DEBUG:boto:path=/jkcgbj-27/ 2022-09-09T22:53:07.720 DEBUG:boto:auth_path=/jkcgbj-27/ 2022-09-09T22:53:07.720 DEBUG:boto:path=/jkcgbj-27/?versioning 2022-09-09T22:53:07.721 DEBUG:boto:auth_path=/jkcgbj-27/?versioning 2022-09-09T22:53:07.721 DEBUG:boto:Method: PUT 2022-09-09T22:53:07.722 DEBUG:boto:Path: /jkcgbj-27/?versioning 2022-09-09T22:53:07.723 DEBUG:boto:Data: Enabled Disabled 2022-09-09T22:53:07.724 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.725 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:07.726 DEBUG:boto:Port: 8000 2022-09-09T22:53:07.726 DEBUG:boto:Params: {} 2022-09-09T22:53:07.727 DEBUG:boto:Token: None 2022-09-09T22:53:07.728 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-27/?versioning 2022-09-09T22:53:07.729 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:mZtFhbL8MRjtF+t5UmiUqu91iBk= 2022-09-09T22:53:07.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:mZtFhbL8MRjtF+t5UmiUqu91iBk=', 'Content-Length': '228'} 2022-09-09T22:53:07.741 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a054d1433f9c7803-00631bc3d3-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.743 DEBUG:boto:path=/jkcgbj-27/ 2022-09-09T22:53:07.744 DEBUG:boto:auth_path=/jkcgbj-27/ 2022-09-09T22:53:07.745 DEBUG:boto:path=/jkcgbj-27/?versioning 2022-09-09T22:53:07.746 DEBUG:boto:auth_path=/jkcgbj-27/?versioning 2022-09-09T22:53:07.746 DEBUG:boto:Method: GET 2022-09-09T22:53:07.747 DEBUG:boto:Path: /jkcgbj-27/?versioning 2022-09-09T22:53:07.748 DEBUG:boto:Data: 2022-09-09T22:53:07.749 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.750 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:07.751 DEBUG:boto:Port: 8000 2022-09-09T22:53:07.752 DEBUG:boto:Params: {} 2022-09-09T22:53:07.753 DEBUG:boto:Token: None 2022-09-09T22:53:07.754 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-27/?versioning 2022-09-09T22:53:07.755 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:kch7gXEe8lQwseBdtfoxetHXe4k= 2022-09-09T22:53:07.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:kch7gXEe8lQwseBdtfoxetHXe4k=', 'Content-Length': '0'} 2022-09-09T22:53:07.758 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001276d1722386358a-00631bc3d3-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.798 DEBUG:boto:EnabledDisabled 2022-09-09T22:53:07.800 DEBUG:boto:path=/jkcgbj-28/ 2022-09-09T22:53:07.800 DEBUG:boto:auth_path=/jkcgbj-28/ 2022-09-09T22:53:07.801 DEBUG:boto:path=/jkcgbj-28/?versioning 2022-09-09T22:53:07.801 DEBUG:boto:auth_path=/jkcgbj-28/?versioning 2022-09-09T22:53:07.802 DEBUG:boto:Method: PUT 2022-09-09T22:53:07.803 DEBUG:boto:Path: /jkcgbj-28/?versioning 2022-09-09T22:53:07.805 DEBUG:boto:Data: Enabled Disabled 2022-09-09T22:53:07.805 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.806 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:07.807 DEBUG:boto:Port: 8000 2022-09-09T22:53:07.807 DEBUG:boto:Params: {} 2022-09-09T22:53:07.808 DEBUG:boto:Token: None 2022-09-09T22:53:07.809 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-28/?versioning 2022-09-09T22:53:07.810 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jErw64PUcQUWAnSBr9hcDh6ll84= 2022-09-09T22:53:07.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jErw64PUcQUWAnSBr9hcDh6ll84=', 'Content-Length': '228'} 2022-09-09T22:53:07.831 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009b88789acb2c98e2-00631bc3d3-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.832 DEBUG:boto:path=/jkcgbj-28/ 2022-09-09T22:53:07.832 DEBUG:boto:auth_path=/jkcgbj-28/ 2022-09-09T22:53:07.833 DEBUG:boto:path=/jkcgbj-28/?versioning 2022-09-09T22:53:07.833 DEBUG:boto:auth_path=/jkcgbj-28/?versioning 2022-09-09T22:53:07.834 DEBUG:boto:Method: GET 2022-09-09T22:53:07.834 DEBUG:boto:Path: /jkcgbj-28/?versioning 2022-09-09T22:53:07.835 DEBUG:boto:Data: 2022-09-09T22:53:07.835 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.836 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:07.837 DEBUG:boto:Port: 8000 2022-09-09T22:53:07.838 DEBUG:boto:Params: {} 2022-09-09T22:53:07.839 DEBUG:boto:Token: None 2022-09-09T22:53:07.840 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-28/?versioning 2022-09-09T22:53:07.841 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ABd3do3iZoKzXxOHWIdHZNmpEj8= 2022-09-09T22:53:07.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ABd3do3iZoKzXxOHWIdHZNmpEj8=', 'Content-Length': '0'} 2022-09-09T22:53:07.843 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006e344e229f1bf6c1-00631bc3d3-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.844 DEBUG:boto:EnabledDisabled 2022-09-09T22:53:07.845 DEBUG:boto:path=/jkcgbj-29/ 2022-09-09T22:53:07.845 DEBUG:boto:auth_path=/jkcgbj-29/ 2022-09-09T22:53:07.846 DEBUG:boto:path=/jkcgbj-29/?versioning 2022-09-09T22:53:07.847 DEBUG:boto:auth_path=/jkcgbj-29/?versioning 2022-09-09T22:53:07.847 DEBUG:boto:Method: PUT 2022-09-09T22:53:07.848 DEBUG:boto:Path: /jkcgbj-29/?versioning 2022-09-09T22:53:07.849 DEBUG:boto:Data: Enabled Disabled 2022-09-09T22:53:07.850 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.850 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:07.851 DEBUG:boto:Port: 8001 2022-09-09T22:53:07.852 DEBUG:boto:Params: {} 2022-09-09T22:53:07.853 DEBUG:boto:Token: None 2022-09-09T22:53:07.854 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-29/?versioning 2022-09-09T22:53:07.855 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:eKLkNTstyLVxLarC4DheUprub0w= 2022-09-09T22:53:07.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:eKLkNTstyLVxLarC4DheUprub0w=', 'Content-Length': '228'} 2022-09-09T22:53:07.875 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b3254d6edf6af408-00631bc3d3-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.876 DEBUG:boto:path=/jkcgbj-29/ 2022-09-09T22:53:07.876 DEBUG:boto:auth_path=/jkcgbj-29/ 2022-09-09T22:53:07.877 DEBUG:boto:path=/jkcgbj-29/?versioning 2022-09-09T22:53:07.878 DEBUG:boto:auth_path=/jkcgbj-29/?versioning 2022-09-09T22:53:07.878 DEBUG:boto:Method: GET 2022-09-09T22:53:07.879 DEBUG:boto:Path: /jkcgbj-29/?versioning 2022-09-09T22:53:07.880 DEBUG:boto:Data: 2022-09-09T22:53:07.880 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.881 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:07.881 DEBUG:boto:Port: 8001 2022-09-09T22:53:07.882 DEBUG:boto:Params: {} 2022-09-09T22:53:07.882 DEBUG:boto:Token: None 2022-09-09T22:53:07.883 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-29/?versioning 2022-09-09T22:53:07.884 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KPl+sSE7yUzegSsnixaigPxM5Z4= 2022-09-09T22:53:07.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KPl+sSE7yUzegSsnixaigPxM5Z4=', 'Content-Length': '0'} 2022-09-09T22:53:07.887 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006aa3ad8a151f7d84-00631bc3d3-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.888 DEBUG:boto:EnabledDisabled 2022-09-09T22:53:07.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2022-09-09T22:53:07.889 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jkcgbj-30 2022-09-09T22:53:07.890 DEBUG:boto:path=/jkcgbj-30/ 2022-09-09T22:53:07.890 DEBUG:boto:auth_path=/jkcgbj-30/ 2022-09-09T22:53:07.891 DEBUG:boto:Method: PUT 2022-09-09T22:53:07.891 DEBUG:boto:Path: /jkcgbj-30/ 2022-09-09T22:53:07.892 DEBUG:boto:Data: 2022-09-09T22:53:07.892 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.893 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:07.893 DEBUG:boto:Port: 8000 2022-09-09T22:53:07.894 DEBUG:boto:Params: {} 2022-09-09T22:53:07.894 DEBUG:boto:Token: None 2022-09-09T22:53:07.895 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-30/ 2022-09-09T22:53:07.895 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:E4CdnKWnSDQjzJxJBqvBkOQUIi4= 2022-09-09T22:53:07.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:E4CdnKWnSDQjzJxJBqvBkOQUIi4=', 'Content-Length': '0'} 2022-09-09T22:53:07.918 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d1b078676aa32bfe-00631bc3d3-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:07.920 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jkcgbj-31 2022-09-09T22:53:07.921 DEBUG:boto:path=/jkcgbj-31/ 2022-09-09T22:53:07.922 DEBUG:boto:auth_path=/jkcgbj-31/ 2022-09-09T22:53:07.923 DEBUG:boto:Method: PUT 2022-09-09T22:53:07.923 DEBUG:boto:Path: /jkcgbj-31/ 2022-09-09T22:53:07.924 DEBUG:boto:Data: 2022-09-09T22:53:07.925 DEBUG:boto:Headers: {} 2022-09-09T22:53:07.926 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:07.927 DEBUG:boto:Port: 8000 2022-09-09T22:53:07.928 DEBUG:boto:Params: {} 2022-09-09T22:53:07.929 DEBUG:boto:Token: None 2022-09-09T22:53:07.931 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:53:07 GMT /jkcgbj-31/ 2022-09-09T22:53:07.932 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:gTfvusbeKaYanyMAq8HwNqhioFY= 2022-09-09T22:53:07.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:07 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:gTfvusbeKaYanyMAq8HwNqhioFY=', 'Content-Length': '0'} 2022-09-09T22:53:08.001 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004269529617b5ec13-00631bc3d3-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:08.002 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jkcgbj-32 2022-09-09T22:53:08.002 DEBUG:boto:path=/jkcgbj-32/ 2022-09-09T22:53:08.003 DEBUG:boto:auth_path=/jkcgbj-32/ 2022-09-09T22:53:08.003 DEBUG:boto:Method: PUT 2022-09-09T22:53:08.004 DEBUG:boto:Path: /jkcgbj-32/ 2022-09-09T22:53:08.004 DEBUG:boto:Data: 2022-09-09T22:53:08.005 DEBUG:boto:Headers: {} 2022-09-09T22:53:08.006 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:08.006 DEBUG:boto:Port: 8001 2022-09-09T22:53:08.007 DEBUG:boto:Params: {} 2022-09-09T22:53:08.007 DEBUG:boto:Token: None 2022-09-09T22:53:08.008 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:53:08 GMT /jkcgbj-32/ 2022-09-09T22:53:08.009 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pgcfeSrMaWoCJ2JSgc/G4TyOynM= 2022-09-09T22:53:08.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:08 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pgcfeSrMaWoCJ2JSgc/G4TyOynM=', 'Content-Length': '0'} 2022-09-09T22:53:08.058 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000053454e52891e375b-00631bc3d4-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:08.059 DEBUG:boto:path=/jkcgbj-30/ 2022-09-09T22:53:08.060 DEBUG:boto:auth_path=/jkcgbj-30/ 2022-09-09T22:53:08.060 DEBUG:boto:path=/jkcgbj-30/?acl 2022-09-09T22:53:08.061 DEBUG:boto:auth_path=/jkcgbj-30/?acl 2022-09-09T22:53:08.061 DEBUG:boto:Method: GET 2022-09-09T22:53:08.062 DEBUG:boto:Path: /jkcgbj-30/?acl 2022-09-09T22:53:08.063 DEBUG:boto:Data: 2022-09-09T22:53:08.063 DEBUG:boto:Headers: {} 2022-09-09T22:53:08.064 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:08.065 DEBUG:boto:Port: 8000 2022-09-09T22:53:08.066 DEBUG:boto:Params: {} 2022-09-09T22:53:08.067 DEBUG:boto:Token: None 2022-09-09T22:53:08.067 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:08 GMT /jkcgbj-30/?acl 2022-09-09T22:53:08.068 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:IHRMu8PBWC+Vh8Mp6qMMnKZVG1E= 2022-09-09T22:53:08.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:08 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:IHRMu8PBWC+Vh8Mp6qMMnKZVG1E=', 'Content-Length': '0'} 2022-09-09T22:53:08.071 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000419934b95c45325b-00631bc3d4-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Fri, 09 Sep 2022 22:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:08.072 DEBUG:boto:path=/jkcgbj-30/ 2022-09-09T22:53:08.072 DEBUG:boto:auth_path=/jkcgbj-30/ 2022-09-09T22:53:08.073 DEBUG:boto:path=/jkcgbj-30/?acl 2022-09-09T22:53:08.073 DEBUG:boto:auth_path=/jkcgbj-30/?acl 2022-09-09T22:53:08.074 DEBUG:boto:Method: PUT 2022-09-09T22:53:08.075 DEBUG:boto:Path: /jkcgbj-30/?acl 2022-09-09T22:53:08.076 DEBUG:boto:Data: 2022-09-09T22:53:08.077 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-09T22:53:08.078 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:08.079 DEBUG:boto:Port: 8000 2022-09-09T22:53:08.080 DEBUG:boto:Params: {} 2022-09-09T22:53:08.081 DEBUG:boto:Token: None 2022-09-09T22:53:08.082 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:53:08 GMT x-amz-acl:public-read /jkcgbj-30/?acl 2022-09-09T22:53:08.082 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Ipnc75K9T1LLUuYB7FrFGS8RcYs= 2022-09-09T22:53:08.083 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:08 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Ipnc75K9T1LLUuYB7FrFGS8RcYs=', 'Content-Length': '0'} 2022-09-09T22:53:08.095 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007e6141b14cbc9baf-00631bc3d4-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:08.096 DEBUG:boto:path=/jkcgbj-30/ 2022-09-09T22:53:08.097 DEBUG:boto:auth_path=/jkcgbj-30/ 2022-09-09T22:53:08.098 DEBUG:boto:path=/jkcgbj-30/?acl 2022-09-09T22:53:08.099 DEBUG:boto:auth_path=/jkcgbj-30/?acl 2022-09-09T22:53:08.100 DEBUG:boto:Method: GET 2022-09-09T22:53:08.100 DEBUG:boto:Path: /jkcgbj-30/?acl 2022-09-09T22:53:08.101 DEBUG:boto:Data: 2022-09-09T22:53:08.101 DEBUG:boto:Headers: {} 2022-09-09T22:53:08.102 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:08.103 DEBUG:boto:Port: 8000 2022-09-09T22:53:08.103 DEBUG:boto:Params: {} 2022-09-09T22:53:08.104 DEBUG:boto:Token: None 2022-09-09T22:53:08.105 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:08 GMT /jkcgbj-30/?acl 2022-09-09T22:53:08.106 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:IHRMu8PBWC+Vh8Mp6qMMnKZVG1E= 2022-09-09T22:53:08.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:08 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:IHRMu8PBWC+Vh8Mp6qMMnKZVG1E=', 'Content-Length': '0'} 2022-09-09T22:53:08.108 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000006e2bd6a847888ff-00631bc3d4-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Fri, 09 Sep 2022 22:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:08.109 DEBUG:boto:path=/jkcgbj-31/ 2022-09-09T22:53:08.109 DEBUG:boto:auth_path=/jkcgbj-31/ 2022-09-09T22:53:08.110 DEBUG:boto:path=/jkcgbj-31/?acl 2022-09-09T22:53:08.111 DEBUG:boto:auth_path=/jkcgbj-31/?acl 2022-09-09T22:53:08.111 DEBUG:boto:Method: GET 2022-09-09T22:53:08.112 DEBUG:boto:Path: /jkcgbj-31/?acl 2022-09-09T22:53:08.112 DEBUG:boto:Data: 2022-09-09T22:53:08.112 DEBUG:boto:Headers: {} 2022-09-09T22:53:08.113 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:08.113 DEBUG:boto:Port: 8000 2022-09-09T22:53:08.114 DEBUG:boto:Params: {} 2022-09-09T22:53:08.115 DEBUG:boto:Token: None 2022-09-09T22:53:08.115 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:08 GMT /jkcgbj-31/?acl 2022-09-09T22:53:08.116 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Duc/jFF0zVk4EECnKShDJ3uu5/M= 2022-09-09T22:53:08.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:08 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Duc/jFF0zVk4EECnKShDJ3uu5/M=', 'Content-Length': '0'} 2022-09-09T22:53:08.119 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d806b6c3e17a8ebb-00631bc3d4-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Fri, 09 Sep 2022 22:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:08.120 DEBUG:boto:path=/jkcgbj-31/ 2022-09-09T22:53:08.120 DEBUG:boto:auth_path=/jkcgbj-31/ 2022-09-09T22:53:08.121 DEBUG:boto:path=/jkcgbj-31/?acl 2022-09-09T22:53:08.121 DEBUG:boto:auth_path=/jkcgbj-31/?acl 2022-09-09T22:53:08.122 DEBUG:boto:Method: PUT 2022-09-09T22:53:08.122 DEBUG:boto:Path: /jkcgbj-31/?acl 2022-09-09T22:53:08.123 DEBUG:boto:Data: 2022-09-09T22:53:08.123 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-09T22:53:08.124 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:08.124 DEBUG:boto:Port: 8000 2022-09-09T22:53:08.125 DEBUG:boto:Params: {} 2022-09-09T22:53:08.125 DEBUG:boto:Token: None 2022-09-09T22:53:08.126 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:53:08 GMT x-amz-acl:public-read /jkcgbj-31/?acl 2022-09-09T22:53:08.127 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wwdWakeXwaeyICBtqhMvV21CtTU= 2022-09-09T22:53:08.127 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:08 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wwdWakeXwaeyICBtqhMvV21CtTU=', 'Content-Length': '0'} 2022-09-09T22:53:08.148 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008528e1f4697380a8-00631bc3d4-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:08.150 DEBUG:boto:path=/jkcgbj-31/ 2022-09-09T22:53:08.151 DEBUG:boto:auth_path=/jkcgbj-31/ 2022-09-09T22:53:08.151 DEBUG:boto:path=/jkcgbj-31/?acl 2022-09-09T22:53:08.152 DEBUG:boto:auth_path=/jkcgbj-31/?acl 2022-09-09T22:53:08.153 DEBUG:boto:Method: GET 2022-09-09T22:53:08.154 DEBUG:boto:Path: /jkcgbj-31/?acl 2022-09-09T22:53:08.154 DEBUG:boto:Data: 2022-09-09T22:53:08.155 DEBUG:boto:Headers: {} 2022-09-09T22:53:08.155 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:08.156 DEBUG:boto:Port: 8000 2022-09-09T22:53:08.156 DEBUG:boto:Params: {} 2022-09-09T22:53:08.157 DEBUG:boto:Token: None 2022-09-09T22:53:08.157 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:08 GMT /jkcgbj-31/?acl 2022-09-09T22:53:08.158 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Duc/jFF0zVk4EECnKShDJ3uu5/M= 2022-09-09T22:53:08.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:08 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Duc/jFF0zVk4EECnKShDJ3uu5/M=', 'Content-Length': '0'} 2022-09-09T22:53:08.160 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d0682e4a0dd0a5a6-00631bc3d4-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Fri, 09 Sep 2022 22:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:08.206 DEBUG:boto:path=/jkcgbj-32/ 2022-09-09T22:53:08.207 DEBUG:boto:auth_path=/jkcgbj-32/ 2022-09-09T22:53:08.207 DEBUG:boto:path=/jkcgbj-32/?acl 2022-09-09T22:53:08.208 DEBUG:boto:auth_path=/jkcgbj-32/?acl 2022-09-09T22:53:08.208 DEBUG:boto:Method: GET 2022-09-09T22:53:08.209 DEBUG:boto:Path: /jkcgbj-32/?acl 2022-09-09T22:53:08.209 DEBUG:boto:Data: 2022-09-09T22:53:08.210 DEBUG:boto:Headers: {} 2022-09-09T22:53:08.211 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:08.212 DEBUG:boto:Port: 8001 2022-09-09T22:53:08.213 DEBUG:boto:Params: {} 2022-09-09T22:53:08.214 DEBUG:boto:Token: None 2022-09-09T22:53:08.215 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:08 GMT /jkcgbj-32/?acl 2022-09-09T22:53:08.216 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6tkZxCU6EHb2mMow6t8hWt17jbM= 2022-09-09T22:53:08.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:08 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6tkZxCU6EHb2mMow6t8hWt17jbM=', 'Content-Length': '0'} 2022-09-09T22:53:08.219 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002c851861f9f3e34a-00631bc3d4-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Fri, 09 Sep 2022 22:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:08.220 DEBUG:boto:path=/jkcgbj-32/ 2022-09-09T22:53:08.220 DEBUG:boto:auth_path=/jkcgbj-32/ 2022-09-09T22:53:08.221 DEBUG:boto:path=/jkcgbj-32/?acl 2022-09-09T22:53:08.222 DEBUG:boto:auth_path=/jkcgbj-32/?acl 2022-09-09T22:53:08.223 DEBUG:boto:Method: PUT 2022-09-09T22:53:08.223 DEBUG:boto:Path: /jkcgbj-32/?acl 2022-09-09T22:53:08.224 DEBUG:boto:Data: 2022-09-09T22:53:08.225 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-09T22:53:08.226 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:08.226 DEBUG:boto:Port: 8001 2022-09-09T22:53:08.227 DEBUG:boto:Params: {} 2022-09-09T22:53:08.228 DEBUG:boto:Token: None 2022-09-09T22:53:08.228 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:53:08 GMT x-amz-acl:public-read /jkcgbj-32/?acl 2022-09-09T22:53:08.229 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:AJpVIlLAgSE0YcHrt/qUsbBvLJo= 2022-09-09T22:53:08.230 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:08 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:AJpVIlLAgSE0YcHrt/qUsbBvLJo=', 'Content-Length': '0'} 2022-09-09T22:53:08.250 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002f73e1d0840dabbd-00631bc3d4-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:08.257 DEBUG:boto:path=/jkcgbj-32/ 2022-09-09T22:53:08.258 DEBUG:boto:auth_path=/jkcgbj-32/ 2022-09-09T22:53:08.259 DEBUG:boto:path=/jkcgbj-32/?acl 2022-09-09T22:53:08.260 DEBUG:boto:auth_path=/jkcgbj-32/?acl 2022-09-09T22:53:08.261 DEBUG:boto:Method: GET 2022-09-09T22:53:08.262 DEBUG:boto:Path: /jkcgbj-32/?acl 2022-09-09T22:53:08.263 DEBUG:boto:Data: 2022-09-09T22:53:08.263 DEBUG:boto:Headers: {} 2022-09-09T22:53:08.264 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:08.265 DEBUG:boto:Port: 8001 2022-09-09T22:53:08.266 DEBUG:boto:Params: {} 2022-09-09T22:53:08.266 DEBUG:boto:Token: None 2022-09-09T22:53:08.267 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:08 GMT /jkcgbj-32/?acl 2022-09-09T22:53:08.268 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6tkZxCU6EHb2mMow6t8hWt17jbM= 2022-09-09T22:53:08.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:08 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6tkZxCU6EHb2mMow6t8hWt17jbM=', 'Content-Length': '0'} 2022-09-09T22:53:08.270 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e6af383cd870f5be-00631bc3d4-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Fri, 09 Sep 2022 22:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:08.318 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2022-09-09T22:53:08.319 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jkcgbj-33 2022-09-09T22:53:08.320 DEBUG:boto:path=/jkcgbj-33/ 2022-09-09T22:53:08.320 DEBUG:boto:auth_path=/jkcgbj-33/ 2022-09-09T22:53:08.321 DEBUG:boto:Method: PUT 2022-09-09T22:53:08.322 DEBUG:boto:Path: /jkcgbj-33/ 2022-09-09T22:53:08.322 DEBUG:boto:Data: 2022-09-09T22:53:08.323 DEBUG:boto:Headers: {} 2022-09-09T22:53:08.323 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:08.324 DEBUG:boto:Port: 8000 2022-09-09T22:53:08.324 DEBUG:boto:Params: {} 2022-09-09T22:53:08.325 DEBUG:boto:Token: None 2022-09-09T22:53:08.326 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:53:08 GMT /jkcgbj-33/ 2022-09-09T22:53:08.326 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:VqTOmoeqzR4fK03XVM7k1q6vhNk= 2022-09-09T22:53:08.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:08 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:VqTOmoeqzR4fK03XVM7k1q6vhNk=', 'Content-Length': '0'} 2022-09-09T22:53:08.375 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000badea3c7f22d7cae-00631bc3d4-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:08.375 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jkcgbj-34 2022-09-09T22:53:08.376 DEBUG:boto:path=/jkcgbj-34/ 2022-09-09T22:53:08.376 DEBUG:boto:auth_path=/jkcgbj-34/ 2022-09-09T22:53:08.377 DEBUG:boto:Method: PUT 2022-09-09T22:53:08.378 DEBUG:boto:Path: /jkcgbj-34/ 2022-09-09T22:53:08.378 DEBUG:boto:Data: 2022-09-09T22:53:08.379 DEBUG:boto:Headers: {} 2022-09-09T22:53:08.379 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:08.380 DEBUG:boto:Port: 8000 2022-09-09T22:53:08.380 DEBUG:boto:Params: {} 2022-09-09T22:53:08.381 DEBUG:boto:Token: None 2022-09-09T22:53:08.381 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:53:08 GMT /jkcgbj-34/ 2022-09-09T22:53:08.382 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:HnaD7ZJk1TP00nfppfP6alxjQcU= 2022-09-09T22:53:08.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:08 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:HnaD7ZJk1TP00nfppfP6alxjQcU=', 'Content-Length': '0'} 2022-09-09T22:53:08.461 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000e8deb585474a1a9-00631bc3d4-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:08.462 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jkcgbj-35 2022-09-09T22:53:08.463 DEBUG:boto:path=/jkcgbj-35/ 2022-09-09T22:53:08.463 DEBUG:boto:auth_path=/jkcgbj-35/ 2022-09-09T22:53:08.464 DEBUG:boto:Method: PUT 2022-09-09T22:53:08.464 DEBUG:boto:Path: /jkcgbj-35/ 2022-09-09T22:53:08.465 DEBUG:boto:Data: 2022-09-09T22:53:08.465 DEBUG:boto:Headers: {} 2022-09-09T22:53:08.466 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:08.466 DEBUG:boto:Port: 8001 2022-09-09T22:53:08.467 DEBUG:boto:Params: {} 2022-09-09T22:53:08.468 DEBUG:boto:Token: None 2022-09-09T22:53:08.468 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:53:08 GMT /jkcgbj-35/ 2022-09-09T22:53:08.469 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:gxa2QtzioQgYZziw5LpZ2BGXg7I= 2022-09-09T22:53:08.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:08 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:gxa2QtzioQgYZziw5LpZ2BGXg7I=', 'Content-Length': '0'} 2022-09-09T22:53:08.516 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008c3f6c147ea616d3-00631bc3d4-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:08.517 DEBUG:boto:path=/jkcgbj-33/ 2022-09-09T22:53:08.517 DEBUG:boto:auth_path=/jkcgbj-33/ 2022-09-09T22:53:08.518 DEBUG:boto:path=/jkcgbj-33/?cors 2022-09-09T22:53:08.518 DEBUG:boto:auth_path=/jkcgbj-33/?cors 2022-09-09T22:53:08.519 DEBUG:boto:Method: PUT 2022-09-09T22:53:08.519 DEBUG:boto:Path: /jkcgbj-33/?cors 2022-09-09T22:53:08.520 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2022-09-09T22:53:08.520 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2022-09-09T22:53:08.521 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:08.521 DEBUG:boto:Port: 8000 2022-09-09T22:53:08.522 DEBUG:boto:Params: {} 2022-09-09T22:53:08.522 DEBUG:boto:Token: None 2022-09-09T22:53:08.523 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 09 Sep 2022 22:53:08 GMT /jkcgbj-33/?cors 2022-09-09T22:53:08.523 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1Qm+FPwFg5Y/biVLil5u3sKyeFE= 2022-09-09T22:53:08.524 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:08 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1Qm+FPwFg5Y/biVLil5u3sKyeFE=', 'Content-Length': '218'} 2022-09-09T22:53:08.534 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c1d03d9895ba2edf-00631bc3d4-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:08.535 DEBUG:boto:path=/jkcgbj-33/ 2022-09-09T22:53:08.536 DEBUG:boto:auth_path=/jkcgbj-33/ 2022-09-09T22:53:08.536 DEBUG:boto:path=/jkcgbj-33/?cors 2022-09-09T22:53:08.537 DEBUG:boto:auth_path=/jkcgbj-33/?cors 2022-09-09T22:53:08.537 DEBUG:boto:Method: GET 2022-09-09T22:53:08.538 DEBUG:boto:Path: /jkcgbj-33/?cors 2022-09-09T22:53:08.538 DEBUG:boto:Data: 2022-09-09T22:53:08.539 DEBUG:boto:Headers: {} 2022-09-09T22:53:08.539 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:08.540 DEBUG:boto:Port: 8000 2022-09-09T22:53:08.540 DEBUG:boto:Params: {} 2022-09-09T22:53:08.541 DEBUG:boto:Token: None 2022-09-09T22:53:08.541 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:08 GMT /jkcgbj-33/?cors 2022-09-09T22:53:08.542 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:vDu6Pf2Qzz+ZgLcrFyOh+2PhHZg= 2022-09-09T22:53:08.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:08 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:vDu6Pf2Qzz+ZgLcrFyOh+2PhHZg=', 'Content-Length': '0'} 2022-09-09T22:53:08.545 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000deaa76af9805d3c3-00631bc3d4-10ff-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Fri, 09 Sep 2022 22:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:08.545 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2022-09-09T22:53:08.546 DEBUG:boto:path=/jkcgbj-34/ 2022-09-09T22:53:08.547 DEBUG:boto:auth_path=/jkcgbj-34/ 2022-09-09T22:53:08.547 DEBUG:boto:path=/jkcgbj-34/?cors 2022-09-09T22:53:08.548 DEBUG:boto:auth_path=/jkcgbj-34/?cors 2022-09-09T22:53:08.548 DEBUG:boto:Method: PUT 2022-09-09T22:53:08.549 DEBUG:boto:Path: /jkcgbj-34/?cors 2022-09-09T22:53:08.549 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2022-09-09T22:53:08.550 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2022-09-09T22:53:08.550 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:08.551 DEBUG:boto:Port: 8000 2022-09-09T22:53:08.551 DEBUG:boto:Params: {} 2022-09-09T22:53:08.552 DEBUG:boto:Token: None 2022-09-09T22:53:08.552 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 09 Sep 2022 22:53:08 GMT /jkcgbj-34/?cors 2022-09-09T22:53:08.553 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dHHZmy+bGvTSKC6SrIcNAgGCDYk= 2022-09-09T22:53:08.553 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:08 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dHHZmy+bGvTSKC6SrIcNAgGCDYk=', 'Content-Length': '218'} 2022-09-09T22:53:09.576 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d8dd706e008e38bb-00631bc3d4-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:09.577 DEBUG:boto:path=/jkcgbj-34/ 2022-09-09T22:53:09.577 DEBUG:boto:auth_path=/jkcgbj-34/ 2022-09-09T22:53:09.578 DEBUG:boto:path=/jkcgbj-34/?cors 2022-09-09T22:53:09.579 DEBUG:boto:auth_path=/jkcgbj-34/?cors 2022-09-09T22:53:09.579 DEBUG:boto:Method: GET 2022-09-09T22:53:09.580 DEBUG:boto:Path: /jkcgbj-34/?cors 2022-09-09T22:53:09.580 DEBUG:boto:Data: 2022-09-09T22:53:09.581 DEBUG:boto:Headers: {} 2022-09-09T22:53:09.581 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:09.582 DEBUG:boto:Port: 8000 2022-09-09T22:53:09.582 DEBUG:boto:Params: {} 2022-09-09T22:53:09.583 DEBUG:boto:Token: None 2022-09-09T22:53:09.583 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:09 GMT /jkcgbj-34/?cors 2022-09-09T22:53:09.584 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:kSoxu6fSluWv510VQcx+nUtaS2w= 2022-09-09T22:53:09.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:kSoxu6fSluWv510VQcx+nUtaS2w=', 'Content-Length': '0'} 2022-09-09T22:53:09.587 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fba25518ce7c3879-00631bc3d5-10e8-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Fri, 09 Sep 2022 22:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:09.588 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2022-09-09T22:53:09.589 DEBUG:boto:path=/jkcgbj-35/ 2022-09-09T22:53:09.590 DEBUG:boto:auth_path=/jkcgbj-35/ 2022-09-09T22:53:09.591 DEBUG:boto:path=/jkcgbj-35/?cors 2022-09-09T22:53:09.592 DEBUG:boto:auth_path=/jkcgbj-35/?cors 2022-09-09T22:53:09.593 DEBUG:boto:Method: PUT 2022-09-09T22:53:09.594 DEBUG:boto:Path: /jkcgbj-35/?cors 2022-09-09T22:53:09.595 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2022-09-09T22:53:09.596 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2022-09-09T22:53:09.597 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:09.597 DEBUG:boto:Port: 8001 2022-09-09T22:53:09.597 DEBUG:boto:Params: {} 2022-09-09T22:53:09.598 DEBUG:boto:Token: None 2022-09-09T22:53:09.599 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 09 Sep 2022 22:53:09 GMT /jkcgbj-35/?cors 2022-09-09T22:53:09.599 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1b2EF6hkBOZZEFgP/W9vrFJc8TA= 2022-09-09T22:53:09.600 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1b2EF6hkBOZZEFgP/W9vrFJc8TA=', 'Content-Length': '218'} 2022-09-09T22:53:10.620 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ec2935b5da696cfc-00631bc3d5-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:10.623 DEBUG:boto:path=/jkcgbj-35/ 2022-09-09T22:53:10.624 DEBUG:boto:auth_path=/jkcgbj-35/ 2022-09-09T22:53:10.624 DEBUG:boto:path=/jkcgbj-35/?cors 2022-09-09T22:53:10.625 DEBUG:boto:auth_path=/jkcgbj-35/?cors 2022-09-09T22:53:10.625 DEBUG:boto:Method: GET 2022-09-09T22:53:10.626 DEBUG:boto:Path: /jkcgbj-35/?cors 2022-09-09T22:53:10.626 DEBUG:boto:Data: 2022-09-09T22:53:10.627 DEBUG:boto:Headers: {} 2022-09-09T22:53:10.628 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:10.629 DEBUG:boto:Port: 8001 2022-09-09T22:53:10.630 DEBUG:boto:Params: {} 2022-09-09T22:53:10.631 DEBUG:boto:Token: None 2022-09-09T22:53:10.632 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:53:10 GMT /jkcgbj-35/?cors 2022-09-09T22:53:10.634 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:o5iUuXr/fpeSuy6kaXtFED5XZUM= 2022-09-09T22:53:10.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:o5iUuXr/fpeSuy6kaXtFED5XZUM=', 'Content-Length': '0'} 2022-09-09T22:53:10.637 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000028da0745e0bf996f-00631bc3d6-1103-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Fri, 09 Sep 2022 22:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:10.637 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2022-09-09T22:53:10.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2022-09-09T22:53:10.639 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jkcgbj-36 2022-09-09T22:53:10.639 DEBUG:boto:path=/jkcgbj-36/ 2022-09-09T22:53:10.640 DEBUG:boto:auth_path=/jkcgbj-36/ 2022-09-09T22:53:10.640 DEBUG:boto:Method: PUT 2022-09-09T22:53:10.641 DEBUG:boto:Path: /jkcgbj-36/ 2022-09-09T22:53:10.641 DEBUG:boto:Data: 2022-09-09T22:53:10.642 DEBUG:boto:Headers: {} 2022-09-09T22:53:10.642 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:53:10.643 DEBUG:boto:Port: 8000 2022-09-09T22:53:10.645 DEBUG:boto:Params: {} 2022-09-09T22:53:10.646 DEBUG:boto:Token: None 2022-09-09T22:53:10.646 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:53:10 GMT /jkcgbj-36/ 2022-09-09T22:53:10.647 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:A2gjEWcERay/bcd6rgAXTjRr1VU= 2022-09-09T22:53:10.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:A2gjEWcERay/bcd6rgAXTjRr1VU=', 'Content-Length': '0'} 2022-09-09T22:53:10.677 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a9e298d50fb7e804-00631bc3d6-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:10.678 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jkcgbj-37 2022-09-09T22:53:10.678 DEBUG:boto:path=/jkcgbj-37/ 2022-09-09T22:53:10.679 DEBUG:boto:auth_path=/jkcgbj-37/ 2022-09-09T22:53:10.679 DEBUG:boto:Method: PUT 2022-09-09T22:53:10.680 DEBUG:boto:Path: /jkcgbj-37/ 2022-09-09T22:53:10.680 DEBUG:boto:Data: 2022-09-09T22:53:10.681 DEBUG:boto:Headers: {} 2022-09-09T22:53:10.681 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:53:10.683 DEBUG:boto:Port: 8000 2022-09-09T22:53:10.683 DEBUG:boto:Params: {} 2022-09-09T22:53:10.685 DEBUG:boto:Token: None 2022-09-09T22:53:10.685 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:53:10 GMT /jkcgbj-37/ 2022-09-09T22:53:10.687 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:K1rnds9n4QK+DArnVJ97RXPtzN8= 2022-09-09T22:53:10.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:K1rnds9n4QK+DArnVJ97RXPtzN8=', 'Content-Length': '0'} 2022-09-09T22:53:10.753 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e836d70bafe8cc26-00631bc3d6-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:10.754 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jkcgbj-38 2022-09-09T22:53:10.755 DEBUG:boto:path=/jkcgbj-38/ 2022-09-09T22:53:10.755 DEBUG:boto:auth_path=/jkcgbj-38/ 2022-09-09T22:53:10.756 DEBUG:boto:Method: PUT 2022-09-09T22:53:10.756 DEBUG:boto:Path: /jkcgbj-38/ 2022-09-09T22:53:10.757 DEBUG:boto:Data: 2022-09-09T22:53:10.757 DEBUG:boto:Headers: {} 2022-09-09T22:53:10.758 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:53:10.758 DEBUG:boto:Port: 8001 2022-09-09T22:53:10.759 DEBUG:boto:Params: {} 2022-09-09T22:53:10.760 DEBUG:boto:Token: None 2022-09-09T22:53:10.761 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:53:10 GMT /jkcgbj-38/ 2022-09-09T22:53:10.762 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:gs9dw6YAt+irQ2KgA/PhENoSBng= 2022-09-09T22:53:10.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:53:10 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:gs9dw6YAt+irQ2KgA/PhENoSBng=', 'Content-Length': '0'} 2022-09-09T22:53:10.808 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a326ede214fe1dde-00631bc3d6-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:53:10.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:53:10.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:53:10.811 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:53:10.852 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:53:10.852 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:53:10.902 INFO:teuthology.orchestra.run.smithi005.stderr:No --period given, using current period=503b1d61-4513-48c8-a308-17b76219610c 2022-09-09T22:53:10.926 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T22:53:10.927 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:53:10.927 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:53:10.927 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:28:06.146147Z" 2022-09-09T22:53:10.928 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:53:10.929 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:53:10.929 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:53:10.929 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:53:10.930 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:53:10.930 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:53:10.930 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:53:10.931 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:30:37.757014Z" 2022-09-09T22:53:10.931 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:53:10.931 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:53:10.931 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662763990.796002_230.1", 2022-09-09T22:53:10.931 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:53:10.796002Z" 2022-09-09T22:53:10.932 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:53:10.932 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T22:53:10.934 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:53:10.934 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-09T22:53:10.935 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:53:10.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:53:10.936 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:53:11.001 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:53:11.002 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:53:11.085 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:11.076+0000 7f309dfb7dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:53:11.087 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:11.076+0000 7f309dfb7dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:53:11.102 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:53:11.103 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:53:11.104 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:53:11.104 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:53:11.105 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:53:11.105 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:53:11.106 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:11.107 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:11.107 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:53:11.107 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:11.108 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:53:11.108 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:11.108 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:11.108 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:53:11.109 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:11.109 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:53:11.109 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:11.109 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:53:11.110 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:11.110 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:11.110 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:11.111 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:11.111 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:53:11.111 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:11.111 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:11.112 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:53:11.112 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:11.112 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:53:11.112 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:11.113 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:53:11.113 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:11.113 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:11.113 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:11.114 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:11.114 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:53:11.114 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:11.115 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:11.115 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:53:11.116 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:11.116 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:53:11.116 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:11.117 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:53:11.117 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:11.118 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:11.119 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:11.119 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:11.119 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:53:11.120 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:11.120 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:11.120 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.629370_169.1", 2022-09-09T22:53:11.120 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:11.121 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:53:11.121 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:11.121 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.629370Z", 2022-09-09T22:53:11.122 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:11.122 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:11.122 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:11.123 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:53:11.123 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:11.123 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:53:11.124 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:53:11.124 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:53:11.124 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:11.125 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:53:11Z" 2022-09-09T22:53:11.125 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:53:11.127 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.629370_169.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.629370Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:53:11Z" } 2022-09-09T22:53:11.128 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.629370_169.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.629370Z', 'realm_epoch': 2}}] 2022-09-09T22:53:11.129 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:53:11.129 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.629370_169.1'} 2022-09-09T22:53:11.130 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.629370_169.1 2022-09-09T22:53:16.131 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:53:16.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:53:16.134 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:53:16.181 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:53:16.182 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:53:16.275 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:16.272+0000 7f85da8bcdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:53:16.281 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:16.276+0000 7f85da8bcdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:53:16.332 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:53:16.333 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:53:16.334 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:53:16.334 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:53:16.335 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:53:16.336 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:53:16.337 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:16.338 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:16.339 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:53:16.340 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:16.340 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:53:16.341 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:16.341 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:16.342 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:53:16.342 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:16.343 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:53:16.344 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:16.344 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:53:16.344 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:16.345 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:16.345 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:16.346 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:16.346 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:53:16.347 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:16.347 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:16.348 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:53:16.348 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:16.351 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:53:16.352 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:16.352 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:53:16.352 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:16.353 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:16.353 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:16.354 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:16.355 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:53:16.355 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:16.356 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:16.356 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:53:16.357 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:16.357 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:53:16.358 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:16.358 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:53:16.358 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:16.359 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:16.359 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:16.360 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:16.360 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:53:16.361 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:16.361 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:16.362 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.591517_166.1", 2022-09-09T22:53:16.362 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:16.362 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:53:16.363 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:16.363 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.591517Z", 2022-09-09T22:53:16.363 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:16.363 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:16.364 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:16.364 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:53:16.365 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:16.365 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:53:16.366 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:53:16.366 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:53:16.367 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:16.367 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:53:16Z" 2022-09-09T22:53:16.368 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:53:16.370 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.591517_166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.591517Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:53:16Z" } 2022-09-09T22:53:16.371 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.591517_166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.591517Z', 'realm_epoch': 2}}] 2022-09-09T22:53:16.372 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:53:16.373 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.591517_166.1'} 2022-09-09T22:53:16.374 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.591517_166.1 2022-09-09T22:53:21.376 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:53:21.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:53:21.378 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:53:21.433 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:53:21.434 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:53:21.495 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:21.492+0000 7f5369333dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:53:21.497 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:21.492+0000 7f5369333dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:53:21.549 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:53:21.550 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:53:21.550 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:53:21.551 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:53:21.551 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:53:21.552 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:53:21.552 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:21.552 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:21.553 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:53:21.553 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:21.554 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:53:21.554 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:21.555 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:21.555 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:53:21.555 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:21.556 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:53:21.556 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:21.557 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:53:21.557 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:21.557 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:21.558 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:21.558 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:21.558 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:53:21.559 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:21.559 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:21.559 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:53:21.559 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:21.560 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:53:21.560 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:21.560 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:53:21.560 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:21.561 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:21.561 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:21.561 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:21.561 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:53:21.562 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:21.562 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:21.562 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:53:21.563 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:21.563 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:53:21.563 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:21.563 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:53:21.564 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:21.564 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:21.564 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:21.565 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:21.565 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:53:21.565 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:21.565 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:21.566 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.861162_181.1", 2022-09-09T22:53:21.566 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:21.566 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:53:21.566 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:21.567 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.861162Z", 2022-09-09T22:53:21.567 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:21.567 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:21.568 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:21.568 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:53:21.568 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:21.569 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:53:21.569 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:53:21.569 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:53:21.569 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:21.570 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:53:21Z" 2022-09-09T22:53:21.570 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:53:21.572 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.861162_181.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.861162Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:53:21Z" } 2022-09-09T22:53:21.572 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.861162_181.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.861162Z', 'realm_epoch': 2}}] 2022-09-09T22:53:21.573 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:53:21.574 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.861162_181.1'} 2022-09-09T22:53:21.575 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.861162_181.1 2022-09-09T22:53:26.577 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:53:26.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:53:26.579 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:53:26.628 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:53:26.628 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:53:26.682 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:26.676+0000 7fe67a6afdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:53:26.683 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:26.680+0000 7fe67a6afdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:53:26.729 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:53:26.729 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:53:26.730 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:53:26.730 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:53:26.731 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:53:26.731 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:53:26.732 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:26.732 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:26.732 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:53:26.733 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:26.733 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:53:26.734 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:26.734 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:26.734 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:53:26.735 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:26.735 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:53:26.735 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:26.735 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:53:26.737 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:26.737 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:26.737 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:26.738 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:26.738 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:53:26.739 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:26.739 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:26.740 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:53:26.740 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:26.741 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:53:26.741 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:26.741 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:53:26.742 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:26.742 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:26.743 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:26.743 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:26.744 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:53:26.744 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:26.745 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:26.746 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:53:26.747 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:26.747 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:53:26.747 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:26.748 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:53:26.748 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:26.748 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:26.748 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:26.749 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:26.749 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:53:26.749 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:26.749 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:26.750 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763988.086181_195.1", 2022-09-09T22:53:26.750 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:26.750 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:53:26.750 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:26.751 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.086181Z", 2022-09-09T22:53:26.751 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:26.751 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:26.752 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:26.752 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:53:26.752 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:26.752 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:53:26.753 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:53:26.753 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:53:26.753 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:26.753 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:53:26Z" 2022-09-09T22:53:26.753 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:53:26.755 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763988.086181_195.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:08.086181Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:53:26Z" } 2022-09-09T22:53:26.756 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763988.086181_195.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:08.086181Z', 'realm_epoch': 2}}] 2022-09-09T22:53:26.756 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:53:26.757 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763988.086181_195.1'} 2022-09-09T22:53:26.757 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763988.086181_195.1 2022-09-09T22:53:31.758 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:53:31.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:53:31.760 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:53:31.802 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:53:31.802 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:53:31.850 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:31.844+0000 7f41b0dc2dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:53:31.851 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:31.848+0000 7f41b0dc2dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:53:31.876 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:53:31.877 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:53:31.877 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:53:31.877 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:53:31.878 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:53:31.878 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:53:31.878 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:31.879 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:31.879 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:53:31.879 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:31.880 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:53:31.880 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:31.880 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:31.880 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:53:31.881 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:31.881 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:53:31.881 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:31.881 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:53:31.882 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:31.882 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:31.882 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:31.882 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:31.883 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:53:31.883 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:31.883 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:31.883 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:53:31.884 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:31.884 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:53:31.884 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:31.884 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:53:31.885 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:31.885 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:31.885 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:31.885 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:31.886 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:53:31.886 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:31.886 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:31.886 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:53:31.887 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:31.887 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:53:31.887 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:31.887 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:53:31.888 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:31.888 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:31.888 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:31.888 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:31.889 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:53:31.889 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:31.889 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:31.890 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.739796_178.1", 2022-09-09T22:53:31.890 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:31.890 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:53:31.890 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:31.891 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.739796Z", 2022-09-09T22:53:31.891 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:31.891 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:31.892 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:31.892 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:53:31.892 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:31.892 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:53:31.893 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:53:31.893 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:53:31.893 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:31.893 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:53:31Z" 2022-09-09T22:53:31.894 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:53:31.895 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.739796_178.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.739796Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:53:31Z" } 2022-09-09T22:53:31.896 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.739796_178.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.739796Z', 'realm_epoch': 2}}] 2022-09-09T22:53:31.896 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:53:31.897 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.739796_178.1'} 2022-09-09T22:53:31.897 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.739796_178.1 2022-09-09T22:53:36.898 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:53:36.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:53:36.900 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:53:36.949 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:53:36.949 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:53:37.002 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:36.996+0000 7f101324bdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:53:37.004 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:37.000+0000 7f101324bdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:53:37.031 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:53:37.032 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:53:37.032 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:53:37.033 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:53:37.033 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:53:37.033 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:53:37.034 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:37.034 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:37.034 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:53:37.035 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:37.035 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:53:37.035 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:37.035 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:37.036 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:53:37.036 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:37.036 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:53:37.037 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:37.037 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:53:37.037 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:37.037 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:37.038 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:37.038 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:37.038 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:53:37.038 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:37.039 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:37.039 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:53:37.039 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:37.040 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:53:37.040 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:37.040 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:53:37.040 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:37.041 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:37.041 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:37.041 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:37.042 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:53:37.042 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:37.042 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:37.042 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:53:37.043 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:37.043 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:53:37.043 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:37.044 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:53:37.044 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:37.044 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:37.044 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:37.045 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:37.045 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:53:37.045 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:37.046 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:37.046 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.815014_179.1", 2022-09-09T22:53:37.046 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:37.046 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:53:37.047 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:37.047 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.815014Z", 2022-09-09T22:53:37.047 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:37.048 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:37.048 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:37.048 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:53:37.048 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:37.049 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:53:37.049 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:53:37.049 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:53:37.050 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:37.050 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:53:37Z" 2022-09-09T22:53:37.050 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:53:37.052 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.815014_179.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.815014Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:53:37Z" } 2022-09-09T22:53:37.053 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.815014_179.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.815014Z', 'realm_epoch': 2}}] 2022-09-09T22:53:37.054 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:53:37.054 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.815014_179.1'} 2022-09-09T22:53:37.055 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.815014_179.1 2022-09-09T22:53:42.057 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:53:42.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:53:42.058 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:53:42.104 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:53:42.104 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:53:42.181 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:42.176+0000 7f6e4f828dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:53:42.182 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:42.180+0000 7f6e4f828dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:53:42.244 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:53:42.245 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:53:42.245 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:53:42.246 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:53:42.246 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:53:42.247 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:53:42.247 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:42.247 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:42.248 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:53:42.248 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:42.249 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:53:42.249 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:42.250 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:42.250 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:53:42.251 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:42.251 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:53:42.252 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:42.252 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:53:42.252 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:42.253 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:42.253 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:42.254 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:42.254 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:53:42.255 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:42.255 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:42.256 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:53:42.256 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:42.257 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:53:42.257 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:42.258 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:53:42.258 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:42.259 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:42.259 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:42.259 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:42.260 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:53:42.260 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:42.261 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:42.261 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:53:42.261 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:42.262 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:53:42.262 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:42.263 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:53:42.263 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:42.263 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:42.264 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:42.264 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:42.264 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:53:42.264 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:42.265 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:42.265 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.957457_187.1", 2022-09-09T22:53:42.265 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:42.265 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:53:42.266 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:42.266 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.957457Z", 2022-09-09T22:53:42.266 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:42.266 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:42.267 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:42.267 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:53:42.267 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:42.267 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:53:42.267 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:53:42.268 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:53:42.268 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:42.268 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:53:42Z" 2022-09-09T22:53:42.268 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:53:42.270 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.957457_187.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.957457Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:53:42Z" } 2022-09-09T22:53:42.270 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.957457_187.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.957457Z', 'realm_epoch': 2}}] 2022-09-09T22:53:42.271 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:53:42.271 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.957457_187.1'} 2022-09-09T22:53:42.272 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.957457_187.1 2022-09-09T22:53:47.273 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:53:47.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:53:47.275 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:53:47.316 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:53:47.317 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:53:47.375 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:47.372+0000 7ff12f1a6dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:53:47.376 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:47.372+0000 7ff12f1a6dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:53:47.402 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:53:47.403 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:53:47.403 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:53:47.403 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:53:47.404 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:53:47.404 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:53:47.405 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:47.405 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:47.405 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:53:47.405 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:47.405 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:53:47.406 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:47.406 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:47.406 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:53:47.406 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:47.407 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:53:47.407 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:47.407 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:53:47.408 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:47.408 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:47.408 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:47.408 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:47.409 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:53:47.409 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:47.409 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:47.410 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:53:47.410 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:47.410 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:53:47.410 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:47.411 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:53:47.411 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:47.411 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:47.411 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:47.412 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:47.412 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:53:47.412 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:47.412 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:47.413 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:53:47.413 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:47.413 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:53:47.413 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:47.414 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:53:47.414 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:47.414 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:47.414 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:47.415 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:47.415 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:53:47.415 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:47.415 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:47.416 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.957457_187.1", 2022-09-09T22:53:47.416 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:47.416 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:53:47.416 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:47.416 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.957457Z", 2022-09-09T22:53:47.417 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:47.417 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:47.417 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:47.417 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:53:47.418 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:47.418 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:53:47.418 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:53:47.418 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:53:47.419 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:47.419 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:53:47Z" 2022-09-09T22:53:47.419 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:53:47.421 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.957457_187.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.957457Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:53:47Z" } 2022-09-09T22:53:47.421 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.957457_187.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.957457Z', 'realm_epoch': 2}}] 2022-09-09T22:53:47.422 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:53:47.422 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.957457_187.1'} 2022-09-09T22:53:47.423 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.957457_187.1 2022-09-09T22:53:52.424 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:53:52.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:53:52.426 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:53:52.484 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:53:52.485 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:53:52.543 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:52.540+0000 7f99df72edc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:53:52.545 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:52.540+0000 7f99df72edc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:53:52.579 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:53:52.580 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:53:52.580 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:53:52.581 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:53:52.581 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:53:52.583 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:53:52.583 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:52.584 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:52.584 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:53:52.584 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:52.585 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:53:52.585 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:52.585 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:52.585 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:53:52.586 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:52.586 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:53:52.586 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:52.587 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:53:52.587 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:52.587 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:52.587 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:52.588 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:52.588 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:53:52.588 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:52.588 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:52.589 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:53:52.589 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:52.589 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:53:52.590 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:52.590 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:53:52.590 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:52.590 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:52.591 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:52.591 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:52.591 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:53:52.592 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:52.592 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:52.592 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:53:52.592 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:52.593 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:53:52.593 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:52.593 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:53:52.593 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:52.594 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:52.594 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:52.594 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:52.594 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:53:52.595 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:52.595 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:52.595 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.815014_179.1", 2022-09-09T22:53:52.595 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:52.596 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:53:52.596 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:52.596 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.815014Z", 2022-09-09T22:53:52.596 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:52.597 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:52.597 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:52.597 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:53:52.597 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:52.598 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:53:52.598 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:53:52.598 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:53:52.598 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:52.599 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:53:52Z" 2022-09-09T22:53:52.599 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:53:52.599 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.815014_179.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.815014Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:53:52Z" } 2022-09-09T22:53:52.600 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.815014_179.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.815014Z', 'realm_epoch': 2}}] 2022-09-09T22:53:52.601 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:53:52.602 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.815014_179.1'} 2022-09-09T22:53:52.602 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.815014_179.1 2022-09-09T22:53:57.604 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:53:57.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:53:57.606 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:53:57.652 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:53:57.652 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:53:57.703 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:57.699+0000 7fc92c1f6dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:53:57.704 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:53:57.699+0000 7fc92c1f6dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:53:57.727 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:53:57.728 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:53:57.728 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:53:57.729 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:53:57.730 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:53:57.730 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:53:57.731 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:57.731 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:57.731 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:53:57.732 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:57.732 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:53:57.732 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:57.733 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:57.733 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:53:57.733 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:57.734 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:53:57.734 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:57.734 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:53:57.734 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:57.735 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:57.735 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:57.735 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:57.736 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:53:57.736 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:57.736 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:57.736 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:53:57.737 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:57.737 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:53:57.737 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:57.737 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:53:57.738 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:57.738 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:57.739 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:57.739 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:57.739 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:53:57.740 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:57.740 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:57.740 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:53:57.741 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:57.741 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:53:57.741 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:57.742 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:53:57.742 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:57.742 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:57.742 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:57.743 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:53:57.743 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:53:57.743 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:53:57.743 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:53:57.744 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.957457_187.1", 2022-09-09T22:53:57.744 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:53:57.744 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:53:57.744 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:53:57.745 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.957457Z", 2022-09-09T22:53:57.745 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:53:57.745 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:57.746 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:53:57.746 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:53:57.746 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:57.746 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:53:57.747 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:53:57.747 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:53:57.747 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:53:57.747 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:53:57Z" 2022-09-09T22:53:57.748 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:53:57.751 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.957457_187.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.957457Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:53:57Z" } 2022-09-09T22:53:57.752 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.957457_187.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.957457Z', 'realm_epoch': 2}}] 2022-09-09T22:53:57.752 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:53:57.753 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.957457_187.1'} 2022-09-09T22:53:57.753 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.957457_187.1 2022-09-09T22:54:02.754 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:54:02.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:54:02.756 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:54:02.802 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:54:02.803 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:54:02.867 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:54:02.863+0000 7fec98d27dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:54:02.868 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:54:02.863+0000 7fec98d27dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:54:02.896 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:54:02.897 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:54:02.897 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:54:02.898 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:54:02.898 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:54:02.898 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:54:02.898 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:02.899 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:02.899 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:54:02.899 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:02.900 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:54:02.900 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:02.900 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:02.900 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:54:02.901 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:02.901 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:54:02.902 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:02.902 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:54:02.903 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:02.903 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:02.904 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:02.904 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:02.905 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:54:02.905 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:02.906 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:02.907 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:54:02.907 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:02.908 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:02.908 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:02.908 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:54:02.909 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:02.909 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:02.909 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:02.909 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:02.910 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:54:02.910 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:02.910 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:02.911 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:54:02.911 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:02.911 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:02.912 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:02.912 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:54:02.913 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:02.913 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:02.913 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:02.914 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:02.914 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:54:02.915 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:02.915 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:02.915 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.966754_190.1", 2022-09-09T22:54:02.916 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:02.916 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:02.916 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:02.916 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.966754Z", 2022-09-09T22:54:02.917 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:02.917 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:02.917 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:02.917 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:54:02.918 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:02.918 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:54:02.918 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:54:02.919 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:54:02.919 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:02.919 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:54:02Z" 2022-09-09T22:54:02.919 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:54:02.921 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.966754_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.966754Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:54:02Z" } 2022-09-09T22:54:02.922 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.966754_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.966754Z', 'realm_epoch': 2}}] 2022-09-09T22:54:02.923 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:54:02.923 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.966754_190.1'} 2022-09-09T22:54:02.924 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.966754_190.1 2022-09-09T22:54:07.925 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:54:07.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:54:07.927 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:54:07.975 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:54:07.975 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:54:08.037 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:54:08.031+0000 7fe315e47dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:54:08.038 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:54:08.035+0000 7fe315e47dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:54:08.065 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:54:08.065 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:54:08.065 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:54:08.066 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:54:08.066 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:54:08.066 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:54:08.067 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:08.067 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:08.067 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:54:08.067 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:08.068 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:54:08.068 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:08.068 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:08.068 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:54:08.069 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:08.069 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:54:08.069 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:08.069 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:54:08.070 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:08.070 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:08.070 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:08.070 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:08.071 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:54:08.071 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:08.071 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:08.071 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:54:08.072 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:08.072 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:08.072 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:08.072 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:54:08.073 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:08.073 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:08.074 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:08.074 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:08.075 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:54:08.075 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:08.075 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:08.075 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:54:08.076 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:08.076 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:08.076 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:08.076 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:54:08.077 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:08.077 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:08.077 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:08.077 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:08.078 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:54:08.078 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:08.078 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:08.078 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.739796_178.1", 2022-09-09T22:54:08.079 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:08.079 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:08.079 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:08.079 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.739796Z", 2022-09-09T22:54:08.080 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:08.080 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:08.080 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:08.080 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:54:08.080 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:08.081 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:54:08.081 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:54:08.081 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:54:08.082 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:08.082 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:54:08Z" 2022-09-09T22:54:08.082 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:54:08.083 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.739796_178.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.739796Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:54:08Z" } 2022-09-09T22:54:08.084 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.739796_178.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.739796Z', 'realm_epoch': 2}}] 2022-09-09T22:54:08.085 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:54:08.085 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.739796_178.1'} 2022-09-09T22:54:08.086 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.739796_178.1 2022-09-09T22:54:13.087 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:54:13.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:54:13.089 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:54:13.135 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:54:13.136 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:54:13.191 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:54:13.187+0000 7ff703934dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:54:13.194 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:54:13.187+0000 7ff703934dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:54:13.224 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:54:13.224 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:54:13.225 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:54:13.225 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:54:13.225 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:54:13.225 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:54:13.226 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:13.226 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:13.226 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:54:13.226 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:13.227 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:54:13.227 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:13.227 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:13.227 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:54:13.228 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:13.228 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:54:13.228 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:13.228 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:54:13.229 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:13.229 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:13.229 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:13.229 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:13.230 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:54:13.230 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:13.230 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:13.231 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:54:13.231 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:13.231 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:13.232 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:13.232 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:54:13.232 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:13.232 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:13.233 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:13.233 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:13.233 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:54:13.233 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:13.234 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:13.234 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:54:13.234 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:13.234 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:13.235 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:13.235 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:54:13.235 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:13.235 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:13.236 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:13.236 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:13.236 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:54:13.236 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:13.237 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:13.237 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.963310_189.1", 2022-09-09T22:54:13.237 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:13.237 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:13.237 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:13.238 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.963310Z", 2022-09-09T22:54:13.238 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:13.238 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:13.239 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:13.239 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:54:13.239 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:13.239 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:54:13.239 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:54:13.240 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:54:13.240 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:13.240 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:54:13Z" 2022-09-09T22:54:13.240 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:54:13.242 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.963310_189.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.963310Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:54:13Z" } 2022-09-09T22:54:13.243 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.963310_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.963310Z', 'realm_epoch': 2}}] 2022-09-09T22:54:13.244 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:54:13.244 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.963310_189.1'} 2022-09-09T22:54:13.245 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.963310_189.1 2022-09-09T22:54:18.246 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:54:18.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:54:18.249 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:54:18.297 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:54:18.298 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:54:18.364 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:54:18.359+0000 7f9ed36a4dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:54:18.366 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:54:18.363+0000 7f9ed36a4dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:54:18.395 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:54:18.396 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:54:18.396 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:54:18.396 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:54:18.397 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:54:18.397 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:54:18.397 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:18.397 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:18.398 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:54:18.398 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:18.399 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:54:18.399 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:18.399 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:18.400 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:54:18.400 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:18.401 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:54:18.401 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:18.401 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:54:18.401 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:18.401 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:18.402 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:18.402 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:18.402 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:54:18.403 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:18.403 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:18.403 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:54:18.403 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:18.404 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:18.404 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:18.404 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:54:18.404 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:18.405 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:18.405 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:18.405 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:18.406 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:54:18.406 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:18.406 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:18.406 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:54:18.407 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:18.407 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:18.407 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:18.407 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:54:18.408 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:18.408 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:18.408 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:18.409 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:18.409 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:54:18.410 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:18.410 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:18.410 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.815014_179.1", 2022-09-09T22:54:18.410 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:18.411 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:18.411 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:18.411 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.815014Z", 2022-09-09T22:54:18.411 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:18.414 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:18.414 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:18.415 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:54:18.415 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:18.416 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:54:18.416 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:54:18.416 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:54:18.417 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:18.417 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:54:18Z" 2022-09-09T22:54:18.417 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:54:18.419 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.815014_179.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.815014Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:54:18Z" } 2022-09-09T22:54:18.420 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.815014_179.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.815014Z', 'realm_epoch': 2}}] 2022-09-09T22:54:18.421 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:54:18.421 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.815014_179.1'} 2022-09-09T22:54:18.422 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.815014_179.1 2022-09-09T22:54:23.423 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:54:23.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:54:23.425 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:54:23.475 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:54:23.476 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:54:23.586 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:54:23.583+0000 7f2a06c2bdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:54:23.588 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:54:23.583+0000 7f2a06c2bdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:54:23.603 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:54:23.604 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:54:23.604 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:54:23.605 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:54:23.605 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:54:23.606 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:54:23.606 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:23.607 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:23.607 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:54:23.607 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:23.608 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:54:23.608 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:23.609 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:23.609 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:54:23.609 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:23.610 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:54:23.610 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:23.610 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:54:23.611 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:23.612 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:23.612 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:23.612 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:23.612 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:54:23.613 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:23.613 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:23.613 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:54:23.614 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:23.615 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:23.615 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:23.615 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:54:23.616 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:23.616 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:23.616 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:23.616 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:23.617 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:54:23.617 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:23.617 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:23.617 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:54:23.618 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:23.618 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:23.619 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:23.619 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:54:23.620 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:23.620 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:23.621 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:23.621 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:23.621 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:54:23.622 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:23.623 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:23.623 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.815014_179.1", 2022-09-09T22:54:23.623 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:23.624 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:23.624 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:23.625 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.815014Z", 2022-09-09T22:54:23.625 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:23.626 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:23.626 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:23.627 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:54:23.627 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:23.627 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:54:23.628 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:54:23.628 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:54:23.628 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:23.629 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:54:23Z" 2022-09-09T22:54:23.629 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:54:23.630 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.815014_179.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.815014Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:54:23Z" } 2022-09-09T22:54:23.631 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.815014_179.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.815014Z', 'realm_epoch': 2}}] 2022-09-09T22:54:23.632 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:54:23.632 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.815014_179.1'} 2022-09-09T22:54:23.633 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.815014_179.1 2022-09-09T22:54:28.634 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:54:28.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:54:28.636 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:54:28.684 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:54:28.685 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:54:28.747 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:54:28.747+0000 7fdcbb440dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:54:28.749 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:54:28.747+0000 7fdcbb440dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:54:28.776 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:54:28.777 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:54:28.777 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:54:28.777 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:54:28.778 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:54:28.778 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:54:28.778 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:28.779 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:28.779 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:54:28.779 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:28.779 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:54:28.780 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:28.780 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:28.780 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:54:28.780 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:28.781 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:54:28.781 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:28.781 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:54:28.781 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:28.782 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:28.782 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:28.782 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:28.782 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:54:28.783 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:28.783 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:28.783 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:54:28.783 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:28.784 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:28.784 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:28.784 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:54:28.785 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:28.785 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:28.785 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:28.785 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:28.786 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:54:28.786 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:28.786 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:28.786 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:54:28.787 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:28.787 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:28.787 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:28.787 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:54:28.788 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:28.788 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:28.788 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:28.789 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:28.789 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:54:28.789 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:28.789 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:28.790 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.815014_179.1", 2022-09-09T22:54:28.790 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:28.790 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:28.790 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:28.791 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.815014Z", 2022-09-09T22:54:28.791 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:28.791 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:28.791 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:28.791 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:54:28.792 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:28.792 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:54:28.792 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:54:28.793 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:54:28.793 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:28.794 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:54:28Z" 2022-09-09T22:54:28.794 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:54:28.795 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.815014_179.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.815014Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:54:28Z" } 2022-09-09T22:54:28.796 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.815014_179.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.815014Z', 'realm_epoch': 2}}] 2022-09-09T22:54:28.797 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:54:28.797 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.815014_179.1'} 2022-09-09T22:54:28.797 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.815014_179.1 2022-09-09T22:54:33.798 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:54:33.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:54:33.800 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:54:33.849 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:54:33.850 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:54:33.912 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:54:33.911+0000 7f0104e13dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:54:33.914 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:54:33.915+0000 7f0104e13dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:54:33.942 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:54:33.942 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:54:33.942 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:54:33.943 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:54:33.943 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:54:33.943 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:54:33.943 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:33.944 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:33.944 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:54:33.944 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:33.944 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:54:33.945 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:33.945 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:33.945 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:54:33.945 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:33.946 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:54:33.946 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:33.946 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:54:33.947 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:33.947 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:33.947 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:33.947 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:33.948 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:54:33.948 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:33.948 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:33.949 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:54:33.949 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:33.949 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:33.949 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:33.950 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:54:33.950 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:33.950 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:33.951 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:33.951 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:33.951 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:54:33.952 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:33.952 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:33.952 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:54:33.953 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:33.953 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:33.953 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:33.953 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:54:33.954 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:33.954 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:33.954 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:33.955 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:33.955 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:54:33.955 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:33.955 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:33.956 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.909698_184.1", 2022-09-09T22:54:33.956 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:33.956 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:33.957 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:33.957 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.909698Z", 2022-09-09T22:54:33.957 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:33.957 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:33.958 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:33.958 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:54:33.958 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:33.959 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:54:33.959 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:54:33.959 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:54:33.959 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:33.960 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:54:33Z" 2022-09-09T22:54:33.960 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:54:33.961 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.909698_184.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.909698Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:54:33Z" } 2022-09-09T22:54:33.962 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.909698_184.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.909698Z', 'realm_epoch': 2}}] 2022-09-09T22:54:33.963 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:54:33.963 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.909698_184.1'} 2022-09-09T22:54:33.964 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.909698_184.1 2022-09-09T22:54:38.965 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:54:38.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:54:38.967 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:54:39.014 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:54:39.015 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:54:39.071 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:54:39.071+0000 7f4cf0c27dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:54:39.073 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:54:39.071+0000 7f4cf0c27dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:54:39.098 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:54:39.099 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:54:39.099 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:54:39.100 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:54:39.100 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:54:39.101 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:54:39.101 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:39.101 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:39.101 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:54:39.102 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:39.102 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:54:39.103 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:39.104 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:39.104 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:54:39.105 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:39.105 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:54:39.105 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:39.106 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:54:39.107 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:39.107 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:39.107 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:39.108 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:39.108 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:54:39.109 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:39.109 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:39.110 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:54:39.110 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:39.111 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:39.111 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:39.111 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:54:39.112 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:39.112 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:39.112 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:39.113 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:39.113 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:54:39.113 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:39.113 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:39.114 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:54:39.114 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:39.114 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:39.115 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:39.115 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:54:39.116 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:39.116 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:39.117 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:39.117 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:39.118 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:54:39.118 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:39.119 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:39.119 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.739796_178.1", 2022-09-09T22:54:39.120 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:39.121 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:39.122 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:39.122 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.739796Z", 2022-09-09T22:54:39.122 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:39.122 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:39.124 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:39.125 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:54:39.125 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:39.126 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:54:39.127 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:54:39.127 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:54:39.127 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:39.128 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:54:39Z" 2022-09-09T22:54:39.129 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:54:39.131 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.739796_178.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.739796Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:54:39Z" } 2022-09-09T22:54:39.133 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.739796_178.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.739796Z', 'realm_epoch': 2}}] 2022-09-09T22:54:39.134 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:54:39.135 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.739796_178.1'} 2022-09-09T22:54:39.136 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.739796_178.1 2022-09-09T22:54:44.137 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:54:44.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:54:44.139 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:54:44.190 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:54:44.191 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:54:44.240 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:54:44.239+0000 7f912995cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:54:44.242 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:54:44.239+0000 7f912995cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:54:44.267 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:54:44.268 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:54:44.268 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:54:44.268 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:54:44.269 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:54:44.269 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:54:44.269 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:44.270 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:44.270 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:54:44.270 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:44.271 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:54:44.272 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:44.272 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:44.272 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:54:44.272 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:44.273 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:54:44.273 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:44.273 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:54:44.273 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:44.274 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:44.274 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:44.274 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:44.275 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:54:44.275 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:44.276 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:44.276 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:54:44.277 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:44.277 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:44.277 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:44.278 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:54:44.278 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:44.278 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:44.279 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:44.279 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:44.279 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:54:44.279 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:44.280 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:44.280 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:54:44.280 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:44.281 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:44.281 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:44.281 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:54:44.281 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:44.282 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:44.282 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:44.282 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:44.282 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:54:44.283 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:44.283 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:44.283 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763988.086181_195.1", 2022-09-09T22:54:44.283 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:44.284 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:44.284 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:44.284 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.086181Z", 2022-09-09T22:54:44.284 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:44.285 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:44.285 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:44.285 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:54:44.286 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:44.286 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:54:44.286 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:54:44.286 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:54:44.287 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:44.287 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:54:44Z" 2022-09-09T22:54:44.287 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:54:44.289 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763988.086181_195.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:08.086181Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:54:44Z" } 2022-09-09T22:54:44.289 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763988.086181_195.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:08.086181Z', 'realm_epoch': 2}}] 2022-09-09T22:54:44.290 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:54:44.291 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763988.086181_195.1'} 2022-09-09T22:54:44.291 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763988.086181_195.1 2022-09-09T22:54:49.293 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:54:49.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:54:49.295 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:54:49.342 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:54:49.342 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:54:49.416 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:54:49.414+0000 7f52670d7dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:54:49.418 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:54:49.418+0000 7f52670d7dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:54:49.449 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:54:49.450 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:54:49.450 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:54:49.450 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:54:49.451 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:54:49.451 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:54:49.451 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:49.451 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:49.452 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:54:49.452 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:49.452 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:54:49.452 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:49.453 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:49.453 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:54:49.453 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:49.453 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:54:49.454 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:49.454 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:54:49.454 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:49.454 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:49.455 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:49.455 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:49.455 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:54:49.455 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:49.456 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:49.456 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:54:49.457 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:49.458 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:49.458 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:49.458 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:54:49.458 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:49.460 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:49.460 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:49.460 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:49.460 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:54:49.461 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:49.461 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:49.461 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:54:49.461 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:49.462 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:49.462 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:49.462 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:54:49.462 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:49.463 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:49.463 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:49.463 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:49.463 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:54:49.464 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:49.464 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:49.464 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.815014_179.1", 2022-09-09T22:54:49.464 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:49.464 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:49.465 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:49.465 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.815014Z", 2022-09-09T22:54:49.465 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:49.465 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:49.466 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:49.466 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:54:49.466 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:49.466 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:54:49.467 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:54:49.467 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:54:49.467 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:49.467 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:54:49Z" 2022-09-09T22:54:49.468 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:54:49.469 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.815014_179.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.815014Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:54:49Z" } 2022-09-09T22:54:49.469 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.815014_179.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.815014Z', 'realm_epoch': 2}}] 2022-09-09T22:54:49.470 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:54:49.470 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.815014_179.1'} 2022-09-09T22:54:49.471 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.815014_179.1 2022-09-09T22:54:54.473 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:54:54.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:54:54.475 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:54:54.516 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:54:54.517 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:54:54.574 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:54:54.570+0000 7ff5c6ea3dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:54:54.575 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:54:54.574+0000 7ff5c6ea3dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:54:54.601 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:54:54.601 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:54:54.601 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:54:54.602 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:54:54.602 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:54:54.602 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:54:54.602 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:54.603 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:54.603 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:54:54.603 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:54.604 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:54:54.604 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:54.604 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:54.604 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:54:54.605 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:54.605 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:54:54.605 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:54.605 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:54:54.606 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:54.606 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:54.606 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:54.606 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:54.607 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:54:54.607 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:54.607 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:54.607 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:54:54.608 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:54.608 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:54.608 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:54.608 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:54:54.609 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:54.609 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:54.609 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:54.609 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:54.610 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:54:54.610 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:54.610 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:54.610 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:54:54.611 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:54.611 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:54.611 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:54.611 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:54:54.612 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:54.612 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:54.612 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:54.612 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:54.613 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:54:54.613 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:54.613 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:54.613 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.905416_183.1", 2022-09-09T22:54:54.614 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:54.614 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:54.614 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:54.615 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.905416Z", 2022-09-09T22:54:54.615 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:54.615 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:54.615 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:54.616 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:54:54.616 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:54.616 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:54:54.616 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:54:54.617 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:54:54.617 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:54.617 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:54:54Z" 2022-09-09T22:54:54.617 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:54:54.619 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.905416_183.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.905416Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:54:54Z" } 2022-09-09T22:54:54.620 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.905416_183.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.905416Z', 'realm_epoch': 2}}] 2022-09-09T22:54:54.620 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:54:54.621 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.905416_183.1'} 2022-09-09T22:54:54.621 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.905416_183.1 2022-09-09T22:54:59.623 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:54:59.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:54:59.624 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:54:59.675 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:54:59.675 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:54:59.730 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:54:59.730+0000 7fcfa8c94dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:54:59.732 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:54:59.730+0000 7fcfa8c94dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:54:59.759 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:54:59.760 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:54:59.760 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:54:59.760 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:54:59.760 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:54:59.761 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:54:59.761 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:59.761 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:59.762 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:54:59.762 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:59.762 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:54:59.763 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:59.763 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:59.763 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:54:59.763 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:59.764 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:54:59.766 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:59.767 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:54:59.768 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:59.768 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:59.769 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:59.769 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:59.770 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:54:59.770 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:59.771 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:59.771 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:54:59.772 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:59.772 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:59.773 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:59.773 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:54:59.774 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:59.775 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:59.777 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:59.778 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:59.778 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:54:59.779 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:59.779 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:59.780 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:54:59.780 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:59.781 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:59.781 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:59.781 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:54:59.782 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:59.782 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:59.783 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:59.783 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:54:59.784 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:54:59.784 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:54:59.785 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:54:59.785 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.815014_179.1", 2022-09-09T22:54:59.786 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:54:59.786 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:54:59.787 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:54:59.787 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.815014Z", 2022-09-09T22:54:59.788 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:54:59.788 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:59.789 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:54:59.789 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:54:59.789 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:59.789 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:54:59.790 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:54:59.790 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:54:59.791 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:54:59.791 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:54:59Z" 2022-09-09T22:54:59.791 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:54:59.793 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.815014_179.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.815014Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:54:59Z" } 2022-09-09T22:54:59.794 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.815014_179.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.815014Z', 'realm_epoch': 2}}] 2022-09-09T22:54:59.795 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:54:59.796 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.815014_179.1'} 2022-09-09T22:54:59.796 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.815014_179.1 2022-09-09T22:55:04.798 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:55:04.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:55:04.800 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:55:04.850 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:55:04.851 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:55:04.906 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:55:04.902+0000 7fbd60d1adc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:55:04.907 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:55:04.906+0000 7fbd60d1adc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:55:04.934 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:55:04.934 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:55:04.935 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:55:04.935 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:55:04.935 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:55:04.936 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:55:04.936 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:04.936 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:04.937 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:55:04.937 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:04.938 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:55:04.938 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:04.938 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:04.938 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:55:04.939 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:04.939 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:55:04.939 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:04.940 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:55:04.940 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:04.941 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:04.941 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:04.941 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:04.942 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:55:04.942 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:04.943 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:04.943 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:55:04.944 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:04.944 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:04.945 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:04.945 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:55:04.946 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:04.946 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:04.947 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:04.947 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:04.948 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:55:04.948 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:04.949 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:04.949 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:55:04.949 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:04.950 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:04.951 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:04.951 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:55:04.952 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:04.952 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:04.952 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:04.952 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:04.953 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:55:04.953 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:04.953 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:04.953 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.815014_179.1", 2022-09-09T22:55:04.954 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:04.954 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:04.954 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:04.955 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.815014Z", 2022-09-09T22:55:04.955 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:04.955 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:04.955 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:04.956 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:55:04.956 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:04.956 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:55:04.956 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:55:04.957 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:55:04.957 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:04.957 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:55:04Z" 2022-09-09T22:55:04.958 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:55:04.959 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.815014_179.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.815014Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:55:04Z" } 2022-09-09T22:55:04.960 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.815014_179.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.815014Z', 'realm_epoch': 2}}] 2022-09-09T22:55:04.960 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:55:04.961 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.815014_179.1'} 2022-09-09T22:55:04.962 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.815014_179.1 2022-09-09T22:55:09.963 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:55:09.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:55:09.965 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:55:10.013 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:55:10.014 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:55:10.072 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:55:10.070+0000 7f5c42670dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:55:10.074 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:55:10.074+0000 7f5c42670dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:55:10.106 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:55:10.107 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:55:10.107 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:55:10.108 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:55:10.108 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:55:10.109 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:55:10.109 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:10.110 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:10.110 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:55:10.111 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:10.111 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:55:10.112 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:10.112 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:10.113 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:55:10.113 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:10.114 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:55:10.114 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:10.114 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:55:10.115 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:10.115 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:10.115 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:10.116 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:10.116 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:55:10.116 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:10.116 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:10.117 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:55:10.117 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:10.117 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:10.117 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:10.118 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:55:10.118 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:10.118 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:10.119 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:10.119 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:10.119 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:55:10.119 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:10.120 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:10.120 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:55:10.120 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:10.120 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:10.121 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:10.121 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:55:10.121 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:10.121 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:10.122 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:10.122 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:10.122 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:55:10.123 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:10.123 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:10.123 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.966754_190.1", 2022-09-09T22:55:10.123 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:10.124 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:10.124 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:10.124 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.966754Z", 2022-09-09T22:55:10.124 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:10.125 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:10.125 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:10.125 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:55:10.125 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:10.126 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:55:10.126 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:55:10.126 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:55:10.126 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:10.127 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:55:10Z" 2022-09-09T22:55:10.127 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:55:10.128 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.966754_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.966754Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:55:10Z" } 2022-09-09T22:55:10.129 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.966754_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.966754Z', 'realm_epoch': 2}}] 2022-09-09T22:55:10.129 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:55:10.130 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.966754_190.1'} 2022-09-09T22:55:10.131 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.966754_190.1 2022-09-09T22:55:15.132 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:55:15.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:55:15.134 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:55:15.184 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:55:15.184 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:55:15.262 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:55:15.262+0000 7fba9d7d5dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:55:15.264 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:55:15.262+0000 7fba9d7d5dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:55:15.294 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:55:15.295 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:55:15.295 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:55:15.295 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:55:15.295 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:55:15.296 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:55:15.296 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:15.296 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:15.296 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:55:15.297 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:15.297 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:55:15.297 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:15.297 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:15.302 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:55:15.303 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:15.303 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:55:15.303 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:15.303 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:55:15.304 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:15.304 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:15.304 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:15.304 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:15.305 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:55:15.305 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:15.305 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:15.305 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:55:15.306 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:15.306 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:15.306 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:15.306 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:55:15.307 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:15.307 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:15.307 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:15.307 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:15.308 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:55:15.308 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:15.309 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:15.309 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:55:15.310 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:15.310 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:15.310 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:15.311 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:55:15.311 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:15.312 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:15.312 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:15.313 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:15.314 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:55:15.314 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:15.315 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:15.315 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.909698_184.1", 2022-09-09T22:55:15.315 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:15.316 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:15.316 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:15.317 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.909698Z", 2022-09-09T22:55:15.317 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:15.318 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:15.318 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:15.318 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:55:15.319 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:15.319 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:55:15.320 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:55:15.320 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:55:15.320 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:15.321 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:55:15Z" 2022-09-09T22:55:15.321 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:55:15.323 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.909698_184.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.909698Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:55:15Z" } 2022-09-09T22:55:15.325 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.909698_184.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.909698Z', 'realm_epoch': 2}}] 2022-09-09T22:55:15.326 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:55:15.327 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.909698_184.1'} 2022-09-09T22:55:15.328 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.909698_184.1 2022-09-09T22:55:20.329 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:55:20.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:55:20.331 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:55:20.381 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:55:20.381 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:55:20.445 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:55:20.442+0000 7fd48d499dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:55:20.447 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:55:20.446+0000 7fd48d499dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:55:20.477 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:55:20.478 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:55:20.478 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:55:20.479 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:55:20.479 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:55:20.479 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:55:20.479 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:20.480 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:20.480 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:55:20.480 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:20.480 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:55:20.481 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:20.481 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:20.481 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:55:20.481 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:20.482 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:55:20.482 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:20.482 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:55:20.483 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:20.483 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:20.483 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:20.484 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:20.484 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:55:20.484 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:20.484 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:20.485 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:55:20.485 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:20.485 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:20.485 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:20.486 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:55:20.486 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:20.486 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:20.486 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:20.487 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:20.487 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:55:20.487 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:20.487 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:20.488 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:55:20.488 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:20.488 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:20.489 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:20.489 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:55:20.489 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:20.489 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:20.490 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:20.490 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:20.490 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:55:20.490 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:20.491 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:20.491 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.911381_185.1", 2022-09-09T22:55:20.491 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:20.491 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:20.492 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:20.492 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.911381Z", 2022-09-09T22:55:20.492 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:20.493 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:20.493 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:20.493 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:55:20.494 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:20.494 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:55:20.494 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:55:20.495 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:55:20.495 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:20.495 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:55:20Z" 2022-09-09T22:55:20.495 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:55:20.497 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.911381_185.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.911381Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:55:20Z" } 2022-09-09T22:55:20.498 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.911381_185.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.911381Z', 'realm_epoch': 2}}] 2022-09-09T22:55:20.498 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:55:20.499 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.911381_185.1'} 2022-09-09T22:55:20.499 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.911381_185.1 2022-09-09T22:55:25.501 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:55:25.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:55:25.504 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:55:25.550 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:55:25.550 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:55:25.614 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:55:25.610+0000 7fc89d6f4dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:55:25.615 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:55:25.614+0000 7fc89d6f4dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:55:25.645 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:55:25.645 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:55:25.646 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:55:25.646 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:55:25.646 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:55:25.647 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:55:25.647 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:25.647 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:25.648 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:55:25.648 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:25.649 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:55:25.649 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:25.649 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:25.650 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:55:25.650 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:25.650 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:55:25.651 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:25.651 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:55:25.651 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:25.651 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:25.652 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:25.652 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:25.652 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:55:25.652 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:25.653 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:25.653 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:55:25.653 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:25.653 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:25.654 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:25.654 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:55:25.654 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:25.654 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:25.655 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:25.655 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:25.655 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:55:25.655 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:25.656 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:25.656 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:55:25.656 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:25.656 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:25.657 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:25.657 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:55:25.657 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:25.657 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:25.658 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:25.658 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:25.658 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:55:25.658 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:25.659 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:25.659 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.815014_179.1", 2022-09-09T22:55:25.659 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:25.659 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:25.660 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:25.660 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.815014Z", 2022-09-09T22:55:25.660 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:25.660 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:25.661 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:25.661 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:55:25.661 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:25.661 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:55:25.662 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:55:25.662 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:55:25.662 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:25.662 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:55:25Z" 2022-09-09T22:55:25.663 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:55:25.664 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.815014_179.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.815014Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:55:25Z" } 2022-09-09T22:55:25.664 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.815014_179.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.815014Z', 'realm_epoch': 2}}] 2022-09-09T22:55:25.665 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:55:25.665 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.815014_179.1'} 2022-09-09T22:55:25.666 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.815014_179.1 2022-09-09T22:55:30.668 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:55:30.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:55:30.670 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:55:30.718 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:55:30.718 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:55:30.786 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:55:30.786+0000 7f5dd6d0edc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:55:30.788 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:55:30.786+0000 7f5dd6d0edc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:55:30.818 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:55:30.819 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:55:30.819 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:55:30.819 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:55:30.819 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:55:30.820 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:55:30.820 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:30.820 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:30.821 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:55:30.821 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:30.821 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:55:30.821 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:30.822 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:30.822 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:55:30.822 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:30.822 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:55:30.823 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:30.823 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:55:30.823 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:30.823 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:30.824 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:30.824 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:30.824 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:55:30.824 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:30.825 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:30.825 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:55:30.825 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:30.825 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:30.826 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:30.826 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:55:30.826 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:30.826 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:30.827 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:30.827 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:30.827 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:55:30.827 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:30.828 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:30.828 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:55:30.828 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:30.828 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:30.829 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:30.829 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:55:30.829 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:30.829 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:30.830 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:30.830 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:30.830 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:55:30.830 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:30.831 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:30.831 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.966754_190.1", 2022-09-09T22:55:30.831 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:30.831 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:30.832 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:30.832 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.966754Z", 2022-09-09T22:55:30.832 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:30.832 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:30.832 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:30.833 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:55:30.833 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:30.833 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:55:30.833 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:55:30.834 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:55:30.834 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:30.834 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:55:30Z" 2022-09-09T22:55:30.835 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:55:30.836 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.966754_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.966754Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:55:30Z" } 2022-09-09T22:55:30.837 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.966754_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.966754Z', 'realm_epoch': 2}}] 2022-09-09T22:55:30.837 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:55:30.838 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.966754_190.1'} 2022-09-09T22:55:30.838 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.966754_190.1 2022-09-09T22:55:35.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:55:35.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:55:35.841 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:55:35.889 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:55:35.889 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:55:35.949 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:55:35.946+0000 7fdb71f1bdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:55:35.951 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:55:35.950+0000 7fdb71f1bdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:55:35.978 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:55:35.979 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:55:35.979 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:55:35.979 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:55:35.980 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:55:35.980 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:55:35.980 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:35.980 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:35.981 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:55:35.981 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:35.981 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:55:35.982 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:35.982 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:35.982 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:55:35.983 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:35.983 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:55:35.983 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:35.983 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:55:35.984 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:35.984 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:35.984 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:35.984 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:35.985 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:55:35.985 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:35.985 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:35.985 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:55:35.986 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:35.986 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:35.986 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:35.986 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:55:35.987 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:35.987 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:35.987 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:35.987 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:35.988 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:55:35.988 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:35.988 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:35.988 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:55:35.989 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:35.989 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:35.989 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:35.989 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:55:35.990 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:35.990 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:35.990 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:35.991 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:35.991 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:55:35.991 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:35.992 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:35.992 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.905416_183.1", 2022-09-09T22:55:35.993 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:35.993 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:35.993 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:35.993 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.905416Z", 2022-09-09T22:55:35.994 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:35.994 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:35.994 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:35.995 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:55:35.995 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:35.995 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:55:35.995 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:55:35.996 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:55:35.996 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:35.996 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:55:35Z" 2022-09-09T22:55:35.996 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:55:35.998 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.905416_183.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.905416Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:55:35Z" } 2022-09-09T22:55:35.998 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.905416_183.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.905416Z', 'realm_epoch': 2}}] 2022-09-09T22:55:35.999 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:55:35.999 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.905416_183.1'} 2022-09-09T22:55:36.000 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.905416_183.1 2022-09-09T22:55:41.001 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:55:41.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:55:41.003 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:55:41.051 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:55:41.051 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:55:41.120 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:55:41.118+0000 7f495e5f1dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:55:41.121 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:55:41.122+0000 7f495e5f1dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:55:41.150 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:55:41.150 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:55:41.151 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:55:41.151 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:55:41.151 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:55:41.152 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:55:41.152 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:41.152 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:41.152 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:55:41.153 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:41.153 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:55:41.153 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:41.153 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:41.154 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:55:41.154 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:41.154 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:55:41.155 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:41.155 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:55:41.155 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:41.155 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:41.156 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:41.156 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:41.156 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:55:41.157 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:41.157 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:41.157 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:55:41.157 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:41.158 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:41.158 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:41.158 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:55:41.158 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:41.159 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:41.159 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:41.159 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:41.160 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:55:41.160 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:41.160 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:41.160 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:55:41.161 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:41.161 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:41.161 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:41.161 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:55:41.162 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:41.162 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:41.162 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:41.163 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:41.163 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:55:41.163 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:41.163 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:41.164 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763988.045233_194.1", 2022-09-09T22:55:41.164 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:41.164 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:41.164 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:41.165 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.045233Z", 2022-09-09T22:55:41.165 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:41.165 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:41.165 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:41.166 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:55:41.166 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:41.166 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:55:41.166 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:55:41.167 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:55:41.167 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:41.167 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:55:41Z" 2022-09-09T22:55:41.167 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:55:41.169 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763988.045233_194.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:08.045233Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:55:41Z" } 2022-09-09T22:55:41.169 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763988.045233_194.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:08.045233Z', 'realm_epoch': 2}}] 2022-09-09T22:55:41.170 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:55:41.170 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763988.045233_194.1'} 2022-09-09T22:55:41.171 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763988.045233_194.1 2022-09-09T22:55:46.172 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:55:46.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:55:46.175 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:55:46.222 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:55:46.223 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:55:46.283 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:55:46.281+0000 7fac95c8bdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:55:46.286 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:55:46.285+0000 7fac95c8bdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:55:46.315 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:55:46.316 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:55:46.316 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:55:46.316 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:55:46.317 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:55:46.317 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:55:46.317 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:46.318 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:46.318 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:55:46.318 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:46.319 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:55:46.319 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:46.319 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:46.319 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:55:46.320 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:46.320 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:55:46.320 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:46.320 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:55:46.321 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:46.321 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:46.321 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:46.322 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:46.323 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:55:46.323 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:46.324 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:46.324 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:55:46.325 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:46.325 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:46.325 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:46.326 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:55:46.326 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:46.326 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:46.327 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:46.327 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:46.327 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:55:46.327 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:46.328 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:46.328 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:55:46.328 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:46.328 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:46.329 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:46.329 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:55:46.329 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:46.329 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:46.329 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:46.330 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:46.330 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:55:46.330 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:46.330 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:46.331 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.909698_184.1", 2022-09-09T22:55:46.331 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:46.331 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:46.331 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:46.332 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.909698Z", 2022-09-09T22:55:46.332 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:46.332 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:46.332 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:46.333 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:55:46.333 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:46.333 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:55:46.333 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:55:46.334 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:55:46.334 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:46.334 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:55:46Z" 2022-09-09T22:55:46.334 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:55:46.336 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.909698_184.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.909698Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:55:46Z" } 2022-09-09T22:55:46.336 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.909698_184.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.909698Z', 'realm_epoch': 2}}] 2022-09-09T22:55:46.337 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:55:46.337 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.909698_184.1'} 2022-09-09T22:55:46.338 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.909698_184.1 2022-09-09T22:55:51.339 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:55:51.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:55:51.341 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:55:51.408 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:55:51.409 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:55:51.496 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:55:51.493+0000 7fa9d411bdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:55:51.497 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:55:51.493+0000 7fa9d411bdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:55:51.523 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:55:51.524 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:55:51.524 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:55:51.524 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:55:51.525 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:55:51.525 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:55:51.526 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:51.526 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:51.526 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:55:51.527 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:51.527 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:55:51.527 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:51.527 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:51.528 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:55:51.528 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:51.528 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:55:51.528 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:51.529 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:55:51.529 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:51.529 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:51.530 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:51.530 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:51.530 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:55:51.530 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:51.531 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:51.531 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:55:51.531 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:51.531 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:51.532 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:51.532 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:55:51.532 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:51.533 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:51.533 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:51.533 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:51.534 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:55:51.534 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:51.535 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:51.535 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:55:51.536 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:51.536 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:51.537 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:51.537 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:55:51.538 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:51.538 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:51.539 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:51.539 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:51.540 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:55:51.540 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:51.541 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:51.541 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.815014_179.1", 2022-09-09T22:55:51.542 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:51.542 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:51.543 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:51.543 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.815014Z", 2022-09-09T22:55:51.544 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:51.544 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:51.544 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:51.545 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:55:51.545 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:51.546 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:55:51.546 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:55:51.546 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:55:51.547 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:51.547 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:55:51Z" 2022-09-09T22:55:51.548 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:55:51.549 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.815014_179.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.815014Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:55:51Z" } 2022-09-09T22:55:51.550 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.815014_179.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.815014Z', 'realm_epoch': 2}}] 2022-09-09T22:55:51.551 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:55:51.552 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.815014_179.1'} 2022-09-09T22:55:51.552 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.815014_179.1 2022-09-09T22:55:56.554 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:55:56.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:55:56.557 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:55:56.596 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:55:56.597 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:55:56.646 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:55:56.645+0000 7ff361b5bdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:55:56.650 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:55:56.649+0000 7ff361b5bdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:55:56.682 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:55:56.683 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:55:56.683 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:55:56.683 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:55:56.684 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:55:56.684 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:55:56.684 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:56.684 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:56.685 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:55:56.685 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:56.685 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:55:56.685 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:56.686 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:56.686 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:55:56.686 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:56.687 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:55:56.687 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:56.687 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:55:56.687 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:56.688 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:56.688 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:56.688 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:56.688 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:55:56.689 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:56.689 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:56.689 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:55:56.689 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:56.690 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:56.690 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:56.690 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:55:56.691 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:56.691 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:56.691 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:56.692 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:56.692 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:55:56.692 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:56.692 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:56.693 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:55:56.693 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:56.693 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:56.694 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:56.694 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:55:56.694 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:56.694 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:56.695 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:56.695 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:55:56.695 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:55:56.695 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:55:56.696 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:55:56.696 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.909698_184.1", 2022-09-09T22:55:56.696 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:55:56.696 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:55:56.697 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:55:56.697 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.909698Z", 2022-09-09T22:55:56.697 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:55:56.698 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:56.698 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:55:56.698 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:55:56.698 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:56.699 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:55:56.699 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:55:56.699 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:55:56.700 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:55:56.700 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:55:56Z" 2022-09-09T22:55:56.700 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:55:56.702 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.909698_184.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.909698Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:55:56Z" } 2022-09-09T22:55:56.703 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.909698_184.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.909698Z', 'realm_epoch': 2}}] 2022-09-09T22:55:56.703 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:55:56.704 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.909698_184.1'} 2022-09-09T22:55:56.704 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.909698_184.1 2022-09-09T22:56:01.706 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:56:01.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:56:01.708 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:56:01.754 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:56:01.754 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:56:01.824 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:56:01.821+0000 7fe6306e6dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:56:01.826 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:56:01.825+0000 7fe6306e6dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:56:01.856 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:56:01.856 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:56:01.857 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:56:01.857 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:56:01.858 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:56:01.858 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:56:01.859 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:01.859 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:01.859 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:56:01.860 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:01.860 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:56:01.861 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:01.861 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:01.862 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:56:01.862 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:01.862 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:56:01.863 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:01.863 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:56:01.864 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:01.864 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:01.865 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:01.866 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:01.866 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:56:01.867 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:01.868 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:01.868 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:56:01.869 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:01.869 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:01.870 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:01.870 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:56:01.871 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:01.871 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:01.871 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:01.872 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:01.872 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:56:01.873 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:01.873 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:01.874 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:56:01.874 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:01.874 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:01.875 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:01.875 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:56:01.875 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:01.875 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:01.876 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:01.876 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:01.876 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:56:01.876 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:01.877 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:01.877 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.815014_179.1", 2022-09-09T22:56:01.877 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:01.877 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:01.878 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:01.878 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.815014Z", 2022-09-09T22:56:01.879 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:01.879 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:01.879 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:01.879 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:56:01.880 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:01.880 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:56:01.880 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:56:01.880 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:56:01.881 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:01.881 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:56:01Z" 2022-09-09T22:56:01.881 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:56:01.883 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.815014_179.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.815014Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:56:01Z" } 2022-09-09T22:56:01.884 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.815014_179.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.815014Z', 'realm_epoch': 2}}] 2022-09-09T22:56:01.885 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:56:01.886 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.815014_179.1'} 2022-09-09T22:56:01.886 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.815014_179.1 2022-09-09T22:56:06.888 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:56:06.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:56:06.889 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:56:06.939 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:56:06.939 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:56:07.006 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:56:07.005+0000 7f0753649dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:56:07.008 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:56:07.005+0000 7f0753649dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:56:07.035 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:56:07.036 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:56:07.036 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:56:07.037 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:56:07.037 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:56:07.037 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:56:07.038 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:07.038 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:07.038 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:56:07.039 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:07.039 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:56:07.040 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:07.040 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:07.040 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:56:07.040 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:07.041 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:56:07.041 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:07.041 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:56:07.042 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:07.042 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:07.042 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:07.043 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:07.043 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:56:07.043 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:07.043 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:07.043 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:56:07.044 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:07.044 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:07.044 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:07.044 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:56:07.045 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:07.045 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:07.045 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:07.045 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:07.046 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:56:07.046 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:07.046 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:07.046 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:56:07.047 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:07.047 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:07.047 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:07.047 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:56:07.048 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:07.049 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:07.049 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:07.049 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:07.050 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:56:07.050 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:07.050 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:07.050 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.815014_179.1", 2022-09-09T22:56:07.051 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:07.051 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:07.051 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:07.051 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.815014Z", 2022-09-09T22:56:07.052 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:07.052 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:07.052 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:07.052 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:56:07.053 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:07.053 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:56:07.053 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:56:07.053 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:56:07.054 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:07.054 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:56:07Z" 2022-09-09T22:56:07.054 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:56:07.055 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.815014_179.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.815014Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:56:07Z" } 2022-09-09T22:56:07.056 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.815014_179.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.815014Z', 'realm_epoch': 2}}] 2022-09-09T22:56:07.057 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:56:07.057 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.815014_179.1'} 2022-09-09T22:56:07.058 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.815014_179.1 2022-09-09T22:56:12.058 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:56:12.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:56:12.060 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:56:12.108 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:56:12.109 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:56:12.181 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:56:12.177+0000 7f84a5940dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:56:12.182 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:56:12.181+0000 7f84a5940dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:56:12.212 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:56:12.212 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:56:12.213 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:56:12.213 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:56:12.213 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:56:12.213 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:56:12.214 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:12.214 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:12.214 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:56:12.215 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:12.215 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:56:12.215 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:12.215 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:12.216 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:56:12.216 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:12.216 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:56:12.216 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:12.217 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:56:12.217 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:12.217 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:12.217 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:12.218 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:12.218 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:56:12.218 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:12.218 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:12.219 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:56:12.219 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:12.219 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:12.219 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:12.220 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:56:12.220 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:12.220 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:12.220 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:12.221 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:12.221 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:56:12.221 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:12.221 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:12.221 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:56:12.222 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:12.222 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:12.222 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:12.223 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:56:12.223 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:12.223 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:12.223 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:12.223 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:12.224 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:56:12.224 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:12.224 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:12.224 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.815014_179.1", 2022-09-09T22:56:12.225 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:12.225 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:12.225 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:12.225 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.815014Z", 2022-09-09T22:56:12.226 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:12.226 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:12.226 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:12.226 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:56:12.227 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:12.227 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:56:12.227 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:56:12.227 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:56:12.228 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:12.228 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:56:12Z" 2022-09-09T22:56:12.228 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:56:12.229 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.815014_179.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.815014Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:56:12Z" } 2022-09-09T22:56:12.230 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.815014_179.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.815014Z', 'realm_epoch': 2}}] 2022-09-09T22:56:12.231 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:56:12.231 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.815014_179.1'} 2022-09-09T22:56:12.232 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.815014_179.1 2022-09-09T22:56:17.233 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:56:17.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:56:17.235 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:56:17.281 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:56:17.282 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:56:17.350 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:56:17.349+0000 7f2be54dbdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:56:17.352 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:56:17.349+0000 7f2be54dbdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:56:17.382 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:56:17.383 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:56:17.383 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:56:17.383 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:56:17.383 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:56:17.384 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:56:17.384 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:17.384 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:17.384 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:56:17.385 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:17.385 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:56:17.385 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:17.385 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:17.386 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:56:17.386 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:17.386 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:56:17.386 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:17.387 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:56:17.387 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:17.387 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:17.388 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:17.388 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:17.388 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:56:17.388 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:17.389 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:17.389 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:56:17.389 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:17.389 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:17.390 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:17.390 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:56:17.390 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:17.391 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:17.391 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:17.391 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:17.391 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:56:17.392 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:17.392 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:17.392 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:56:17.393 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:17.393 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:17.393 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:17.393 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:56:17.394 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:17.394 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:17.394 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:17.394 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:17.395 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:56:17.395 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:17.395 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:17.395 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763988.086181_195.1", 2022-09-09T22:56:17.396 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:17.396 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:17.396 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:17.396 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.086181Z", 2022-09-09T22:56:17.397 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:17.397 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:17.397 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:17.397 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:56:17.398 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:17.398 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:56:17.398 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:56:17.399 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:56:17.399 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:17.399 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:56:17Z" 2022-09-09T22:56:17.399 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:56:17.401 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763988.086181_195.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:08.086181Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:56:17Z" } 2022-09-09T22:56:17.401 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763988.086181_195.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:08.086181Z', 'realm_epoch': 2}}] 2022-09-09T22:56:17.402 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:56:17.403 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763988.086181_195.1'} 2022-09-09T22:56:17.404 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763988.086181_195.1 2022-09-09T22:56:22.405 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:56:22.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:56:22.407 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:56:22.456 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:56:22.456 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:56:22.515 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:56:22.513+0000 7f716fe75dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:56:22.517 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:56:22.513+0000 7f716fe75dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:56:22.544 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:56:22.544 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:56:22.545 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:56:22.545 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:56:22.545 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:56:22.546 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:56:22.546 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:22.546 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:22.546 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:56:22.547 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:22.547 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:56:22.548 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:22.548 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:22.548 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:56:22.548 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:22.549 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:56:22.549 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:22.549 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:56:22.549 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:22.550 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:22.550 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:22.550 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:22.550 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:56:22.551 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:22.551 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:22.551 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:56:22.551 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:22.552 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:22.552 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:22.552 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:56:22.552 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:22.553 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:22.553 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:22.553 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:22.553 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:56:22.554 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:22.554 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:22.554 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:56:22.555 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:22.555 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:22.557 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:22.557 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:56:22.557 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:22.557 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:22.558 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:22.558 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:22.558 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:56:22.559 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:22.559 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:22.559 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763988.086181_195.1", 2022-09-09T22:56:22.559 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:22.560 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:22.560 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:22.560 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.086181Z", 2022-09-09T22:56:22.560 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:22.561 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:22.561 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:22.561 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:56:22.561 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:22.562 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:56:22.562 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:56:22.563 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:56:22.563 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:22.563 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:56:22Z" 2022-09-09T22:56:22.563 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:56:22.565 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763988.086181_195.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:08.086181Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:56:22Z" } 2022-09-09T22:56:22.565 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763988.086181_195.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:08.086181Z', 'realm_epoch': 2}}] 2022-09-09T22:56:22.566 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:56:22.566 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763988.086181_195.1'} 2022-09-09T22:56:22.567 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763988.086181_195.1 2022-09-09T22:56:27.568 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:56:27.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:56:27.569 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:56:27.619 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:56:27.620 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:56:27.684 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:56:27.681+0000 7f6f53affdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:56:27.685 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:56:27.685+0000 7f6f53affdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:56:27.708 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:56:27.709 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:56:27.709 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:56:27.709 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:56:27.710 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:56:27.710 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:56:27.710 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:27.710 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:27.711 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:56:27.711 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:27.712 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:56:27.712 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:27.712 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:27.713 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:56:27.713 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:27.713 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:56:27.713 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:27.714 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:56:27.714 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:27.715 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:27.715 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:27.715 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:27.715 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:56:27.716 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:27.716 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:27.716 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:56:27.716 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:27.717 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:27.717 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:27.717 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:56:27.717 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:27.718 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:27.718 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:27.719 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:27.719 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:56:27.720 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:27.720 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:27.721 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:56:27.721 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:27.721 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:27.722 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:27.722 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:56:27.722 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:27.722 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:27.723 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:27.723 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:27.723 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:56:27.723 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:27.724 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:27.724 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.905416_183.1", 2022-09-09T22:56:27.724 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:27.725 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:27.725 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:27.726 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.905416Z", 2022-09-09T22:56:27.726 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:27.726 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:27.726 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:27.727 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:56:27.727 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:27.727 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:56:27.728 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:56:27.728 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:56:27.728 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:27.728 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:56:27Z" 2022-09-09T22:56:27.728 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:56:27.730 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.905416_183.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.905416Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:56:27Z" } 2022-09-09T22:56:27.731 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.905416_183.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.905416Z', 'realm_epoch': 2}}] 2022-09-09T22:56:27.732 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:56:27.733 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.905416_183.1'} 2022-09-09T22:56:27.733 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.905416_183.1 2022-09-09T22:56:32.735 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:56:32.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:56:32.736 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:56:32.781 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:56:32.851 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:56:32.852 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:56:32.829+0000 7fa73bccedc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:56:32.853 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:56:32.829+0000 7fa73bccedc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:56:32.867 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:56:32.868 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:56:32.868 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:56:32.868 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:56:32.869 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:56:32.869 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:56:32.869 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:32.870 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:32.870 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:56:32.870 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:32.870 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:56:32.871 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:32.871 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:32.871 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:56:32.871 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:32.872 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:56:32.872 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:32.872 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:56:32.873 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:32.873 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:32.873 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:32.873 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:32.874 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:56:32.874 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:32.874 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:32.874 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:56:32.875 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:32.875 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:32.875 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:32.875 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:56:32.876 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:32.876 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:32.876 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:32.877 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:32.877 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:56:32.877 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:32.877 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:32.878 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:56:32.878 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:32.878 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:32.878 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:32.879 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:56:32.879 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:32.879 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:32.880 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:32.880 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:32.880 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:56:32.880 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:32.881 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:32.881 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.815014_179.1", 2022-09-09T22:56:32.881 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:32.881 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:32.882 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:32.882 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.815014Z", 2022-09-09T22:56:32.882 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:32.883 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:32.883 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:32.883 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:56:32.883 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:32.884 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:56:32.884 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:56:32.884 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:56:32.884 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:32.885 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:56:32Z" 2022-09-09T22:56:32.885 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:56:32.886 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.815014_179.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.815014Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:56:32Z" } 2022-09-09T22:56:32.887 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.815014_179.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.815014Z', 'realm_epoch': 2}}] 2022-09-09T22:56:32.888 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:56:32.888 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.815014_179.1'} 2022-09-09T22:56:32.889 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.815014_179.1 2022-09-09T22:56:37.890 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:56:37.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:56:37.892 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:56:37.937 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:56:37.938 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:56:37.990 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:56:37.988+0000 7fe3e9bdbdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:56:37.992 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:56:37.988+0000 7fe3e9bdbdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:56:38.020 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:56:38.020 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:56:38.020 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:56:38.021 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:56:38.021 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:56:38.021 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:56:38.022 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:38.022 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:38.022 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:56:38.022 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:38.023 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:56:38.023 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:38.023 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:38.023 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:56:38.024 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:38.024 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:56:38.024 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:38.024 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:56:38.025 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:38.025 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:38.025 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:38.025 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:38.026 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:56:38.026 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:38.026 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:38.027 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:56:38.027 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:38.027 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:38.027 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:38.028 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:56:38.028 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:38.028 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:38.028 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:38.029 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:38.029 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:56:38.029 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:38.029 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:38.030 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:56:38.030 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:38.030 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:38.030 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:38.031 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:56:38.031 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:38.031 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:38.031 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:38.032 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:38.032 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:56:38.032 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:38.032 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:38.033 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763988.086181_195.1", 2022-09-09T22:56:38.033 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:38.033 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:38.033 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:38.034 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.086181Z", 2022-09-09T22:56:38.034 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:38.034 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:38.034 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:38.035 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:56:38.035 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:38.035 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:56:38.035 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:56:38.036 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:56:38.036 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:38.036 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:56:37Z" 2022-09-09T22:56:38.036 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:56:38.037 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763988.086181_195.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:08.086181Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:56:37Z" } 2022-09-09T22:56:38.038 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763988.086181_195.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:08.086181Z', 'realm_epoch': 2}}] 2022-09-09T22:56:38.039 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:56:38.039 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763988.086181_195.1'} 2022-09-09T22:56:38.040 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763988.086181_195.1 2022-09-09T22:56:43.041 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:56:43.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:56:43.043 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:56:43.089 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:56:43.090 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:56:43.146 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:56:43.144+0000 7f127f590dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:56:43.148 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:56:43.144+0000 7f127f590dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:56:43.179 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:56:43.179 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:56:43.180 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:56:43.180 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:56:43.181 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:56:43.181 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:56:43.181 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:43.182 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:43.182 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:56:43.182 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:43.182 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:56:43.183 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:43.183 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:43.183 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:56:43.183 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:43.184 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:56:43.184 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:43.184 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:56:43.184 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:43.185 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:43.185 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:43.185 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:43.185 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:56:43.186 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:43.186 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:43.186 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:56:43.186 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:43.187 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:43.187 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:43.187 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:56:43.187 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:43.188 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:43.188 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:43.188 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:43.188 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:56:43.189 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:43.189 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:43.189 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:56:43.189 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:43.190 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:43.190 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:43.190 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:56:43.190 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:43.191 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:43.191 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:43.191 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:43.191 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:56:43.192 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:43.192 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:43.192 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.815014_179.1", 2022-09-09T22:56:43.192 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:43.193 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:43.193 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:43.193 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.815014Z", 2022-09-09T22:56:43.193 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:43.194 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:43.194 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:43.194 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:56:43.194 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:43.195 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:56:43.195 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:56:43.195 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:56:43.196 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:43.196 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:56:43Z" 2022-09-09T22:56:43.196 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:56:43.197 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.815014_179.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.815014Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:56:43Z" } 2022-09-09T22:56:43.198 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.815014_179.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.815014Z', 'realm_epoch': 2}}] 2022-09-09T22:56:43.199 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:56:43.199 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.815014_179.1'} 2022-09-09T22:56:43.199 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.815014_179.1 2022-09-09T22:56:48.201 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:56:48.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:56:48.204 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:56:48.253 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:56:48.254 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:56:48.316 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:56:48.312+0000 7f6cc8270dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:56:48.317 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:56:48.316+0000 7f6cc8270dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:56:48.346 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:56:48.347 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:56:48.347 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:56:48.347 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:56:48.348 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:56:48.348 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:56:48.348 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:48.348 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:48.349 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:56:48.349 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:48.349 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:56:48.350 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:48.350 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:48.350 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:56:48.350 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:48.351 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:56:48.351 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:48.351 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:56:48.351 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:48.352 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:48.352 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:48.352 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:48.352 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:56:48.353 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:48.353 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:48.353 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:56:48.354 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:48.354 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:48.354 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:48.354 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:56:48.355 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:48.355 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:48.355 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:48.356 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:48.356 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:56:48.356 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:48.356 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:48.357 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:56:48.357 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:48.357 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:48.358 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:48.358 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:56:48.358 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:48.359 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:48.359 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:48.360 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:48.360 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:56:48.361 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:48.361 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:48.362 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.815014_179.1", 2022-09-09T22:56:48.362 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:48.363 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:48.363 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:48.363 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.815014Z", 2022-09-09T22:56:48.364 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:48.364 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:48.365 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:48.365 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:56:48.366 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:48.366 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:56:48.366 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:56:48.367 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:56:48.367 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:48.367 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:56:48Z" 2022-09-09T22:56:48.367 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:56:48.369 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.815014_179.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.815014Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:56:48Z" } 2022-09-09T22:56:48.369 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.815014_179.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.815014Z', 'realm_epoch': 2}}] 2022-09-09T22:56:48.370 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:56:48.371 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.815014_179.1'} 2022-09-09T22:56:48.372 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.815014_179.1 2022-09-09T22:56:53.374 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:56:53.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:56:53.376 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:56:53.425 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:56:53.425 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:56:53.498 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:56:53.496+0000 7f8538a97dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:56:53.501 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:56:53.496+0000 7f8538a97dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:56:53.535 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:56:53.535 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:56:53.536 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:56:53.536 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:56:53.537 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:56:53.537 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:56:53.538 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:53.538 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:53.539 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:56:53.539 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:53.540 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:56:53.540 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:53.540 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:53.541 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:56:53.541 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:53.542 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:56:53.542 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:53.543 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:56:53.543 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:53.544 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:53.544 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:53.545 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:53.545 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:56:53.545 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:53.546 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:53.546 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:56:53.547 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:53.547 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:53.548 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:53.548 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:56:53.549 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:53.549 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:53.550 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:53.550 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:53.551 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:56:53.551 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:53.552 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:53.552 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:56:53.552 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:53.553 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:53.553 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:53.554 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:56:53.554 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:53.555 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:53.555 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:53.556 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:53.556 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:56:53.557 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:53.557 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:53.558 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763988.086181_195.1", 2022-09-09T22:56:53.558 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:53.559 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:53.559 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:53.560 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.086181Z", 2022-09-09T22:56:53.560 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:53.561 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:53.561 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:53.562 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:56:53.562 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:53.562 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:56:53.563 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:56:53.563 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:56:53.564 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:53.564 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:56:53Z" 2022-09-09T22:56:53.565 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:56:53.567 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763988.086181_195.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:08.086181Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:56:53Z" } 2022-09-09T22:56:53.568 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763988.086181_195.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:08.086181Z', 'realm_epoch': 2}}] 2022-09-09T22:56:53.569 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:56:53.569 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763988.086181_195.1'} 2022-09-09T22:56:53.570 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763988.086181_195.1 2022-09-09T22:56:58.572 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:56:58.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:56:58.574 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:56:58.625 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:56:58.626 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:56:58.681 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:56:58.676+0000 7fed272dadc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:56:58.682 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:56:58.680+0000 7fed272dadc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:56:58.709 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:56:58.710 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:56:58.710 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:56:58.711 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:56:58.711 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:56:58.711 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:56:58.711 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:58.712 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:58.712 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:56:58.712 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:58.712 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:56:58.713 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:58.713 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:58.713 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:56:58.713 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:58.714 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:56:58.714 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:58.714 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:56:58.715 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:58.715 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:58.715 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:58.715 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:58.716 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:56:58.716 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:58.716 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:58.716 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:56:58.717 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:58.717 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:58.717 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:58.717 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:56:58.718 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:58.718 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:58.718 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:58.718 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:58.719 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:56:58.719 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:58.719 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:58.720 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:56:58.720 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:58.720 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:58.720 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:58.721 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:56:58.721 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:58.721 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:58.721 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:58.722 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:56:58.722 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:56:58.722 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:56:58.723 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:56:58.724 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.815014_179.1", 2022-09-09T22:56:58.724 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:56:58.724 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:56:58.725 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:56:58.725 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.815014Z", 2022-09-09T22:56:58.725 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:56:58.726 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:58.726 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:56:58.726 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:56:58.727 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:58.727 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:56:58.727 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:56:58.728 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:56:58.728 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:56:58.728 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:56:58Z" 2022-09-09T22:56:58.728 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:56:58.730 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.815014_179.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.815014Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:56:58Z" } 2022-09-09T22:56:58.730 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.815014_179.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.815014Z', 'realm_epoch': 2}}] 2022-09-09T22:56:58.731 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:56:58.731 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.815014_179.1'} 2022-09-09T22:56:58.732 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.815014_179.1 2022-09-09T22:57:03.733 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:57:03.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:57:03.736 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:57:03.785 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:57:03.785 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:57:03.852 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:57:03.848+0000 7fd4c5494dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:57:03.854 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:57:03.852+0000 7fd4c5494dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:57:03.884 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:57:03.885 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:57:03.885 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:57:03.885 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:57:03.885 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:57:03.886 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:57:03.886 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:03.886 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:03.887 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:57:03.887 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:03.887 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:57:03.887 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:57:03.888 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:57:03.888 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:57:03.888 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:57:03.888 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:57:03.889 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:57:03.889 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:57:03.889 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:03.889 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:03.890 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:03.890 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:03.890 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:57:03.891 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:57:03.891 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:57:03.891 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:57:03.891 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:57:03.892 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:57:03.892 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:57:03.892 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:57:03.892 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:03.893 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:03.893 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:03.893 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:03.893 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:57:03.894 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:57:03.894 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:57:03.894 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:57:03.895 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:57:03.895 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:57:03.895 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:57:03.895 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:57:03.895 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:03.896 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:03.896 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:03.896 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:03.896 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:57:03.897 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:57:03.897 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:57:03.897 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763988.029534_191.1", 2022-09-09T22:57:03.897 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:57:03.898 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:57:03.898 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:57:03.898 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.029534Z", 2022-09-09T22:57:03.899 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:03.899 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:03.900 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:03.900 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:57:03.901 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:03.901 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:57:03.902 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:57:03.902 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:57:03.903 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:03.903 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:57:03Z" 2022-09-09T22:57:03.904 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:57:03.906 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763988.029534_191.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:08.029534Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:57:03Z" } 2022-09-09T22:57:03.907 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763988.029534_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:08.029534Z', 'realm_epoch': 2}}] 2022-09-09T22:57:03.908 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:57:03.909 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763988.029534_191.1'} 2022-09-09T22:57:03.909 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763988.029534_191.1 2022-09-09T22:57:08.910 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:57:08.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:57:08.912 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:57:08.951 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:57:08.952 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:57:09.013 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:57:09.008+0000 7fa15b7f4dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:57:09.014 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:57:09.012+0000 7fa15b7f4dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:57:09.045 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:57:09.046 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:57:09.046 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:57:09.046 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:57:09.046 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:57:09.047 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:57:09.047 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:09.047 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:09.048 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:57:09.048 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:09.048 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:57:09.048 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:57:09.049 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:57:09.049 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:57:09.049 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:57:09.050 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:57:09.050 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:57:09.050 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:57:09.051 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:09.051 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:09.051 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:09.051 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:09.052 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:57:09.052 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:57:09.052 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:57:09.052 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:57:09.053 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:57:09.053 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:57:09.053 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:57:09.054 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:57:09.054 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:09.054 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:09.054 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:09.055 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:09.055 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:57:09.055 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:57:09.056 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:57:09.056 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:57:09.056 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:57:09.056 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:57:09.057 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:57:09.057 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:57:09.057 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:09.057 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:09.058 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:09.058 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:09.058 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:57:09.059 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:57:09.059 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:57:09.059 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.868518_182.1", 2022-09-09T22:57:09.059 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:57:09.060 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:57:09.060 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:57:09.060 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.868518Z", 2022-09-09T22:57:09.060 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:09.061 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:09.061 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:09.061 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:57:09.061 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:09.062 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:57:09.062 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:57:09.062 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:57:09.063 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:09.063 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:57:09Z" 2022-09-09T22:57:09.063 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:57:09.064 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.868518_182.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.868518Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:57:09Z" } 2022-09-09T22:57:09.065 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.868518_182.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.868518Z', 'realm_epoch': 2}}] 2022-09-09T22:57:09.066 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:57:09.066 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.868518_182.1'} 2022-09-09T22:57:09.067 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.868518_182.1 2022-09-09T22:57:14.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:57:14.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:57:14.070 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:57:14.119 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:57:14.120 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:57:14.176 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:57:14.172+0000 7f63e4a3ddc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:57:14.178 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:57:14.176+0000 7f63e4a3ddc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:57:14.204 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:57:14.205 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:57:14.205 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:57:14.205 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:57:14.206 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:57:14.206 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:57:14.206 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:14.207 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:14.207 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:57:14.207 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:14.207 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:57:14.208 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:57:14.208 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:57:14.208 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:57:14.208 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:57:14.209 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:57:14.209 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:57:14.209 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:57:14.209 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:14.210 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:14.210 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:14.210 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:14.211 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:57:14.211 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:57:14.211 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:57:14.211 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:57:14.212 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:57:14.212 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:57:14.212 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:57:14.212 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:57:14.213 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:14.213 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:14.213 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:14.213 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:14.214 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:57:14.214 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:57:14.214 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:57:14.214 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:57:14.215 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:57:14.215 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:57:14.215 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:57:14.215 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:57:14.216 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:14.216 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:14.216 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:14.216 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:14.217 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:57:14.217 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:57:14.217 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:57:14.217 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763987.815014_179.1", 2022-09-09T22:57:14.218 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:57:14.218 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:57:14.218 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:57:14.218 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.815014Z", 2022-09-09T22:57:14.219 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:14.219 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:14.219 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:14.219 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:57:14.220 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:14.220 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:57:14.220 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:57:14.220 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:57:14.221 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:14.221 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:57:14Z" 2022-09-09T22:57:14.221 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:57:14.223 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763987.815014_179.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:07.815014Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:57:14Z" } 2022-09-09T22:57:14.223 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763987.815014_179.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:07.815014Z', 'realm_epoch': 2}}] 2022-09-09T22:57:14.224 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:57:14.224 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763987.815014_179.1'} 2022-09-09T22:57:14.225 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662763990.796002_230.1 target=1_1662763987.815014_179.1 2022-09-09T22:57:19.226 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:57:19.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:57:19.228 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:57:19.276 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:57:19.277 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:57:19.337 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:57:19.332+0000 7eff1eaf4dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:57:19.339 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:57:19.336+0000 7eff1eaf4dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:57:19.368 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:57:19.369 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:57:19.369 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:57:19.369 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:57:19.369 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:57:19.370 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:57:19.370 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:19.370 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:19.371 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:57:19.371 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:19.371 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:57:19.371 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:57:19.372 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:57:19.372 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:57:19.372 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:57:19.372 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:57:19.373 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:57:19.373 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:57:19.373 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:19.373 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:19.374 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:19.374 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:19.374 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:57:19.375 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:57:19.375 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:57:19.375 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:57:19.375 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:57:19.376 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:57:19.376 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:57:19.376 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:57:19.376 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:19.377 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:19.377 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:19.377 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:19.377 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:57:19.378 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:57:19.378 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:57:19.378 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:57:19.379 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:57:19.379 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:57:19.379 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:57:19.379 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:57:19.380 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:19.380 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:19.380 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:19.380 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:19.381 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:57:19.381 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:57:19.381 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:57:19.381 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763990.796002_230.1", 2022-09-09T22:57:19.382 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:57:19.382 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:57:19.382 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:57:19.383 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:10.796002Z", 2022-09-09T22:57:19.383 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:19.383 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:19.383 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:19.384 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:57:19.384 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:19.384 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:57:19.385 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:57:19.385 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:57:19.385 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:19.385 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:57:19Z" 2022-09-09T22:57:19.386 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:57:19.387 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763990.796002_230.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:10.796002Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:57:19Z" } 2022-09-09T22:57:19.388 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763990.796002_230.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:10.796002Z', 'realm_epoch': 2}}] 2022-09-09T22:57:19.388 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:57:19.389 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:57:19.389 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-09T22:57:19.390 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-09T22:57:19.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:57:19.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:57:19.391 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:57:19.440 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:57:19.440 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:57:19.493 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:57:19.491+0000 7f3b64cbfdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:57:19.495 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:57:19.491+0000 7f3b64cbfdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:57:19.522 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:57:19.523 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T22:57:19.523 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T22:57:19.523 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T22:57:19.524 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T22:57:19.524 INFO:teuthology.orchestra.run.smithi005.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:57:19.524 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:57:19.524 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:57:19.525 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T22:57:19.525 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:57:19.525 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T22:57:19.526 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:57:19.526 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:57:19.526 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:57:19.526 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:57:19.527 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 1, 2022-09-09T22:57:19.527 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:57:19.528 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:57:19.528 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:57:19.528 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:57:19.529 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:57:19.529 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:57:19.529 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T22:57:19.530 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:57:19.530 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:57:19.530 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:57:19.531 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:57:19.531 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:57:19.531 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:57:19.532 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:57:19.532 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:57:19.533 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:57:19.533 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:57:19.534 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:57:19.534 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T22:57:19.535 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:57:19.535 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:57:19.536 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:57:19.536 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:57:19.537 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:57:19.538 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:57:19.538 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:57:19.539 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:57:19.539 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:57:19.540 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:57:19.540 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:57:19.540 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T22:57:19.541 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:57:19.541 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:57:19.541 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662763990.796002_230.1", 2022-09-09T22:57:19.541 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:57:19.542 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:57:19.542 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:57:19.543 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:10.796002Z", 2022-09-09T22:57:19.543 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:57:19.543 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:57:19.544 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:57:19.544 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:57:19.544 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:57:19.545 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T22:57:19.545 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 1, 2022-09-09T22:57:19.545 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 1 2022-09-09T22:57:19.545 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:57:19.546 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T22:57:19Z" 2022-09-09T22:57:19.546 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:57:19.547 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763990.796002_230.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:10.796002Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:57:19Z" } 2022-09-09T22:57:19.548 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763990.796002_230.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:10.796002Z', 'realm_epoch': 2}}] 2022-09-09T22:57:19.548 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:57:19.549 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:57:19.550 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-09T22:57:19.550 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-09T22:57:19.551 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:57:19.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:57:19.552 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:57:19.601 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:57:19.601 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:57:19.673 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:57:19.668+0000 7f8a8665adc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:57:19.675 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:57:19.672+0000 7f8a8665adc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:57:19.702 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:57:19.702 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:57:19.702 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:57:19.703 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:57:19.703 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:57:19.703 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:57:19.704 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:19.704 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:19.704 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:57:19.704 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:19.705 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:57:19.705 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:57:19.705 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:57:19.705 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:57:19.706 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:57:19.707 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:57:19.707 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:57:19.708 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:57:19.708 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:19.709 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:19.709 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:19.710 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:19.710 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:57:19.711 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:57:19.711 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:57:19.712 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:57:19.713 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:57:19.713 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:57:19.714 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:57:19.714 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:57:19.715 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:19.715 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:19.716 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:19.716 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:19.717 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:57:19.717 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:57:19.718 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:57:19.718 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:57:19.719 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:57:19.719 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:57:19.720 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:57:19.720 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:57:19.721 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:19.721 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:19.722 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:19.723 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:19.723 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:57:19.724 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:57:19.724 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:57:19.725 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662763990.796002_230.1", 2022-09-09T22:57:19.725 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:57:19.726 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:57:19.726 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:57:19.727 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:10.796002Z", 2022-09-09T22:57:19.728 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:19.728 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:19.729 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:19.729 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:57:19.730 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:19.730 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:57:19.731 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:57:19.731 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:57:19.732 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:19.732 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:57:19Z" 2022-09-09T22:57:19.733 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:57:19.735 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662763990.796002_230.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:53:10.796002Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:57:19Z" } 2022-09-09T22:57:19.736 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662763990.796002_230.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:53:10.796002Z', 'realm_epoch': 2}}] 2022-09-09T22:57:19.737 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:57:19.737 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662763990.796002_230.1'} 2022-09-09T22:57:19.738 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-09T22:57:19.739 DEBUG:boto:path=/jkcgbj-36/ 2022-09-09T22:57:19.739 DEBUG:boto:auth_path=/jkcgbj-36/ 2022-09-09T22:57:19.740 DEBUG:boto:Method: HEAD 2022-09-09T22:57:19.741 DEBUG:boto:Path: /jkcgbj-36/ 2022-09-09T22:57:19.741 DEBUG:boto:Data: 2022-09-09T22:57:19.742 DEBUG:boto:Headers: {} 2022-09-09T22:57:19.742 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:57:19.743 DEBUG:boto:Port: 8000 2022-09-09T22:57:19.743 DEBUG:boto:Params: {} 2022-09-09T22:57:19.745 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T22:57:19.745 DEBUG:boto:Token: None 2022-09-09T22:57:19.747 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:57:19 GMT /jkcgbj-36/ 2022-09-09T22:57:19.748 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:zUEABKEDn/wHlKAtyaK4q5YWVh8= 2022-09-09T22:57:19.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:57:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:zUEABKEDn/wHlKAtyaK4q5YWVh8=', 'Content-Length': '0'} 2022-09-09T22:57:19.759 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000097ac8aa622d5aec0-00631bc4cf-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:57:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:57:19.761 DEBUG:boto:path=/jkcgbj-36/foo 2022-09-09T22:57:19.762 DEBUG:boto:auth_path=/jkcgbj-36/foo 2022-09-09T22:57:19.763 DEBUG:boto:Method: PUT 2022-09-09T22:57:19.765 DEBUG:boto:Path: /jkcgbj-36/foo 2022-09-09T22:57:19.766 DEBUG:boto:Data: 2022-09-09T22:57:19.767 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-09T22:57:19.768 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:57:19.768 DEBUG:boto:Port: 8000 2022-09-09T22:57:19.769 DEBUG:boto:Params: {} 2022-09-09T22:57:19.769 DEBUG:boto:Token: None 2022-09-09T22:57:19.770 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Fri, 09 Sep 2022 22:57:19 GMT /jkcgbj-36/foo 2022-09-09T22:57:19.771 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:xCDPJRW1TaSztINue73r5gFbxx0= 2022-09-09T22:57:19.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:57:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:xCDPJRW1TaSztINue73r5gFbxx0='} 2022-09-09T22:57:19.792 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d674819d9c047c7d-00631bc4cf-10ff-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:57:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:57:19.793 DEBUG:boto:path=/jkcgbj-36/ 2022-09-09T22:57:19.794 DEBUG:boto:auth_path=/jkcgbj-36/ 2022-09-09T22:57:19.794 DEBUG:boto:Method: DELETE 2022-09-09T22:57:19.795 DEBUG:boto:Path: /jkcgbj-36/ 2022-09-09T22:57:19.795 DEBUG:boto:Data: 2022-09-09T22:57:19.796 DEBUG:boto:Headers: {} 2022-09-09T22:57:19.796 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:57:19.797 DEBUG:boto:Port: 8000 2022-09-09T22:57:19.797 DEBUG:boto:Params: {} 2022-09-09T22:57:19.798 DEBUG:boto:Token: None 2022-09-09T22:57:19.799 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:57:19 GMT /jkcgbj-36/ 2022-09-09T22:57:19.799 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LtyCYPFCGKvdnSeUTgQ9lp3NN/I= 2022-09-09T22:57:19.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:57:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LtyCYPFCGKvdnSeUTgQ9lp3NN/I=', 'Content-Length': '0'} 2022-09-09T22:57:19.816 DEBUG:boto:Response headers: [('Content-Length', '234'), ('x-amz-request-id', 'tx000000d5cd74b5f022dae-00631bc4cf-10ff-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:57:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:57:19.818 DEBUG:boto:path=/jkcgbj-37/ 2022-09-09T22:57:19.818 DEBUG:boto:auth_path=/jkcgbj-37/ 2022-09-09T22:57:19.819 DEBUG:boto:Method: HEAD 2022-09-09T22:57:19.820 DEBUG:boto:Path: /jkcgbj-37/ 2022-09-09T22:57:19.820 DEBUG:boto:Data: 2022-09-09T22:57:19.821 DEBUG:boto:Headers: {} 2022-09-09T22:57:19.821 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:57:19.822 DEBUG:boto:Port: 8000 2022-09-09T22:57:19.822 DEBUG:boto:Params: {} 2022-09-09T22:57:19.823 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T22:57:19.823 DEBUG:boto:Token: None 2022-09-09T22:57:19.824 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:57:19 GMT /jkcgbj-37/ 2022-09-09T22:57:19.825 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Fs4nHIR9KZ0MMD62HgkERJXdmgo= 2022-09-09T22:57:19.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:57:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Fs4nHIR9KZ0MMD62HgkERJXdmgo=', 'Content-Length': '0'} 2022-09-09T22:57:19.831 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d6a02c52011f88a-00631bc4cf-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:57:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:57:19.832 DEBUG:boto:path=/jkcgbj-37/foo 2022-09-09T22:57:19.832 DEBUG:boto:auth_path=/jkcgbj-37/foo 2022-09-09T22:57:19.833 DEBUG:boto:Method: PUT 2022-09-09T22:57:19.833 DEBUG:boto:Path: /jkcgbj-37/foo 2022-09-09T22:57:19.834 DEBUG:boto:Data: 2022-09-09T22:57:19.834 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-09T22:57:19.835 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:57:19.835 DEBUG:boto:Port: 8000 2022-09-09T22:57:19.836 DEBUG:boto:Params: {} 2022-09-09T22:57:19.836 DEBUG:boto:Token: None 2022-09-09T22:57:19.837 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Fri, 09 Sep 2022 22:57:19 GMT /jkcgbj-37/foo 2022-09-09T22:57:19.838 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Kh2gjxXj3QLzvDUFU6Y5mG5p5tA= 2022-09-09T22:57:19.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:57:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Kh2gjxXj3QLzvDUFU6Y5mG5p5tA='} 2022-09-09T22:57:19.864 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001b9c0293d06ad8c0-00631bc4cf-10e8-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:57:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:57:19.865 DEBUG:boto:path=/jkcgbj-37/ 2022-09-09T22:57:19.865 DEBUG:boto:auth_path=/jkcgbj-37/ 2022-09-09T22:57:19.866 DEBUG:boto:Method: DELETE 2022-09-09T22:57:19.866 DEBUG:boto:Path: /jkcgbj-37/ 2022-09-09T22:57:19.867 DEBUG:boto:Data: 2022-09-09T22:57:19.867 DEBUG:boto:Headers: {} 2022-09-09T22:57:19.868 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:57:19.868 DEBUG:boto:Port: 8000 2022-09-09T22:57:19.869 DEBUG:boto:Params: {} 2022-09-09T22:57:19.869 DEBUG:boto:Token: None 2022-09-09T22:57:19.870 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:57:19 GMT /jkcgbj-37/ 2022-09-09T22:57:19.870 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:29T4Vt83GKwFmdYXsHYG3Sq6Xfc= 2022-09-09T22:57:19.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:57:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:29T4Vt83GKwFmdYXsHYG3Sq6Xfc=', 'Content-Length': '0'} 2022-09-09T22:57:19.890 DEBUG:boto:Response headers: [('Content-Length', '234'), ('x-amz-request-id', 'tx00000403f2fd667bf9e5a-00631bc4cf-10e8-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:57:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:57:19.891 DEBUG:boto:path=/jkcgbj-38/ 2022-09-09T22:57:19.892 DEBUG:boto:auth_path=/jkcgbj-38/ 2022-09-09T22:57:19.892 DEBUG:boto:Method: HEAD 2022-09-09T22:57:19.893 DEBUG:boto:Path: /jkcgbj-38/ 2022-09-09T22:57:19.893 DEBUG:boto:Data: 2022-09-09T22:57:19.894 DEBUG:boto:Headers: {} 2022-09-09T22:57:19.894 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:57:19.895 DEBUG:boto:Port: 8001 2022-09-09T22:57:19.895 DEBUG:boto:Params: {} 2022-09-09T22:57:19.896 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-09T22:57:19.897 DEBUG:boto:Token: None 2022-09-09T22:57:19.897 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:57:19 GMT /jkcgbj-38/ 2022-09-09T22:57:19.898 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ENj2SartQECh9WJJBO69Hs+k8+8= 2022-09-09T22:57:19.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:57:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ENj2SartQECh9WJJBO69Hs+k8+8=', 'Content-Length': '0'} 2022-09-09T22:57:19.904 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca82d0097a485f75-00631bc4cf-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:57:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:57:19.905 DEBUG:boto:path=/jkcgbj-38/foo 2022-09-09T22:57:19.905 DEBUG:boto:auth_path=/jkcgbj-38/foo 2022-09-09T22:57:19.906 DEBUG:boto:Method: PUT 2022-09-09T22:57:19.907 DEBUG:boto:Path: /jkcgbj-38/foo 2022-09-09T22:57:19.907 DEBUG:boto:Data: 2022-09-09T22:57:19.908 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-09T22:57:19.908 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:57:19.909 DEBUG:boto:Port: 8001 2022-09-09T22:57:19.909 DEBUG:boto:Params: {} 2022-09-09T22:57:19.910 DEBUG:boto:Token: None 2022-09-09T22:57:19.910 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Fri, 09 Sep 2022 22:57:19 GMT /jkcgbj-38/foo 2022-09-09T22:57:19.911 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6edsIHeRjvF1+3mmeRAtRYBxX/c= 2022-09-09T22:57:19.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:57:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6edsIHeRjvF1+3mmeRAtRYBxX/c='} 2022-09-09T22:57:19.935 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fd8465e3886cb470-00631bc4cf-1103-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:57:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:57:19.936 DEBUG:boto:path=/jkcgbj-38/ 2022-09-09T22:57:19.936 DEBUG:boto:auth_path=/jkcgbj-38/ 2022-09-09T22:57:19.937 DEBUG:boto:Method: DELETE 2022-09-09T22:57:19.937 DEBUG:boto:Path: /jkcgbj-38/ 2022-09-09T22:57:19.938 DEBUG:boto:Data: 2022-09-09T22:57:19.938 DEBUG:boto:Headers: {} 2022-09-09T22:57:19.939 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:57:19.939 DEBUG:boto:Port: 8001 2022-09-09T22:57:19.940 DEBUG:boto:Params: {} 2022-09-09T22:57:19.940 DEBUG:boto:Token: None 2022-09-09T22:57:19.941 DEBUG:boto:StringToSign: DELETE Fri, 09 Sep 2022 22:57:19 GMT /jkcgbj-38/ 2022-09-09T22:57:19.942 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Pj4cXNfwOMLVwTj5hk9QvWh6mEE= 2022-09-09T22:57:19.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:57:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Pj4cXNfwOMLVwTj5hk9QvWh6mEE=', 'Content-Length': '0'} 2022-09-09T22:57:19.963 DEBUG:boto:Response headers: [('Content-Length', '234'), ('x-amz-request-id', 'tx0000043b8b4fc9802d28c-00631bc4cf-1103-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:57:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:57:19.965 DEBUG:boto:path=/jkcgbj-36/ 2022-09-09T22:57:19.965 DEBUG:boto:auth_path=/jkcgbj-36/ 2022-09-09T22:57:19.966 DEBUG:boto:Method: HEAD 2022-09-09T22:57:19.967 DEBUG:boto:Path: /jkcgbj-36/ 2022-09-09T22:57:19.968 DEBUG:boto:Data: 2022-09-09T22:57:19.969 DEBUG:boto:Headers: {} 2022-09-09T22:57:19.969 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:57:19.970 DEBUG:boto:Port: 8000 2022-09-09T22:57:19.970 DEBUG:boto:Params: {} 2022-09-09T22:57:19.971 DEBUG:boto:Token: None 2022-09-09T22:57:19.971 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:57:19 GMT /jkcgbj-36/ 2022-09-09T22:57:19.972 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:zUEABKEDn/wHlKAtyaK4q5YWVh8= 2022-09-09T22:57:19.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:57:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:zUEABKEDn/wHlKAtyaK4q5YWVh8=', 'Content-Length': '0'} 2022-09-09T22:57:19.977 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a580cb953f6acf05-00631bc4cf-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:57:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:57:19.978 DEBUG:boto:path=/jkcgbj-37/ 2022-09-09T22:57:19.978 DEBUG:boto:auth_path=/jkcgbj-37/ 2022-09-09T22:57:19.979 DEBUG:boto:Method: HEAD 2022-09-09T22:57:19.979 DEBUG:boto:Path: /jkcgbj-37/ 2022-09-09T22:57:19.980 DEBUG:boto:Data: 2022-09-09T22:57:19.980 DEBUG:boto:Headers: {} 2022-09-09T22:57:19.981 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:57:19.981 DEBUG:boto:Port: 8000 2022-09-09T22:57:19.982 DEBUG:boto:Params: {} 2022-09-09T22:57:19.983 DEBUG:boto:Token: None 2022-09-09T22:57:19.983 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:57:19 GMT /jkcgbj-37/ 2022-09-09T22:57:19.984 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Fs4nHIR9KZ0MMD62HgkERJXdmgo= 2022-09-09T22:57:19.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:57:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Fs4nHIR9KZ0MMD62HgkERJXdmgo=', 'Content-Length': '0'} 2022-09-09T22:57:19.990 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ef998caad566d1a-00631bc4cf-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:57:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:57:19.990 DEBUG:boto:path=/jkcgbj-38/ 2022-09-09T22:57:19.991 DEBUG:boto:auth_path=/jkcgbj-38/ 2022-09-09T22:57:19.992 DEBUG:boto:Method: HEAD 2022-09-09T22:57:19.992 DEBUG:boto:Path: /jkcgbj-38/ 2022-09-09T22:57:19.993 DEBUG:boto:Data: 2022-09-09T22:57:19.993 DEBUG:boto:Headers: {} 2022-09-09T22:57:19.993 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:57:19.994 DEBUG:boto:Port: 8000 2022-09-09T22:57:19.994 DEBUG:boto:Params: {} 2022-09-09T22:57:19.995 DEBUG:boto:Token: None 2022-09-09T22:57:19.996 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:57:19 GMT /jkcgbj-38/ 2022-09-09T22:57:19.996 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ENj2SartQECh9WJJBO69Hs+k8+8= 2022-09-09T22:57:19.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:57:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ENj2SartQECh9WJJBO69Hs+k8+8=', 'Content-Length': '0'} 2022-09-09T22:57:20.003 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002bd0d25405fe7885-00631bc4cf-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:57:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:57:20.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... ok 2022-09-09T22:57:20.006 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jkcgbj-39 2022-09-09T22:57:20.007 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:57:20.007 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:57:20.008 DEBUG:boto:Method: PUT 2022-09-09T22:57:20.008 DEBUG:boto:Path: /jkcgbj-39/ 2022-09-09T22:57:20.009 DEBUG:boto:Data: 2022-09-09T22:57:20.009 DEBUG:boto:Headers: {} 2022-09-09T22:57:20.010 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:57:20.011 DEBUG:boto:Port: 8000 2022-09-09T22:57:20.011 DEBUG:boto:Params: {} 2022-09-09T22:57:20.012 DEBUG:boto:Token: None 2022-09-09T22:57:20.012 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:57:20 GMT /jkcgbj-39/ 2022-09-09T22:57:20.013 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:/VqtMWE0Np0Zi4LkIzq4oAqt+AE= 2022-09-09T22:57:20.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:57:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:/VqtMWE0Np0Zi4LkIzq4oAqt+AE=', 'Content-Length': '0'} 2022-09-09T22:57:20.043 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cdf83ad70fe06ae8-00631bc4d0-10ff-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:57:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:57:20.043 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jkcgbj-40 2022-09-09T22:57:20.044 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:57:20.045 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:57:20.045 DEBUG:boto:Method: PUT 2022-09-09T22:57:20.046 DEBUG:boto:Path: /jkcgbj-40/ 2022-09-09T22:57:20.046 DEBUG:boto:Data: 2022-09-09T22:57:20.047 DEBUG:boto:Headers: {} 2022-09-09T22:57:20.048 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:57:20.048 DEBUG:boto:Port: 8000 2022-09-09T22:57:20.049 DEBUG:boto:Params: {} 2022-09-09T22:57:20.049 DEBUG:boto:Token: None 2022-09-09T22:57:20.050 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:57:20 GMT /jkcgbj-40/ 2022-09-09T22:57:20.051 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6wQ0dlOfipry6GXSlNY3sy2wR38= 2022-09-09T22:57:20.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:57:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6wQ0dlOfipry6GXSlNY3sy2wR38=', 'Content-Length': '0'} 2022-09-09T22:57:20.117 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000066d46f3f6e575893-00631bc4d0-10e8-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:57:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:57:20.118 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jkcgbj-41 2022-09-09T22:57:20.119 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:57:20.119 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:57:20.120 DEBUG:boto:Method: PUT 2022-09-09T22:57:20.120 DEBUG:boto:Path: /jkcgbj-41/ 2022-09-09T22:57:20.121 DEBUG:boto:Data: 2022-09-09T22:57:20.121 DEBUG:boto:Headers: {} 2022-09-09T22:57:20.122 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:57:20.122 DEBUG:boto:Port: 8001 2022-09-09T22:57:20.123 DEBUG:boto:Params: {} 2022-09-09T22:57:20.124 DEBUG:boto:Token: None 2022-09-09T22:57:20.125 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:57:20 GMT /jkcgbj-41/ 2022-09-09T22:57:20.125 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:/46QA+ISiiBWZOoarcu84uzehc4= 2022-09-09T22:57:20.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:57:20 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:/46QA+ISiiBWZOoarcu84uzehc4=', 'Content-Length': '0'} 2022-09-09T22:57:20.185 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b6c762f55d490423-00631bc4d0-1103-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:57:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:57:20.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:57:20.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:57:20.187 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:57:20.226 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:57:20.227 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:57:20.270 INFO:teuthology.orchestra.run.smithi005.stderr:No --period given, using current period=503b1d61-4513-48c8-a308-17b76219610c 2022-09-09T22:57:20.304 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T22:57:20.305 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:57:20.305 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:57:20.305 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:28:06.146147Z" 2022-09-09T22:57:20.306 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:57:20.306 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:57:20.306 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:57:20.306 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:57:20.307 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:57:20.307 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:57:20.307 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:57:20.307 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:30:37.757014Z" 2022-09-09T22:57:20.308 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:57:20.308 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:57:20.308 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662764240.168869_243.1", 2022-09-09T22:57:20.309 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:57:20.168869Z" 2022-09-09T22:57:20.309 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:57:20.309 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T22:57:20.310 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662764240.168869_243.1'} 2022-09-09T22:57:20.311 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-09T22:57:20.312 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:57:20.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:57:20.313 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:57:20.356 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:57:20.357 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:57:20.424 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:57:20.420+0000 7fed1b2c0dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:57:20.426 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:57:20.424+0000 7fed1b2c0dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:57:20.456 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:57:20.456 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:57:20.457 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:57:20.457 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:57:20.457 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:57:20.457 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:57:20.459 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:20.459 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:20.459 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:57:20.459 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:20.460 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:57:20.460 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:57:20.460 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:57:20.460 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:57:20.461 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:57:20.461 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:57:20.461 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:57:20.462 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:57:20.462 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:20.463 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:20.463 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:20.463 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:20.464 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:57:20.464 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:57:20.464 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:57:20.464 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:57:20.465 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:57:20.465 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:57:20.465 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:57:20.465 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:57:20.466 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:20.466 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:20.466 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:20.467 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:20.467 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:57:20.467 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:57:20.468 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:57:20.468 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:57:20.468 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:57:20.468 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:57:20.469 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:57:20.469 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:57:20.469 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:20.469 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:20.470 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:20.470 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:20.471 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:57:20.471 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:57:20.471 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:57:20.471 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764240.168869_243.1", 2022-09-09T22:57:20.472 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:57:20.472 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:57:20.472 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:57:20.473 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:57:20.168869Z", 2022-09-09T22:57:20.473 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:20.473 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:20.473 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:20.474 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:57:20.474 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:20.474 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:57:20.474 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:57:20.475 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:57:20.475 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:20.475 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:57:20Z" 2022-09-09T22:57:20.476 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:57:20.477 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764240.168869_243.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:57:20.168869Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:57:20Z" } 2022-09-09T22:57:20.478 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764240.168869_243.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:57:20.168869Z', 'realm_epoch': 2}}] 2022-09-09T22:57:20.479 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662764240.168869_243.1'} 2022-09-09T22:57:20.479 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662764240.168869_243.1'} 2022-09-09T22:57:20.480 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-09T22:57:20.480 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-09T22:57:20.481 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:57:20.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:57:20.482 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:57:20.532 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:57:20.532 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:57:20.585 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:57:20.583+0000 7f9c55f4cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:57:20.587 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:57:20.583+0000 7f9c55f4cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:57:20.613 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:57:20.613 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T22:57:20.614 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T22:57:20.614 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T22:57:20.614 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T22:57:20.615 INFO:teuthology.orchestra.run.smithi005.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:57:20.615 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:57:20.615 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:57:20.616 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T22:57:20.616 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:57:20.616 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T22:57:20.616 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:57:20.617 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:57:20.617 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:57:20.617 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:57:20.618 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 1, 2022-09-09T22:57:20.618 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:57:20.618 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:57:20.618 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:57:20.619 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:57:20.619 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:57:20.619 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:57:20.620 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T22:57:20.620 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:57:20.620 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:57:20.620 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:57:20.621 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:57:20.621 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:57:20.621 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:57:20.621 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:57:20.622 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:57:20.622 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:57:20.622 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:57:20.623 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:57:20.623 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T22:57:20.623 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:57:20.623 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:57:20.624 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:57:20.624 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:57:20.624 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:57:20.625 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:57:20.625 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:57:20.625 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:57:20.625 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:57:20.626 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:57:20.626 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:57:20.626 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T22:57:20.627 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:57:20.627 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:57:20.627 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662764240.029920_231.1", 2022-09-09T22:57:20.627 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:57:20.628 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:57:20.628 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:57:20.628 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:57:20.029920Z", 2022-09-09T22:57:20.629 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:57:20.629 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:57:20.629 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:57:20.629 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:57:20.630 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:57:20.630 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T22:57:20.630 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 1, 2022-09-09T22:57:20.631 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 1 2022-09-09T22:57:20.631 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:57:20.631 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T22:57:20Z" 2022-09-09T22:57:20.631 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:57:20.633 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764240.029920_231.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:57:20.029920Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:57:20Z" } 2022-09-09T22:57:20.633 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764240.029920_231.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:57:20.029920Z', 'realm_epoch': 2}}] 2022-09-09T22:57:20.634 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662764240.168869_243.1'} 2022-09-09T22:57:20.634 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662764240.029920_231.1'} 2022-09-09T22:57:20.635 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1662764240.168869_243.1 target=1_1662764240.029920_231.1 2022-09-09T22:57:25.637 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:57:25.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:57:25.638 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:57:25.684 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:57:25.685 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:57:25.734 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:57:25.731+0000 7f78ad422dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:57:25.736 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:57:25.735+0000 7f78ad422dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:57:25.763 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:57:25.763 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T22:57:25.763 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T22:57:25.764 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T22:57:25.764 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T22:57:25.764 INFO:teuthology.orchestra.run.smithi005.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:57:25.764 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:57:25.765 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:57:25.765 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T22:57:25.765 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:57:25.766 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T22:57:25.766 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:57:25.766 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:57:25.766 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:57:25.767 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:57:25.767 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 1, 2022-09-09T22:57:25.767 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:57:25.767 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:57:25.768 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:57:25.768 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:57:25.768 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:57:25.769 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:57:25.769 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T22:57:25.769 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:57:25.769 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:57:25.770 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:57:25.770 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:57:25.770 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:57:25.771 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:57:25.771 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:57:25.771 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:57:25.771 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:57:25.772 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:57:25.772 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:57:25.772 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T22:57:25.773 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:57:25.773 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:57:25.773 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:57:25.773 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:57:25.774 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:57:25.774 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:57:25.774 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:57:25.774 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:57:25.775 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:57:25.775 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:57:25.775 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:57:25.776 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T22:57:25.776 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:57:25.776 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:57:25.776 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662764240.168869_243.1", 2022-09-09T22:57:25.777 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:57:25.777 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:57:25.777 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:57:25.777 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:57:20.168869Z", 2022-09-09T22:57:25.778 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:57:25.778 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:57:25.778 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:57:25.779 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:57:25.779 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:57:25.779 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T22:57:25.779 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 1, 2022-09-09T22:57:25.780 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 1 2022-09-09T22:57:25.780 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:57:25.780 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T22:57:25Z" 2022-09-09T22:57:25.780 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:57:25.782 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764240.168869_243.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:57:20.168869Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:57:25Z" } 2022-09-09T22:57:25.783 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764240.168869_243.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:57:20.168869Z', 'realm_epoch': 2}}] 2022-09-09T22:57:25.783 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662764240.168869_243.1'} 2022-09-09T22:57:25.784 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662764240.168869_243.1'} 2022-09-09T22:57:25.784 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-09T22:57:25.785 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-09T22:57:25.785 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:57:25.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:57:25.787 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:57:25.834 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:57:25.834 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:57:25.898 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:57:25.896+0000 7f27ae313dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:57:25.900 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:57:25.896+0000 7f27ae313dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:57:25.927 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:57:25.927 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:57:25.927 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:57:25.928 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:57:25.928 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:57:25.928 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:57:25.928 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:25.929 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:25.929 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:57:25.929 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:25.930 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:57:25.930 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:57:25.930 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:57:25.931 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:57:25.931 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:57:25.931 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:57:25.931 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:57:25.932 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:57:25.932 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:25.932 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:25.932 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:25.933 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:25.933 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:57:25.933 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:57:25.933 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:57:25.934 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:57:25.934 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:57:25.934 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:57:25.934 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:57:25.935 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:57:25.935 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:25.935 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:25.935 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:25.936 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:25.936 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:57:25.936 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:57:25.936 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:57:25.937 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:57:25.937 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:57:25.937 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:57:25.937 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:57:25.938 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:57:25.938 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:25.938 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:25.938 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:25.939 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:25.939 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:57:25.939 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:57:25.939 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:57:25.940 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764240.168869_243.1", 2022-09-09T22:57:25.940 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:57:25.940 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:57:25.940 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:57:25.941 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:57:20.168869Z", 2022-09-09T22:57:25.941 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:57:25.941 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:25.941 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:25.942 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:57:25.942 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:25.942 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:57:25.942 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:57:25.943 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:57:25.943 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:25.943 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:57:25Z" 2022-09-09T22:57:25.943 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:57:25.945 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "503b1d61-4513-48c8-a308-17b76219610c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764240.168869_243.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:57:20.168869Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:57:25Z" } 2022-09-09T22:57:25.945 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764240.168869_243.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:57:20.168869Z', 'realm_epoch': 2}}] 2022-09-09T22:57:25.946 DEBUG:rgw_multi.tests:log_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662764240.168869_243.1'} 2022-09-09T22:57:25.946 DEBUG:rgw_multi.tests:sync_status={0: '1_1662762486.146147_45.1', 1: '', 2: '1_1662762637.757014_85.1', 3: '1_1662764240.168869_243.1'} 2022-09-09T22:57:25.947 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-09T22:57:25.948 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2022-09-09T22:57:25.948 INFO:teuthology.orchestra.run:waiting for 300 2022-09-09T22:57:25.950 INFO:tasks.rgw.c1.client.1.smithi005.stdout:2022-09-09T22:57:25.947+0000 7f56a2744700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 21160) UID: 0 2022-09-09T22:57:25.950 INFO:tasks.rgw.c1.client.1.smithi005.stdout:2022-09-09T22:57:25.947+0000 7f56a5fa2dc0 -1 shutting down 2022-09-09T22:57:29.810 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2022-09-09T22:57:32.051 INFO:tasks.rgw.c1.client.1:Stopped 2022-09-09T22:57:32.052 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:57:32.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:57:32.053 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-09T22:57:32.102 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:57:32.103 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:57:32.151 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:57:32.147+0000 7fe914452dc0 0 NOTICE: overriding master zone: 4b05608c-7197-4e9b-bd29-7989607fa277 2022-09-09T22:57:32.152 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:57:32.147+0000 7fe914452dc0 0 failed reading obj info from .rgw.root:zone_info.4b05608c-7197-4e9b-bd29-7989607fa277: (2) No such file or directory 2022-09-09T22:57:32.152 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:57:32.147+0000 7fe914452dc0 0 WARNING: could not read zone params for zone id=4b05608c-7197-4e9b-bd29-7989607fa277 name=test-zone1 2022-09-09T22:57:32.153 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:57:32.147+0000 7fe914452dc0 0 failed reading obj info from .rgw.root:zone_info.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5: (2) No such file or directory 2022-09-09T22:57:32.153 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:57:32.147+0000 7fe914452dc0 0 WARNING: could not read zone params for zone id=79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5 name=test-zone3 2022-09-09T22:57:32.167 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:57:32.168 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T22:57:32.168 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone2", 2022-09-09T22:57:32.168 INFO:teuthology.orchestra.run.smithi080.stdout: "domain_root": "test-zone2.rgw.meta:root", 2022-09-09T22:57:32.169 INFO:teuthology.orchestra.run.smithi080.stdout: "control_pool": "test-zone2.rgw.control", 2022-09-09T22:57:32.169 INFO:teuthology.orchestra.run.smithi080.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2022-09-09T22:57:32.169 INFO:teuthology.orchestra.run.smithi080.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2022-09-09T22:57:32.170 INFO:teuthology.orchestra.run.smithi080.stdout: "log_pool": "test-zone2.rgw.log", 2022-09-09T22:57:32.170 INFO:teuthology.orchestra.run.smithi080.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2022-09-09T22:57:32.170 INFO:teuthology.orchestra.run.smithi080.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2022-09-09T22:57:32.170 INFO:teuthology.orchestra.run.smithi080.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2022-09-09T22:57:32.171 INFO:teuthology.orchestra.run.smithi080.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2022-09-09T22:57:32.171 INFO:teuthology.orchestra.run.smithi080.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2022-09-09T22:57:32.171 INFO:teuthology.orchestra.run.smithi080.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2022-09-09T22:57:32.172 INFO:teuthology.orchestra.run.smithi080.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2022-09-09T22:57:32.172 INFO:teuthology.orchestra.run.smithi080.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2022-09-09T22:57:32.172 INFO:teuthology.orchestra.run.smithi080.stdout: "otp_pool": "test-zone2.rgw.otp", 2022-09-09T22:57:32.172 INFO:teuthology.orchestra.run.smithi080.stdout: "system_key": { 2022-09-09T22:57:32.172 INFO:teuthology.orchestra.run.smithi080.stdout: "access_key": "3U0V7WQYC6MLEXED", 2022-09-09T22:57:32.173 INFO:teuthology.orchestra.run.smithi080.stdout: "secret_key": "jA4iNzYKmfzg78oKBQTy4mNpjERzasVo" 2022-09-09T22:57:32.173 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:32.173 INFO:teuthology.orchestra.run.smithi080.stdout: "placement_pools": [ 2022-09-09T22:57:32.174 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:32.174 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "default-placement", 2022-09-09T22:57:32.174 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:57:32.175 INFO:teuthology.orchestra.run.smithi080.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2022-09-09T22:57:32.175 INFO:teuthology.orchestra.run.smithi080.stdout: "storage_classes": { 2022-09-09T22:57:32.176 INFO:teuthology.orchestra.run.smithi080.stdout: "STANDARD": { 2022-09-09T22:57:32.176 INFO:teuthology.orchestra.run.smithi080.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2022-09-09T22:57:32.177 INFO:teuthology.orchestra.run.smithi080.stdout: "compression_type": "random" 2022-09-09T22:57:32.177 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:32.178 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:32.178 INFO:teuthology.orchestra.run.smithi080.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2022-09-09T22:57:32.179 INFO:teuthology.orchestra.run.smithi080.stdout: "index_type": 0 2022-09-09T22:57:32.179 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:32.180 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:32.180 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:57:32.181 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T22:57:32.181 INFO:teuthology.orchestra.run.smithi080.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2022-09-09T22:57:32.182 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:57:32.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:57:32.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:57:32.186 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-09T22:57:32.232 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:57:32.233 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:57:32.326 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:57:32.323+0000 7f91b70e5dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:57:32.329 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:57:32.323+0000 7f91b70e5dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:57:32.358 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:57:32.358 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "2efdc3a1-023b-4e78-aaa2-0e069a034dac", 2022-09-09T22:57:32.359 INFO:teuthology.orchestra.run.smithi080.stdout: "epoch": 1, 2022-09-09T22:57:32.359 INFO:teuthology.orchestra.run.smithi080.stdout: "predecessor_uuid": "503b1d61-4513-48c8-a308-17b76219610c", 2022-09-09T22:57:32.359 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": [ 2022-09-09T22:57:32.359 INFO:teuthology.orchestra.run.smithi080.stdout: "1_1662762486.146147_45.1", 2022-09-09T22:57:32.360 INFO:teuthology.orchestra.run.smithi080.stdout: "", 2022-09-09T22:57:32.360 INFO:teuthology.orchestra.run.smithi080.stdout: "1_1662762637.757014_85.1", 2022-09-09T22:57:32.360 INFO:teuthology.orchestra.run.smithi080.stdout: "1_1662764240.168869_243.1" 2022-09-09T22:57:32.361 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:57:32.361 INFO:teuthology.orchestra.run.smithi080.stdout: "period_map": { 2022-09-09T22:57:32.361 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "2efdc3a1-023b-4e78-aaa2-0e069a034dac", 2022-09-09T22:57:32.362 INFO:teuthology.orchestra.run.smithi080.stdout: "zonegroups": [ 2022-09-09T22:57:32.362 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:32.362 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "test-zonegroup", 2022-09-09T22:57:32.363 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zonegroup", 2022-09-09T22:57:32.363 INFO:teuthology.orchestra.run.smithi080.stdout: "api_name": "", 2022-09-09T22:57:32.363 INFO:teuthology.orchestra.run.smithi080.stdout: "is_master": true, 2022-09-09T22:57:32.364 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T22:57:32.364 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi005.front.sepia.ceph.com:8000" 2022-09-09T22:57:32.365 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:57:32.365 INFO:teuthology.orchestra.run.smithi080.stdout: "hostnames": [], 2022-09-09T22:57:32.365 INFO:teuthology.orchestra.run.smithi080.stdout: "hostnames_s3website": [], 2022-09-09T22:57:32.366 INFO:teuthology.orchestra.run.smithi080.stdout: "master_zone": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T22:57:32.366 INFO:teuthology.orchestra.run.smithi080.stdout: "zones": [ 2022-09-09T22:57:32.366 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:32.366 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "3ff609e7-37d9-4b47-8369-34e6c99c4c03", 2022-09-09T22:57:32.367 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone4", 2022-09-09T22:57:32.367 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T22:57:32.367 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi080.front.sepia.ceph.com:8001" 2022-09-09T22:57:32.367 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:57:32.368 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": false, 2022-09-09T22:57:32.368 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": true, 2022-09-09T22:57:32.368 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2022-09-09T22:57:32.368 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": false, 2022-09-09T22:57:32.369 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "pubsub", 2022-09-09T22:57:32.369 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": false, 2022-09-09T22:57:32.369 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [ 2022-09-09T22:57:32.369 INFO:teuthology.orchestra.run.smithi080.stdout: "test-zone1" 2022-09-09T22:57:32.370 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:57:32.370 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "", 2022-09-09T22:57:32.370 INFO:teuthology.orchestra.run.smithi080.stdout: "supported_features": [ 2022-09-09T22:57:32.371 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T22:57:32.371 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:57:32.371 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:32.371 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:32.371 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T22:57:32.372 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone1", 2022-09-09T22:57:32.372 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T22:57:32.372 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi005.front.sepia.ceph.com:8000" 2022-09-09T22:57:32.372 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:57:32.373 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": false, 2022-09-09T22:57:32.373 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": true, 2022-09-09T22:57:32.373 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2022-09-09T22:57:32.373 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": false, 2022-09-09T22:57:32.374 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "", 2022-09-09T22:57:32.374 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": true, 2022-09-09T22:57:32.375 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [], 2022-09-09T22:57:32.375 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "", 2022-09-09T22:57:32.375 INFO:teuthology.orchestra.run.smithi080.stdout: "supported_features": [ 2022-09-09T22:57:32.376 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T22:57:32.376 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:57:32.376 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:32.377 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:32.377 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5", 2022-09-09T22:57:32.377 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone3", 2022-09-09T22:57:32.377 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T22:57:32.378 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi005.front.sepia.ceph.com:8001" 2022-09-09T22:57:32.378 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:57:32.378 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": false, 2022-09-09T22:57:32.378 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": true, 2022-09-09T22:57:32.379 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2022-09-09T22:57:32.379 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": false, 2022-09-09T22:57:32.379 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "", 2022-09-09T22:57:32.379 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": true, 2022-09-09T22:57:32.380 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [], 2022-09-09T22:57:32.380 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "", 2022-09-09T22:57:32.380 INFO:teuthology.orchestra.run.smithi080.stdout: "supported_features": [ 2022-09-09T22:57:32.380 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T22:57:32.381 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:57:32.381 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:32.381 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:32.381 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T22:57:32.382 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone2", 2022-09-09T22:57:32.382 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T22:57:32.391 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi080.front.sepia.ceph.com:8000" 2022-09-09T22:57:32.391 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:57:32.392 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": false, 2022-09-09T22:57:32.392 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": true, 2022-09-09T22:57:32.392 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2022-09-09T22:57:32.393 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": false, 2022-09-09T22:57:32.393 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "", 2022-09-09T22:57:32.393 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": true, 2022-09-09T22:57:32.393 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [], 2022-09-09T22:57:32.394 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "", 2022-09-09T22:57:32.394 INFO:teuthology.orchestra.run.smithi080.stdout: "supported_features": [ 2022-09-09T22:57:32.394 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T22:57:32.394 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:57:32.395 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:32.395 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:57:32.395 INFO:teuthology.orchestra.run.smithi080.stdout: "placement_targets": [ 2022-09-09T22:57:32.395 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:32.397 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "default-placement", 2022-09-09T22:57:32.397 INFO:teuthology.orchestra.run.smithi080.stdout: "tags": [], 2022-09-09T22:57:32.398 INFO:teuthology.orchestra.run.smithi080.stdout: "storage_classes": [ 2022-09-09T22:57:32.398 INFO:teuthology.orchestra.run.smithi080.stdout: "STANDARD" 2022-09-09T22:57:32.398 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:57:32.398 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:32.399 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:57:32.399 INFO:teuthology.orchestra.run.smithi080.stdout: "default_placement": "default-placement", 2022-09-09T22:57:32.399 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T22:57:32.399 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_policy": { 2022-09-09T22:57:32.400 INFO:teuthology.orchestra.run.smithi080.stdout: "groups": [] 2022-09-09T22:57:32.400 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:32.400 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled_features": [ 2022-09-09T22:57:32.400 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T22:57:32.401 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:57:32.401 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:32.401 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T22:57:32.402 INFO:teuthology.orchestra.run.smithi080.stdout: "short_zone_ids": [ 2022-09-09T22:57:32.402 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:32.403 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "3ff609e7-37d9-4b47-8369-34e6c99c4c03", 2022-09-09T22:57:32.403 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 2593176765 2022-09-09T22:57:32.404 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:32.404 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:32.405 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T22:57:32.405 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 1301724575 2022-09-09T22:57:32.406 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:32.406 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:32.406 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5", 2022-09-09T22:57:32.407 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 2787153217 2022-09-09T22:57:32.407 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:32.408 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:57:32.408 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T22:57:32.409 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 1899625128 2022-09-09T22:57:32.409 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:32.409 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:57:32.409 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:32.410 INFO:teuthology.orchestra.run.smithi080.stdout: "master_zonegroup": "test-zonegroup", 2022-09-09T22:57:32.410 INFO:teuthology.orchestra.run.smithi080.stdout: "master_zone": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T22:57:32.411 INFO:teuthology.orchestra.run.smithi080.stdout: "period_config": { 2022-09-09T22:57:32.411 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_quota": { 2022-09-09T22:57:32.411 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false, 2022-09-09T22:57:32.412 INFO:teuthology.orchestra.run.smithi080.stdout: "check_on_raw": false, 2022-09-09T22:57:32.412 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size": -1, 2022-09-09T22:57:32.412 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size_kb": 0, 2022-09-09T22:57:32.412 INFO:teuthology.orchestra.run.smithi080.stdout: "max_objects": -1 2022-09-09T22:57:32.413 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:32.413 INFO:teuthology.orchestra.run.smithi080.stdout: "user_quota": { 2022-09-09T22:57:32.413 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false, 2022-09-09T22:57:32.413 INFO:teuthology.orchestra.run.smithi080.stdout: "check_on_raw": false, 2022-09-09T22:57:32.414 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size": -1, 2022-09-09T22:57:32.414 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size_kb": 0, 2022-09-09T22:57:32.414 INFO:teuthology.orchestra.run.smithi080.stdout: "max_objects": -1 2022-09-09T22:57:32.414 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:32.415 INFO:teuthology.orchestra.run.smithi080.stdout: "user_ratelimit": { 2022-09-09T22:57:32.415 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_ops": 0, 2022-09-09T22:57:32.415 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_ops": 0, 2022-09-09T22:57:32.415 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_bytes": 0, 2022-09-09T22:57:32.416 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_bytes": 0, 2022-09-09T22:57:32.416 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false 2022-09-09T22:57:32.416 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:32.416 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_ratelimit": { 2022-09-09T22:57:32.417 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_ops": 0, 2022-09-09T22:57:32.417 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_ops": 0, 2022-09-09T22:57:32.417 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_bytes": 0, 2022-09-09T22:57:32.417 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_bytes": 0, 2022-09-09T22:57:32.418 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false 2022-09-09T22:57:32.418 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:32.418 INFO:teuthology.orchestra.run.smithi080.stdout: "anonymous_ratelimit": { 2022-09-09T22:57:32.418 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_ops": 0, 2022-09-09T22:57:32.419 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_ops": 0, 2022-09-09T22:57:32.419 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_bytes": 0, 2022-09-09T22:57:32.419 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_bytes": 0, 2022-09-09T22:57:32.419 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false 2022-09-09T22:57:32.420 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:57:32.420 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:57:32.420 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T22:57:32.420 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_name": "test-realm", 2022-09-09T22:57:32.421 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 3 2022-09-09T22:57:32.421 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:57:32.421 INFO:rgw_multi.tests:Set master zone=test-zone2, waiting 60s for reconfiguration.. 2022-09-09T22:57:35.415 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2022-09-09T22:58:32.423 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jkcgbj-42 2022-09-09T22:58:32.425 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:58:32.425 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:58:32.426 DEBUG:boto:Method: PUT 2022-09-09T22:58:32.426 DEBUG:boto:Path: /jkcgbj-42/ 2022-09-09T22:58:32.427 DEBUG:boto:Data: 2022-09-09T22:58:32.427 DEBUG:boto:Headers: {} 2022-09-09T22:58:32.428 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:58:32.428 DEBUG:boto:Port: 8000 2022-09-09T22:58:32.429 DEBUG:boto:Params: {} 2022-09-09T22:58:32.430 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T22:58:32.430 DEBUG:boto:Token: None 2022-09-09T22:58:32.431 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:58:32 GMT /jkcgbj-42/ 2022-09-09T22:58:32.432 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:SLzJ8mmcPMsr/1NacFnG+vgxKfs= 2022-09-09T22:58:32.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:58:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:SLzJ8mmcPMsr/1NacFnG+vgxKfs=', 'Content-Length': '0'} 2022-09-09T22:58:32.514 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f3b4d8fc7d53ba9a-00631bc518-13e7-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:58:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:58:32.515 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jkcgbj-43 2022-09-09T22:58:32.515 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:58:32.516 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:58:32.517 DEBUG:boto:Method: PUT 2022-09-09T22:58:32.518 DEBUG:boto:Path: /jkcgbj-43/ 2022-09-09T22:58:32.519 DEBUG:boto:Data: 2022-09-09T22:58:32.519 DEBUG:boto:Headers: {} 2022-09-09T22:58:32.520 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:58:32.520 DEBUG:boto:Port: 8000 2022-09-09T22:58:32.521 DEBUG:boto:Params: {} 2022-09-09T22:58:32.521 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T22:58:32.522 DEBUG:boto:Token: None 2022-09-09T22:58:32.523 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:58:32 GMT /jkcgbj-43/ 2022-09-09T22:58:32.523 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6osj55czY7vR9mGS5iaOX/cMMdU= 2022-09-09T22:58:32.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:58:32 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6osj55czY7vR9mGS5iaOX/cMMdU=', 'Content-Length': '0'} 2022-09-09T22:58:32.567 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000851778bb7b932cc2-00631bc518-153d-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:58:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:58:32.568 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:58:32.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:58:32.570 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:58:32.620 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:58:32.621 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:58:32.678 INFO:teuthology.orchestra.run.smithi080.stderr:No --period given, using current period=2efdc3a1-023b-4e78-aaa2-0e069a034dac 2022-09-09T22:58:32.703 INFO:teuthology.orchestra.run.smithi080.stdout:[ 2022-09-09T22:58:32.703 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:58:32.704 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:58:32.704 INFO:teuthology.orchestra.run.smithi080.stdout: "last_update": "0.000000" 2022-09-09T22:58:32.704 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:58:32.705 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:58:32.705 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:58:32.705 INFO:teuthology.orchestra.run.smithi080.stdout: "last_update": "0.000000" 2022-09-09T22:58:32.705 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:58:32.706 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:58:32.706 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:58:32.706 INFO:teuthology.orchestra.run.smithi080.stdout: "last_update": "0.000000" 2022-09-09T22:58:32.707 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:58:32.707 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:58:32.707 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764312.564852_2684.1", 2022-09-09T22:58:32.708 INFO:teuthology.orchestra.run.smithi080.stdout: "last_update": "2022-09-09T22:58:32.564852Z" 2022-09-09T22:58:32.708 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:58:32.708 INFO:teuthology.orchestra.run.smithi080.stdout:] 2022-09-09T22:58:32.709 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '', 3: '1_1662764312.564852_2684.1'} 2022-09-09T22:58:32.710 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone1 2022-09-09T22:58:32.711 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:58:32.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:58:32.712 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:58:32.761 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:58:32.762 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:58:32.813 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:58:32.810+0000 7fadb5f41dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:58:32.815 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:58:32.810+0000 7fadb5f41dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:58:32.841 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:58:32.842 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T22:58:32.842 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T22:58:32.843 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T22:58:32.843 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T22:58:32.844 INFO:teuthology.orchestra.run.smithi005.stdout: "period": "2efdc3a1-023b-4e78-aaa2-0e069a034dac", 2022-09-09T22:58:32.844 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 3 2022-09-09T22:58:32.845 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:58:32.845 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T22:58:32.846 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:58:32.846 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T22:58:32.847 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:58:32.847 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 0, 2022-09-09T22:58:32.848 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "bucket.instance:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10", 2022-09-09T22:58:32.848 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:58:32.849 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 81, 2022-09-09T22:58:32.849 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 1, 2022-09-09T22:58:32.849 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:58:32.850 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 0 2022-09-09T22:58:32.850 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:58:32.851 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:58:32.851 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:58:32.851 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T22:58:32.852 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:58:32.852 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:58:32.853 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:58:32.853 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:58:32.853 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:58:32.854 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:58:32.854 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:58:32.855 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 3 2022-09-09T22:58:32.855 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:58:32.856 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:58:32.856 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:58:32.856 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T22:58:32.856 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:58:32.857 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:58:32.857 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:58:32.857 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:58:32.857 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:58:32.858 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:58:32.858 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:58:32.867 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 3 2022-09-09T22:58:32.867 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:58:32.868 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:58:32.869 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:58:32.869 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T22:58:32.870 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:58:32.870 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:58:32.871 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662764312.564852_2684.1", 2022-09-09T22:58:32.871 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:58:32.871 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:58:32.871 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:58:32.872 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:58:32.564852Z", 2022-09-09T22:58:32.873 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 3 2022-09-09T22:58:32.873 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:58:32.874 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:58:32.874 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:58:32.875 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:58:32.875 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T22:58:32.875 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 81, 2022-09-09T22:58:32.876 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 1 2022-09-09T22:58:32.876 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:58:32.877 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T22:58:32Z" 2022-09-09T22:58:32.877 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:58:32.879 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2efdc3a1-023b-4e78-aaa2-0e069a034dac", "realm_epoch": 3 }, "markers": [ { "key": 0, "val": { "state": 0, "marker": "bucket.instance:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10", "next_step_marker": "", "total_entries": 81, "pos": 1, "timestamp": "0.000000", "realm_epoch": 0 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 3 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 3 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764312.564852_2684.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:58:32.564852Z", "realm_epoch": 3 } } ] }, "full_sync": { "total": 81, "complete": 1 }, "current_time": "2022-09-09T22:58:32Z" } 2022-09-09T22:58:32.880 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 0, 'marker': 'bucket.instance:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10', 'next_step_marker': '', 'total_entries': 81, 'pos': 1, 'timestamp': '0.000000', 'realm_epoch': 0}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 3}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 3}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764312.564852_2684.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:58:32.564852Z', 'realm_epoch': 3}}] 2022-09-09T22:58:32.881 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764312.564852_2684.1'} 2022-09-09T22:58:32.882 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764312.564852_2684.1'} 2022-09-09T22:58:32.882 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone1 2022-09-09T22:58:32.883 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:58:32.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:58:32.884 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-09T22:58:32.931 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:58:32.932 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:58:32.980 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:58:32.974+0000 7f34d9e32dc0 0 NOTICE: overriding master zone: cc1fc6ed-f06f-45bc-961e-adec328c8651 2022-09-09T22:58:32.981 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:58:32.978+0000 7f34d9e32dc0 0 failed reading obj info from .rgw.root:zone_info.3ff609e7-37d9-4b47-8369-34e6c99c4c03: (2) No such file or directory 2022-09-09T22:58:32.981 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:58:32.978+0000 7f34d9e32dc0 0 WARNING: could not read zone params for zone id=3ff609e7-37d9-4b47-8369-34e6c99c4c03 name=test-zone4 2022-09-09T22:58:32.982 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:58:32.978+0000 7f34d9e32dc0 0 failed reading obj info from .rgw.root:zone_info.cc1fc6ed-f06f-45bc-961e-adec328c8651: (2) No such file or directory 2022-09-09T22:58:32.982 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:58:32.978+0000 7f34d9e32dc0 0 WARNING: could not read zone params for zone id=cc1fc6ed-f06f-45bc-961e-adec328c8651 name=test-zone2 2022-09-09T22:58:32.992 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:58:32.992 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T22:58:32.993 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zone1", 2022-09-09T22:58:32.993 INFO:teuthology.orchestra.run.smithi005.stdout: "domain_root": "test-zone1.rgw.meta:root", 2022-09-09T22:58:32.993 INFO:teuthology.orchestra.run.smithi005.stdout: "control_pool": "test-zone1.rgw.control", 2022-09-09T22:58:32.993 INFO:teuthology.orchestra.run.smithi005.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2022-09-09T22:58:32.994 INFO:teuthology.orchestra.run.smithi005.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2022-09-09T22:58:32.995 INFO:teuthology.orchestra.run.smithi005.stdout: "log_pool": "test-zone1.rgw.log", 2022-09-09T22:58:32.996 INFO:teuthology.orchestra.run.smithi005.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2022-09-09T22:58:32.996 INFO:teuthology.orchestra.run.smithi005.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2022-09-09T22:58:32.996 INFO:teuthology.orchestra.run.smithi005.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2022-09-09T22:58:32.997 INFO:teuthology.orchestra.run.smithi005.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2022-09-09T22:58:32.997 INFO:teuthology.orchestra.run.smithi005.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2022-09-09T22:58:32.997 INFO:teuthology.orchestra.run.smithi005.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2022-09-09T22:58:32.997 INFO:teuthology.orchestra.run.smithi005.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2022-09-09T22:58:32.997 INFO:teuthology.orchestra.run.smithi005.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2022-09-09T22:58:32.998 INFO:teuthology.orchestra.run.smithi005.stdout: "otp_pool": "test-zone1.rgw.otp", 2022-09-09T22:58:32.998 INFO:teuthology.orchestra.run.smithi005.stdout: "system_key": { 2022-09-09T22:58:32.998 INFO:teuthology.orchestra.run.smithi005.stdout: "access_key": "3U0V7WQYC6MLEXED", 2022-09-09T22:58:32.999 INFO:teuthology.orchestra.run.smithi005.stdout: "secret_key": "jA4iNzYKmfzg78oKBQTy4mNpjERzasVo" 2022-09-09T22:58:32.999 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:58:32.999 INFO:teuthology.orchestra.run.smithi005.stdout: "placement_pools": [ 2022-09-09T22:58:32.999 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:58:32.999 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "default-placement", 2022-09-09T22:58:33.000 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:58:33.000 INFO:teuthology.orchestra.run.smithi005.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2022-09-09T22:58:33.000 INFO:teuthology.orchestra.run.smithi005.stdout: "storage_classes": { 2022-09-09T22:58:33.000 INFO:teuthology.orchestra.run.smithi005.stdout: "STANDARD": { 2022-09-09T22:58:33.001 INFO:teuthology.orchestra.run.smithi005.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2022-09-09T22:58:33.001 INFO:teuthology.orchestra.run.smithi005.stdout: "compression_type": "random" 2022-09-09T22:58:33.001 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:58:33.001 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:58:33.002 INFO:teuthology.orchestra.run.smithi005.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2022-09-09T22:58:33.002 INFO:teuthology.orchestra.run.smithi005.stdout: "index_type": 0 2022-09-09T22:58:33.002 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:58:33.002 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:58:33.003 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:58:33.003 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T22:58:33.003 INFO:teuthology.orchestra.run.smithi005.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2022-09-09T22:58:33.003 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:58:33.005 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:58:33.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:58:33.006 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-09T22:58:33.094 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:58:33.095 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:58:33.178 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:58:33.174+0000 7fd0e6ad2dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:58:33.180 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:58:33.178+0000 7fd0e6ad2dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:58:33.208 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:58:33.209 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T22:58:33.209 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2022-09-09T22:58:33.210 INFO:teuthology.orchestra.run.smithi005.stdout: "predecessor_uuid": "2efdc3a1-023b-4e78-aaa2-0e069a034dac", 2022-09-09T22:58:33.210 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": [ 2022-09-09T22:58:33.210 INFO:teuthology.orchestra.run.smithi005.stdout: "", 2022-09-09T22:58:33.211 INFO:teuthology.orchestra.run.smithi005.stdout: "", 2022-09-09T22:58:33.211 INFO:teuthology.orchestra.run.smithi005.stdout: "", 2022-09-09T22:58:33.211 INFO:teuthology.orchestra.run.smithi005.stdout: "1_1662764312.564852_2684.1" 2022-09-09T22:58:33.211 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:58:33.212 INFO:teuthology.orchestra.run.smithi005.stdout: "period_map": { 2022-09-09T22:58:33.212 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T22:58:33.212 INFO:teuthology.orchestra.run.smithi005.stdout: "zonegroups": [ 2022-09-09T22:58:33.212 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:58:33.213 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "test-zonegroup", 2022-09-09T22:58:33.213 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zonegroup", 2022-09-09T22:58:33.214 INFO:teuthology.orchestra.run.smithi005.stdout: "api_name": "", 2022-09-09T22:58:33.214 INFO:teuthology.orchestra.run.smithi005.stdout: "is_master": true, 2022-09-09T22:58:33.216 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T22:58:33.216 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi005.front.sepia.ceph.com:8000" 2022-09-09T22:58:33.217 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:58:33.218 INFO:teuthology.orchestra.run.smithi005.stdout: "hostnames": [], 2022-09-09T22:58:33.218 INFO:teuthology.orchestra.run.smithi005.stdout: "hostnames_s3website": [], 2022-09-09T22:58:33.219 INFO:teuthology.orchestra.run.smithi005.stdout: "master_zone": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T22:58:33.219 INFO:teuthology.orchestra.run.smithi005.stdout: "zones": [ 2022-09-09T22:58:33.220 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:58:33.220 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "3ff609e7-37d9-4b47-8369-34e6c99c4c03", 2022-09-09T22:58:33.220 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zone4", 2022-09-09T22:58:33.220 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T22:58:33.221 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi080.front.sepia.ceph.com:8001" 2022-09-09T22:58:33.221 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:58:33.221 INFO:teuthology.orchestra.run.smithi005.stdout: "log_meta": false, 2022-09-09T22:58:33.221 INFO:teuthology.orchestra.run.smithi005.stdout: "log_data": true, 2022-09-09T22:58:33.222 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_index_max_shards": 11, 2022-09-09T22:58:33.222 INFO:teuthology.orchestra.run.smithi005.stdout: "read_only": false, 2022-09-09T22:58:33.222 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_type": "pubsub", 2022-09-09T22:58:33.222 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from_all": false, 2022-09-09T22:58:33.223 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from": [ 2022-09-09T22:58:33.223 INFO:teuthology.orchestra.run.smithi005.stdout: "test-zone1" 2022-09-09T22:58:33.223 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:58:33.223 INFO:teuthology.orchestra.run.smithi005.stdout: "redirect_zone": "", 2022-09-09T22:58:33.224 INFO:teuthology.orchestra.run.smithi005.stdout: "supported_features": [ 2022-09-09T22:58:33.224 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T22:58:33.224 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:58:33.225 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:58:33.225 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:58:33.225 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T22:58:33.225 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zone1", 2022-09-09T22:58:33.226 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T22:58:33.226 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi005.front.sepia.ceph.com:8000" 2022-09-09T22:58:33.226 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:58:33.226 INFO:teuthology.orchestra.run.smithi005.stdout: "log_meta": false, 2022-09-09T22:58:33.227 INFO:teuthology.orchestra.run.smithi005.stdout: "log_data": true, 2022-09-09T22:58:33.227 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_index_max_shards": 11, 2022-09-09T22:58:33.227 INFO:teuthology.orchestra.run.smithi005.stdout: "read_only": false, 2022-09-09T22:58:33.227 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_type": "", 2022-09-09T22:58:33.228 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from_all": true, 2022-09-09T22:58:33.228 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from": [], 2022-09-09T22:58:33.228 INFO:teuthology.orchestra.run.smithi005.stdout: "redirect_zone": "", 2022-09-09T22:58:33.229 INFO:teuthology.orchestra.run.smithi005.stdout: "supported_features": [ 2022-09-09T22:58:33.229 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T22:58:33.229 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:58:33.229 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:58:33.230 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:58:33.230 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5", 2022-09-09T22:58:33.230 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zone3", 2022-09-09T22:58:33.230 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T22:58:33.231 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi005.front.sepia.ceph.com:8001" 2022-09-09T22:58:33.231 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:58:33.231 INFO:teuthology.orchestra.run.smithi005.stdout: "log_meta": false, 2022-09-09T22:58:33.231 INFO:teuthology.orchestra.run.smithi005.stdout: "log_data": true, 2022-09-09T22:58:33.232 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_index_max_shards": 11, 2022-09-09T22:58:33.232 INFO:teuthology.orchestra.run.smithi005.stdout: "read_only": false, 2022-09-09T22:58:33.232 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_type": "", 2022-09-09T22:58:33.233 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from_all": true, 2022-09-09T22:58:33.233 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from": [], 2022-09-09T22:58:33.233 INFO:teuthology.orchestra.run.smithi005.stdout: "redirect_zone": "", 2022-09-09T22:58:33.233 INFO:teuthology.orchestra.run.smithi005.stdout: "supported_features": [ 2022-09-09T22:58:33.234 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T22:58:33.234 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:58:33.234 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:58:33.234 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:58:33.235 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T22:58:33.235 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zone2", 2022-09-09T22:58:33.235 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T22:58:33.235 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi080.front.sepia.ceph.com:8000" 2022-09-09T22:58:33.236 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:58:33.236 INFO:teuthology.orchestra.run.smithi005.stdout: "log_meta": false, 2022-09-09T22:58:33.236 INFO:teuthology.orchestra.run.smithi005.stdout: "log_data": true, 2022-09-09T22:58:33.237 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_index_max_shards": 11, 2022-09-09T22:58:33.237 INFO:teuthology.orchestra.run.smithi005.stdout: "read_only": false, 2022-09-09T22:58:33.237 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_type": "", 2022-09-09T22:58:33.238 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from_all": true, 2022-09-09T22:58:33.239 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from": [], 2022-09-09T22:58:33.239 INFO:teuthology.orchestra.run.smithi005.stdout: "redirect_zone": "", 2022-09-09T22:58:33.239 INFO:teuthology.orchestra.run.smithi005.stdout: "supported_features": [ 2022-09-09T22:58:33.239 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T22:58:33.240 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:58:33.240 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:58:33.240 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:58:33.241 INFO:teuthology.orchestra.run.smithi005.stdout: "placement_targets": [ 2022-09-09T22:58:33.241 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:58:33.241 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "default-placement", 2022-09-09T22:58:33.243 INFO:teuthology.orchestra.run.smithi005.stdout: "tags": [], 2022-09-09T22:58:33.243 INFO:teuthology.orchestra.run.smithi005.stdout: "storage_classes": [ 2022-09-09T22:58:33.243 INFO:teuthology.orchestra.run.smithi005.stdout: "STANDARD" 2022-09-09T22:58:33.244 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:58:33.244 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:58:33.244 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:58:33.244 INFO:teuthology.orchestra.run.smithi005.stdout: "default_placement": "default-placement", 2022-09-09T22:58:33.245 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T22:58:33.245 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_policy": { 2022-09-09T22:58:33.245 INFO:teuthology.orchestra.run.smithi005.stdout: "groups": [] 2022-09-09T22:58:33.245 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:58:33.246 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled_features": [ 2022-09-09T22:58:33.246 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T22:58:33.246 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:58:33.246 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:58:33.247 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T22:58:33.247 INFO:teuthology.orchestra.run.smithi005.stdout: "short_zone_ids": [ 2022-09-09T22:58:33.247 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:58:33.247 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "3ff609e7-37d9-4b47-8369-34e6c99c4c03", 2022-09-09T22:58:33.248 INFO:teuthology.orchestra.run.smithi005.stdout: "val": 2593176765 2022-09-09T22:58:33.248 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:58:33.248 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:58:33.248 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T22:58:33.249 INFO:teuthology.orchestra.run.smithi005.stdout: "val": 1301724575 2022-09-09T22:58:33.249 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:58:33.250 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:58:33.250 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5", 2022-09-09T22:58:33.251 INFO:teuthology.orchestra.run.smithi005.stdout: "val": 2787153217 2022-09-09T22:58:33.251 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:58:33.251 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:58:33.252 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T22:58:33.252 INFO:teuthology.orchestra.run.smithi005.stdout: "val": 1899625128 2022-09-09T22:58:33.253 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:58:33.253 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:58:33.254 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:58:33.254 INFO:teuthology.orchestra.run.smithi005.stdout: "master_zonegroup": "test-zonegroup", 2022-09-09T22:58:33.254 INFO:teuthology.orchestra.run.smithi005.stdout: "master_zone": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T22:58:33.255 INFO:teuthology.orchestra.run.smithi005.stdout: "period_config": { 2022-09-09T22:58:33.256 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_quota": { 2022-09-09T22:58:33.256 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false, 2022-09-09T22:58:33.256 INFO:teuthology.orchestra.run.smithi005.stdout: "check_on_raw": false, 2022-09-09T22:58:33.257 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size": -1, 2022-09-09T22:58:33.258 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size_kb": 0, 2022-09-09T22:58:33.259 INFO:teuthology.orchestra.run.smithi005.stdout: "max_objects": -1 2022-09-09T22:58:33.259 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:58:33.260 INFO:teuthology.orchestra.run.smithi005.stdout: "user_quota": { 2022-09-09T22:58:33.260 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false, 2022-09-09T22:58:33.260 INFO:teuthology.orchestra.run.smithi005.stdout: "check_on_raw": false, 2022-09-09T22:58:33.260 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size": -1, 2022-09-09T22:58:33.261 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size_kb": 0, 2022-09-09T22:58:33.261 INFO:teuthology.orchestra.run.smithi005.stdout: "max_objects": -1 2022-09-09T22:58:33.261 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:58:33.262 INFO:teuthology.orchestra.run.smithi005.stdout: "user_ratelimit": { 2022-09-09T22:58:33.262 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_ops": 0, 2022-09-09T22:58:33.262 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_ops": 0, 2022-09-09T22:58:33.263 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_bytes": 0, 2022-09-09T22:58:33.263 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_bytes": 0, 2022-09-09T22:58:33.263 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false 2022-09-09T22:58:33.264 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:58:33.264 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_ratelimit": { 2022-09-09T22:58:33.265 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_ops": 0, 2022-09-09T22:58:33.265 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_ops": 0, 2022-09-09T22:58:33.266 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_bytes": 0, 2022-09-09T22:58:33.266 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_bytes": 0, 2022-09-09T22:58:33.267 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false 2022-09-09T22:58:33.267 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:58:33.268 INFO:teuthology.orchestra.run.smithi005.stdout: "anonymous_ratelimit": { 2022-09-09T22:58:33.268 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_ops": 0, 2022-09-09T22:58:33.269 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_ops": 0, 2022-09-09T22:58:33.269 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_bytes": 0, 2022-09-09T22:58:33.270 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_bytes": 0, 2022-09-09T22:58:33.270 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false 2022-09-09T22:58:33.271 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:58:33.271 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:58:33.271 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T22:58:33.272 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_name": "test-realm", 2022-09-09T22:58:33.272 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 4 2022-09-09T22:58:33.272 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:58:33.273 INFO:rgw_multi.tests:Set master zone=test-zone1, waiting 60s for reconfiguration.. 2022-09-09T22:59:33.275 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jkcgbj-44 2022-09-09T22:59:33.277 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:33.278 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:33.279 DEBUG:boto:Method: PUT 2022-09-09T22:59:33.279 DEBUG:boto:Path: /jkcgbj-44/ 2022-09-09T22:59:33.280 DEBUG:boto:Data: 2022-09-09T22:59:33.281 DEBUG:boto:Headers: {} 2022-09-09T22:59:33.282 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:33.283 DEBUG:boto:Port: 8000 2022-09-09T22:59:33.284 DEBUG:boto:Params: {} 2022-09-09T22:59:33.285 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T22:59:33.287 DEBUG:boto:Token: None 2022-09-09T22:59:33.288 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:59:33 GMT /jkcgbj-44/ 2022-09-09T22:59:33.289 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:h3bcYRTGersH8gzX3TVuQetCTQg= 2022-09-09T22:59:33.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:33 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:h3bcYRTGersH8gzX3TVuQetCTQg=', 'Content-Length': '0'} 2022-09-09T22:59:33.325 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000023d44c2bfe5930c4-00631bc555-13fd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:33.326 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jkcgbj-45 2022-09-09T22:59:33.327 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:33.327 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:33.328 DEBUG:boto:Method: PUT 2022-09-09T22:59:33.329 DEBUG:boto:Path: /jkcgbj-45/ 2022-09-09T22:59:33.329 DEBUG:boto:Data: 2022-09-09T22:59:33.330 DEBUG:boto:Headers: {} 2022-09-09T22:59:33.330 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:33.331 DEBUG:boto:Port: 8000 2022-09-09T22:59:33.331 DEBUG:boto:Params: {} 2022-09-09T22:59:33.332 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T22:59:33.332 DEBUG:boto:Token: None 2022-09-09T22:59:33.333 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:59:33 GMT /jkcgbj-45/ 2022-09-09T22:59:33.334 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:pnVG32Uv+yv+zfbNkhalZzzrqBw= 2022-09-09T22:59:33.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:33 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:pnVG32Uv+yv+zfbNkhalZzzrqBw=', 'Content-Length': '0'} 2022-09-09T22:59:33.404 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d4246735f359e1b3-00631bc555-1549-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:33.406 INFO:tasks.rgw.c1.client.1:Restarting daemon 2022-09-09T22:59:33.407 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2022-09-09T22:59:33.413 INFO:tasks.rgw.c1.client.1:Started 2022-09-09T22:59:33.414 DEBUG:teuthology.orchestra.run.smithi005:> curl http://smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:33.435 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-09T22:59:33.436 INFO:teuthology.orchestra.run.smithi005.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-09T22:59:33.437 INFO:teuthology.orchestra.run.smithi005.stderr: Dload Upload Total Spent Left Speed 2022-09-09T22:59:33.438 INFO:teuthology.orchestra.run.smithi005.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2022-09-09T22:59:33.438 INFO:teuthology.orchestra.run.smithi005.stderr:curl: (7) Failed to connect to smithi005.front.sepia.ceph.com port 8001: Connection refused 2022-09-09T22:59:34.440 DEBUG:teuthology.orchestra.run.smithi005:> curl http://smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:34.459 INFO:teuthology.orchestra.run.smithi005.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-09T22:59:34.460 INFO:teuthology.orchestra.run.smithi005.stderr: Dload Upload Total Spent Left Speed 2022-09-09T22:59:34.464 INFO:teuthology.orchestra.run.smithi005.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 53500 0 --:--:-- --:--:-- --:--:-- 53500 2022-09-09T22:59:34.467 INFO:teuthology.orchestra.run.smithi005.stdout:anonymous 2022-09-09T22:59:34.467 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:34.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:34.469 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:59:34.512 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:34.512 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:34.561 INFO:teuthology.orchestra.run.smithi005.stderr:No --period given, using current period=fd4bc158-06fd-4840-8a27-33fabda67b99 2022-09-09T22:59:34.589 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T22:59:34.590 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:34.590 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:59:34.591 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:59:34.591 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:34.592 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:34.592 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:59:34.592 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:59:34.593 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:34.594 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:34.594 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:59:34.595 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:59:34.595 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:34.596 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:34.596 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662764373.377065_360.1", 2022-09-09T22:59:34.596 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:59:33.377065Z" 2022-09-09T22:59:34.597 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:34.597 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T22:59:34.599 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '', 3: '1_1662764373.377065_360.1'} 2022-09-09T22:59:34.600 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-09T22:59:34.601 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:34.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:34.602 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:59:34.650 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:34.651 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:34.725 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:59:34.721+0000 7f06976dcdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:59:34.727 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:59:34.725+0000 7f06976dcdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:59:34.753 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:59:34.754 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:59:34.755 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:59:34.756 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:59:34.756 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:59:34.756 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T22:59:34.757 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T22:59:34.757 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:34.758 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:59:34.758 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:34.759 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:59:34.759 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:59:34.760 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:59:34.760 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:59:34.761 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:59:34.761 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:59:34.762 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:59:34.762 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:59:34.762 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:59:34.763 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:34.763 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:34.763 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:34.763 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:59:34.764 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:59:34.764 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:59:34.764 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:59:34.765 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:59:34.765 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:59:34.765 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:59:34.765 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:59:34.766 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:59:34.766 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:34.766 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:34.767 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:34.767 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:59:34.767 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:59:34.767 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:59:34.768 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:59:34.768 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:59:34.768 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:59:34.768 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:59:34.769 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:59:34.769 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:59:34.769 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:34.770 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:34.770 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:34.770 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:59:34.771 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:59:34.771 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:59:34.771 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764373.377065_360.1", 2022-09-09T22:59:34.772 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:59:34.772 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:59:34.773 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:59:34.773 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:33.377065Z", 2022-09-09T22:59:34.774 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T22:59:34.774 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:34.775 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:34.775 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:59:34.776 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:34.776 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:59:34.777 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:59:34.777 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:59:34.778 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:34.778 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:59:34Z" 2022-09-09T22:59:34.779 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:59:34.781 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764373.377065_360.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:33.377065Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:59:34Z" } 2022-09-09T22:59:34.782 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764373.377065_360.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:33.377065Z', 'realm_epoch': 4}}] 2022-09-09T22:59:34.783 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764373.377065_360.1'} 2022-09-09T22:59:34.784 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764373.377065_360.1'} 2022-09-09T22:59:34.784 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-09T22:59:34.785 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-09T22:59:34.785 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:34.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:34.786 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:59:34.830 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:34.831 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:34.883 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:59:34.880+0000 7f7e2accedc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:59:34.884 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:59:34.884+0000 7f7e2accedc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:59:34.911 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:59:34.911 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T22:59:34.911 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T22:59:34.912 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T22:59:34.912 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T22:59:34.913 INFO:teuthology.orchestra.run.smithi005.stdout: "period": "2efdc3a1-023b-4e78-aaa2-0e069a034dac", 2022-09-09T22:59:34.913 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 3 2022-09-09T22:59:34.913 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:34.914 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T22:59:34.914 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:34.915 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T22:59:34.915 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:59:34.915 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:59:34.915 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:59:34.916 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:59:34.916 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 1, 2022-09-09T22:59:34.916 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:59:34.916 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:59:34.917 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:59:34.917 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:34.917 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:34.918 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:34.918 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T22:59:34.918 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:59:34.918 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:59:34.919 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:59:34.919 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:59:34.920 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:59:34.920 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:59:34.920 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:59:34.920 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:59:34.921 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:34.921 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:34.921 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:34.922 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T22:59:34.922 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:59:34.922 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:59:34.922 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:59:34.923 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:59:34.923 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:59:34.923 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:59:34.923 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:59:34.924 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:59:34.924 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:34.924 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:34.924 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:34.925 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T22:59:34.925 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:59:34.926 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:59:34.926 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662764312.475928_2677.1", 2022-09-09T22:59:34.926 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:59:34.927 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:59:34.927 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:59:34.927 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:58:32.475928Z", 2022-09-09T22:59:34.928 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 3 2022-09-09T22:59:34.928 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:34.928 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:34.928 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:59:34.929 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:34.929 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T22:59:34.929 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 1, 2022-09-09T22:59:34.929 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 1 2022-09-09T22:59:34.930 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:34.930 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T22:59:34Z" 2022-09-09T22:59:34.930 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:59:34.932 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2efdc3a1-023b-4e78-aaa2-0e069a034dac", "realm_epoch": 3 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764312.475928_2677.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:58:32.475928Z", "realm_epoch": 3 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:59:34Z" } 2022-09-09T22:59:34.932 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764312.475928_2677.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:58:32.475928Z', 'realm_epoch': 3}}] 2022-09-09T22:59:34.933 WARNING:rgw_multi.tests:zone test-zone3 is syncing realm epoch=3, behind current realm epoch=4 2022-09-09T22:59:39.934 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:39.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:39.935 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:59:39.982 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:39.983 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:40.035 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:59:40.032+0000 7fabaaeeedc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:59:40.037 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T22:59:40.036+0000 7fabaaeeedc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:59:40.063 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T22:59:40.063 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T22:59:40.064 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T22:59:40.064 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T22:59:40.064 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T22:59:40.065 INFO:teuthology.orchestra.run.smithi005.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T22:59:40.065 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 4 2022-09-09T22:59:40.065 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:40.065 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T22:59:40.066 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:40.066 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T22:59:40.067 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:59:40.067 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:59:40.067 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:59:40.067 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:59:40.068 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 1, 2022-09-09T22:59:40.068 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:59:40.068 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:59:40.069 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:59:40.069 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:40.069 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:40.070 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:40.070 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T22:59:40.071 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:59:40.071 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:59:40.072 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:59:40.072 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:59:40.073 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:59:40.073 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:59:40.074 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T22:59:40.074 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:59:40.074 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:40.075 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:40.075 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:40.076 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T22:59:40.076 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:59:40.076 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:59:40.077 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:59:40.077 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:59:40.078 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:59:40.078 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:59:40.078 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:59:40.079 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T22:59:40.079 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:40.079 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:40.080 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:40.080 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T22:59:40.080 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T22:59:40.080 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T22:59:40.081 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662764373.377065_360.1", 2022-09-09T22:59:40.081 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T22:59:40.081 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T22:59:40.081 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T22:59:40.082 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:59:33.377065Z", 2022-09-09T22:59:40.082 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 4 2022-09-09T22:59:40.082 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:40.083 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:40.083 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T22:59:40.083 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:40.083 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T22:59:40.083 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 1, 2022-09-09T22:59:40.084 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 1 2022-09-09T22:59:40.084 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:40.084 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T22:59:40Z" 2022-09-09T22:59:40.084 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T22:59:40.086 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764373.377065_360.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:33.377065Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:59:40Z" } 2022-09-09T22:59:40.087 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764373.377065_360.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:33.377065Z', 'realm_epoch': 4}}] 2022-09-09T22:59:40.088 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764373.377065_360.1'} 2022-09-09T22:59:40.089 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764373.377065_360.1'} 2022-09-09T22:59:40.090 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-09T22:59:40.091 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-09T22:59:40.092 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:40.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:40.094 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:59:40.141 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:40.141 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:40.199 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:59:40.197+0000 7fc4ffc55dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:59:40.201 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:59:40.201+0000 7fc4ffc55dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:59:40.229 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:59:40.229 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:59:40.230 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:59:40.230 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:59:40.231 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:59:40.231 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T22:59:40.231 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T22:59:40.231 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:40.232 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:59:40.232 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:40.232 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:59:40.233 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:59:40.233 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:59:40.233 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:59:40.233 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:59:40.234 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:59:40.234 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:59:40.235 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:59:40.235 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:59:40.235 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:40.235 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:40.236 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:40.236 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:59:40.236 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:59:40.236 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:59:40.237 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:59:40.237 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:59:40.237 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:59:40.237 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:59:40.238 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:59:40.238 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:59:40.238 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:40.239 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:40.239 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:40.240 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:59:40.240 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:59:40.240 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:59:40.241 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:59:40.241 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:59:40.242 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:59:40.242 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:59:40.243 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:59:40.243 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:59:40.244 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:40.244 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:40.244 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:40.245 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:59:40.245 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:59:40.246 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:59:40.246 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764373.377065_360.1", 2022-09-09T22:59:40.247 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:59:40.247 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:59:40.247 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:59:40.247 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:33.377065Z", 2022-09-09T22:59:40.248 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T22:59:40.248 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:40.248 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:40.248 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:59:40.249 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:40.249 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:59:40.249 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:59:40.249 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:59:40.250 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:40.250 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:59:40Z" 2022-09-09T22:59:40.250 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:59:40.252 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764373.377065_360.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:33.377065Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:59:40Z" } 2022-09-09T22:59:40.252 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764373.377065_360.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:33.377065Z', 'realm_epoch': 4}}] 2022-09-09T22:59:40.253 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764373.377065_360.1'} 2022-09-09T22:59:40.254 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764373.377065_360.1'} 2022-09-09T22:59:40.254 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-09T22:59:40.255 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-39 zones={test-zone2, test-zone1} 2022-09-09T22:59:40.255 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.256 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.256 DEBUG:boto:Method: HEAD 2022-09-09T22:59:40.257 DEBUG:boto:Path: /jkcgbj-39/ 2022-09-09T22:59:40.257 DEBUG:boto:Data: 2022-09-09T22:59:40.258 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.258 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:40.259 DEBUG:boto:Port: 8000 2022-09-09T22:59:40.259 DEBUG:boto:Params: {} 2022-09-09T22:59:40.260 DEBUG:boto:Token: None 2022-09-09T22:59:40.261 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/ 2022-09-09T22:59:40.261 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:glcR3o5yuCpAmcI1r/bOqrBYoeg= 2022-09-09T22:59:40.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:glcR3o5yuCpAmcI1r/bOqrBYoeg=', 'Content-Length': '0'} 2022-09-09T22:59:40.268 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055821188c6938ae8-00631bc55c-1549-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.269 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.270 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.270 DEBUG:boto:Method: HEAD 2022-09-09T22:59:40.271 DEBUG:boto:Path: /jkcgbj-39/ 2022-09-09T22:59:40.271 DEBUG:boto:Data: 2022-09-09T22:59:40.272 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.273 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:40.273 DEBUG:boto:Port: 8000 2022-09-09T22:59:40.274 DEBUG:boto:Params: {} 2022-09-09T22:59:40.275 DEBUG:boto:Token: None 2022-09-09T22:59:40.275 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/ 2022-09-09T22:59:40.276 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:glcR3o5yuCpAmcI1r/bOqrBYoeg= 2022-09-09T22:59:40.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:glcR3o5yuCpAmcI1r/bOqrBYoeg=', 'Content-Length': '0'} 2022-09-09T22:59:40.283 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d52809591a7019f-00631bc55c-13fd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.284 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:40.285 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.286 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.287 DEBUG:boto:path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.288 DEBUG:boto:auth_path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.289 DEBUG:boto:Method: GET 2022-09-09T22:59:40.290 DEBUG:boto:Path: /jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.291 DEBUG:boto:Data: 2022-09-09T22:59:40.292 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.293 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:40.294 DEBUG:boto:Port: 8000 2022-09-09T22:59:40.295 DEBUG:boto:Params: {} 2022-09-09T22:59:40.297 DEBUG:boto:Token: None 2022-09-09T22:59:40.298 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/?versions 2022-09-09T22:59:40.299 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis= 2022-09-09T22:59:40.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis=', 'Content-Length': '0'} 2022-09-09T22:59:40.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b0c7601f1fde271-00631bc55c-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.308 DEBUG:boto:b'jkcgbj-39999false' 2022-09-09T22:59:40.310 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:40.311 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.312 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.313 DEBUG:boto:path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.314 DEBUG:boto:auth_path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.315 DEBUG:boto:Method: GET 2022-09-09T22:59:40.316 DEBUG:boto:Path: /jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.317 DEBUG:boto:Data: 2022-09-09T22:59:40.318 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.319 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:40.319 DEBUG:boto:Port: 8000 2022-09-09T22:59:40.320 DEBUG:boto:Params: {} 2022-09-09T22:59:40.321 DEBUG:boto:Token: None 2022-09-09T22:59:40.322 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/?versions 2022-09-09T22:59:40.322 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis= 2022-09-09T22:59:40.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis=', 'Content-Length': '0'} 2022-09-09T22:59:40.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f507db7cd99a369b-00631bc55c-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.329 DEBUG:boto:b'jkcgbj-39999false' 2022-09-09T22:59:40.330 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.331 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.331 DEBUG:boto:path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.332 DEBUG:boto:auth_path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.332 DEBUG:boto:Method: GET 2022-09-09T22:59:40.333 DEBUG:boto:Path: /jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.333 DEBUG:boto:Data: 2022-09-09T22:59:40.334 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.335 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:40.335 DEBUG:boto:Port: 8000 2022-09-09T22:59:40.336 DEBUG:boto:Params: {} 2022-09-09T22:59:40.337 DEBUG:boto:Token: None 2022-09-09T22:59:40.337 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/?versions 2022-09-09T22:59:40.338 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis= 2022-09-09T22:59:40.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis=', 'Content-Length': '0'} 2022-09-09T22:59:40.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1d55b2bf5565042-00631bc55c-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.345 DEBUG:boto:b'jkcgbj-39999false' 2022-09-09T22:59:40.345 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.346 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.346 DEBUG:boto:path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.347 DEBUG:boto:auth_path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.347 DEBUG:boto:Method: GET 2022-09-09T22:59:40.348 DEBUG:boto:Path: /jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.349 DEBUG:boto:Data: 2022-09-09T22:59:40.349 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.350 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:40.351 DEBUG:boto:Port: 8000 2022-09-09T22:59:40.351 DEBUG:boto:Params: {} 2022-09-09T22:59:40.352 DEBUG:boto:Token: None 2022-09-09T22:59:40.352 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/?versions 2022-09-09T22:59:40.353 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis= 2022-09-09T22:59:40.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis=', 'Content-Length': '0'} 2022-09-09T22:59:40.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6712f1d87d77589-00631bc55c-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.360 DEBUG:boto:b'jkcgbj-39999false' 2022-09-09T22:59:40.361 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-39 zones={test-zone2, test-zone1} 2022-09-09T22:59:40.362 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-39 zones={test-zone3, test-zone1} 2022-09-09T22:59:40.362 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.363 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.364 DEBUG:boto:Method: HEAD 2022-09-09T22:59:40.365 DEBUG:boto:Path: /jkcgbj-39/ 2022-09-09T22:59:40.366 DEBUG:boto:Data: 2022-09-09T22:59:40.367 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.368 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:40.369 DEBUG:boto:Port: 8001 2022-09-09T22:59:40.370 DEBUG:boto:Params: {} 2022-09-09T22:59:40.371 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-09T22:59:40.372 DEBUG:boto:Token: None 2022-09-09T22:59:40.373 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/ 2022-09-09T22:59:40.375 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:glcR3o5yuCpAmcI1r/bOqrBYoeg= 2022-09-09T22:59:40.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:glcR3o5yuCpAmcI1r/bOqrBYoeg=', 'Content-Length': '0'} 2022-09-09T22:59:40.387 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000562c77cf2a2b2f78-00631bc55c-1403-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.388 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.388 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.389 DEBUG:boto:Method: HEAD 2022-09-09T22:59:40.389 DEBUG:boto:Path: /jkcgbj-39/ 2022-09-09T22:59:40.390 DEBUG:boto:Data: 2022-09-09T22:59:40.391 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.392 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:40.393 DEBUG:boto:Port: 8000 2022-09-09T22:59:40.394 DEBUG:boto:Params: {} 2022-09-09T22:59:40.395 DEBUG:boto:Token: None 2022-09-09T22:59:40.396 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/ 2022-09-09T22:59:40.397 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:glcR3o5yuCpAmcI1r/bOqrBYoeg= 2022-09-09T22:59:40.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:glcR3o5yuCpAmcI1r/bOqrBYoeg=', 'Content-Length': '0'} 2022-09-09T22:59:40.404 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000785e758b21762089-00631bc55c-13fd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.405 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:40.407 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.408 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.409 DEBUG:boto:path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.409 DEBUG:boto:auth_path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.410 DEBUG:boto:Method: GET 2022-09-09T22:59:40.411 DEBUG:boto:Path: /jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.412 DEBUG:boto:Data: 2022-09-09T22:59:40.413 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.414 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:40.414 DEBUG:boto:Port: 8001 2022-09-09T22:59:40.415 DEBUG:boto:Params: {} 2022-09-09T22:59:40.416 DEBUG:boto:Token: None 2022-09-09T22:59:40.416 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/?versions 2022-09-09T22:59:40.417 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis= 2022-09-09T22:59:40.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis=', 'Content-Length': '0'} 2022-09-09T22:59:40.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b27d0ca795bc09a-00631bc55c-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.425 DEBUG:boto:b'jkcgbj-39999false' 2022-09-09T22:59:40.427 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:40.427 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.428 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.429 DEBUG:boto:path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.430 DEBUG:boto:auth_path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.431 DEBUG:boto:Method: GET 2022-09-09T22:59:40.431 DEBUG:boto:Path: /jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.432 DEBUG:boto:Data: 2022-09-09T22:59:40.432 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.433 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:40.433 DEBUG:boto:Port: 8000 2022-09-09T22:59:40.434 DEBUG:boto:Params: {} 2022-09-09T22:59:40.435 DEBUG:boto:Token: None 2022-09-09T22:59:40.435 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/?versions 2022-09-09T22:59:40.436 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis= 2022-09-09T22:59:40.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis=', 'Content-Length': '0'} 2022-09-09T22:59:40.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005652e3c15e833993-00631bc55c-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.443 DEBUG:boto:b'jkcgbj-39999false' 2022-09-09T22:59:40.444 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.444 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.445 DEBUG:boto:path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.445 DEBUG:boto:auth_path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.446 DEBUG:boto:Method: GET 2022-09-09T22:59:40.447 DEBUG:boto:Path: /jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.448 DEBUG:boto:Data: 2022-09-09T22:59:40.448 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.449 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:40.449 DEBUG:boto:Port: 8001 2022-09-09T22:59:40.450 DEBUG:boto:Params: {} 2022-09-09T22:59:40.451 DEBUG:boto:Token: None 2022-09-09T22:59:40.451 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/?versions 2022-09-09T22:59:40.452 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis= 2022-09-09T22:59:40.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis=', 'Content-Length': '0'} 2022-09-09T22:59:40.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015b7f73beab18345-00631bc55c-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.460 DEBUG:boto:b'jkcgbj-39999false' 2022-09-09T22:59:40.461 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.461 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.462 DEBUG:boto:path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.463 DEBUG:boto:auth_path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.464 DEBUG:boto:Method: GET 2022-09-09T22:59:40.464 DEBUG:boto:Path: /jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.465 DEBUG:boto:Data: 2022-09-09T22:59:40.465 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.466 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:40.467 DEBUG:boto:Port: 8000 2022-09-09T22:59:40.467 DEBUG:boto:Params: {} 2022-09-09T22:59:40.468 DEBUG:boto:Token: None 2022-09-09T22:59:40.469 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/?versions 2022-09-09T22:59:40.469 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis= 2022-09-09T22:59:40.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis=', 'Content-Length': '0'} 2022-09-09T22:59:40.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000163a2c05b158242f-00631bc55c-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.522 DEBUG:boto:b'jkcgbj-39999false' 2022-09-09T22:59:40.524 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-39 zones={test-zone3, test-zone1} 2022-09-09T22:59:40.524 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-39 zones={test-zone1, test-zone2} 2022-09-09T22:59:40.525 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.525 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.526 DEBUG:boto:Method: HEAD 2022-09-09T22:59:40.527 DEBUG:boto:Path: /jkcgbj-39/ 2022-09-09T22:59:40.527 DEBUG:boto:Data: 2022-09-09T22:59:40.528 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.528 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:40.529 DEBUG:boto:Port: 8000 2022-09-09T22:59:40.529 DEBUG:boto:Params: {} 2022-09-09T22:59:40.530 DEBUG:boto:Token: None 2022-09-09T22:59:40.531 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/ 2022-09-09T22:59:40.531 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:glcR3o5yuCpAmcI1r/bOqrBYoeg= 2022-09-09T22:59:40.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:glcR3o5yuCpAmcI1r/bOqrBYoeg=', 'Content-Length': '0'} 2022-09-09T22:59:40.537 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e74a4b27dff2af4b-00631bc55c-13fd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.538 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.539 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.540 DEBUG:boto:Method: HEAD 2022-09-09T22:59:40.541 DEBUG:boto:Path: /jkcgbj-39/ 2022-09-09T22:59:40.542 DEBUG:boto:Data: 2022-09-09T22:59:40.543 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.544 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:40.545 DEBUG:boto:Port: 8000 2022-09-09T22:59:40.545 DEBUG:boto:Params: {} 2022-09-09T22:59:40.546 DEBUG:boto:Token: None 2022-09-09T22:59:40.547 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/ 2022-09-09T22:59:40.548 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:glcR3o5yuCpAmcI1r/bOqrBYoeg= 2022-09-09T22:59:40.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:glcR3o5yuCpAmcI1r/bOqrBYoeg=', 'Content-Length': '0'} 2022-09-09T22:59:40.553 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093525f49bc94cc0c-00631bc55c-1549-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.554 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:40.555 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.555 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.556 DEBUG:boto:path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.556 DEBUG:boto:auth_path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.557 DEBUG:boto:Method: GET 2022-09-09T22:59:40.557 DEBUG:boto:Path: /jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.558 DEBUG:boto:Data: 2022-09-09T22:59:40.558 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.559 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:40.559 DEBUG:boto:Port: 8000 2022-09-09T22:59:40.560 DEBUG:boto:Params: {} 2022-09-09T22:59:40.560 DEBUG:boto:Token: None 2022-09-09T22:59:40.561 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/?versions 2022-09-09T22:59:40.561 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis= 2022-09-09T22:59:40.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis=', 'Content-Length': '0'} 2022-09-09T22:59:40.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d080e37239a87b4f-00631bc55c-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.571 DEBUG:boto:b'jkcgbj-39999false' 2022-09-09T22:59:40.573 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:40.574 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.575 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.575 DEBUG:boto:path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.576 DEBUG:boto:auth_path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.577 DEBUG:boto:Method: GET 2022-09-09T22:59:40.578 DEBUG:boto:Path: /jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.579 DEBUG:boto:Data: 2022-09-09T22:59:40.580 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.581 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:40.582 DEBUG:boto:Port: 8000 2022-09-09T22:59:40.582 DEBUG:boto:Params: {} 2022-09-09T22:59:40.583 DEBUG:boto:Token: None 2022-09-09T22:59:40.583 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/?versions 2022-09-09T22:59:40.584 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis= 2022-09-09T22:59:40.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis=', 'Content-Length': '0'} 2022-09-09T22:59:40.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017a95678fe3343e5-00631bc55c-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.590 DEBUG:boto:b'jkcgbj-39999false' 2022-09-09T22:59:40.591 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.592 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.592 DEBUG:boto:path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.593 DEBUG:boto:auth_path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.593 DEBUG:boto:Method: GET 2022-09-09T22:59:40.594 DEBUG:boto:Path: /jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.595 DEBUG:boto:Data: 2022-09-09T22:59:40.596 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.597 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:40.597 DEBUG:boto:Port: 8000 2022-09-09T22:59:40.598 DEBUG:boto:Params: {} 2022-09-09T22:59:40.598 DEBUG:boto:Token: None 2022-09-09T22:59:40.599 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/?versions 2022-09-09T22:59:40.600 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis= 2022-09-09T22:59:40.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis=', 'Content-Length': '0'} 2022-09-09T22:59:40.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebe243b57811370e-00631bc55c-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.650 DEBUG:boto:b'jkcgbj-39999false' 2022-09-09T22:59:40.651 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.652 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.652 DEBUG:boto:path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.652 DEBUG:boto:auth_path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.653 DEBUG:boto:Method: GET 2022-09-09T22:59:40.654 DEBUG:boto:Path: /jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.654 DEBUG:boto:Data: 2022-09-09T22:59:40.655 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.655 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:40.656 DEBUG:boto:Port: 8000 2022-09-09T22:59:40.656 DEBUG:boto:Params: {} 2022-09-09T22:59:40.657 DEBUG:boto:Token: None 2022-09-09T22:59:40.657 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/?versions 2022-09-09T22:59:40.658 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis= 2022-09-09T22:59:40.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis=', 'Content-Length': '0'} 2022-09-09T22:59:40.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000becb3459228b078-00631bc55c-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.666 DEBUG:boto:b'jkcgbj-39999false' 2022-09-09T22:59:40.666 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-39 zones={test-zone1, test-zone2} 2022-09-09T22:59:40.667 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-39 zones={test-zone3, test-zone2} 2022-09-09T22:59:40.667 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.668 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.668 DEBUG:boto:Method: HEAD 2022-09-09T22:59:40.669 DEBUG:boto:Path: /jkcgbj-39/ 2022-09-09T22:59:40.669 DEBUG:boto:Data: 2022-09-09T22:59:40.670 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.671 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:40.671 DEBUG:boto:Port: 8001 2022-09-09T22:59:40.672 DEBUG:boto:Params: {} 2022-09-09T22:59:40.672 DEBUG:boto:Token: None 2022-09-09T22:59:40.673 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/ 2022-09-09T22:59:40.674 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:glcR3o5yuCpAmcI1r/bOqrBYoeg= 2022-09-09T22:59:40.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:glcR3o5yuCpAmcI1r/bOqrBYoeg=', 'Content-Length': '0'} 2022-09-09T22:59:40.680 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011f2f653844842b2-00631bc55c-1403-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.680 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.681 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.682 DEBUG:boto:Method: HEAD 2022-09-09T22:59:40.683 DEBUG:boto:Path: /jkcgbj-39/ 2022-09-09T22:59:40.684 DEBUG:boto:Data: 2022-09-09T22:59:40.685 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.685 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:40.687 DEBUG:boto:Port: 8000 2022-09-09T22:59:40.688 DEBUG:boto:Params: {} 2022-09-09T22:59:40.689 DEBUG:boto:Token: None 2022-09-09T22:59:40.690 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/ 2022-09-09T22:59:40.691 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:glcR3o5yuCpAmcI1r/bOqrBYoeg= 2022-09-09T22:59:40.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:glcR3o5yuCpAmcI1r/bOqrBYoeg=', 'Content-Length': '0'} 2022-09-09T22:59:40.697 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bcc5097d64d2c598-00631bc55c-1549-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.698 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:40.698 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.699 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.700 DEBUG:boto:path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.700 DEBUG:boto:auth_path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.701 DEBUG:boto:Method: GET 2022-09-09T22:59:40.701 DEBUG:boto:Path: /jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.702 DEBUG:boto:Data: 2022-09-09T22:59:40.703 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.704 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:40.705 DEBUG:boto:Port: 8001 2022-09-09T22:59:40.705 DEBUG:boto:Params: {} 2022-09-09T22:59:40.706 DEBUG:boto:Token: None 2022-09-09T22:59:40.707 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/?versions 2022-09-09T22:59:40.708 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis= 2022-09-09T22:59:40.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis=', 'Content-Length': '0'} 2022-09-09T22:59:40.715 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2c2855216773a6f-00631bc55c-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.758 DEBUG:boto:b'jkcgbj-39999false' 2022-09-09T22:59:40.759 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:40.760 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.760 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.761 DEBUG:boto:path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.761 DEBUG:boto:auth_path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.762 DEBUG:boto:Method: GET 2022-09-09T22:59:40.762 DEBUG:boto:Path: /jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.763 DEBUG:boto:Data: 2022-09-09T22:59:40.764 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.764 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:40.765 DEBUG:boto:Port: 8000 2022-09-09T22:59:40.766 DEBUG:boto:Params: {} 2022-09-09T22:59:40.767 DEBUG:boto:Token: None 2022-09-09T22:59:40.768 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/?versions 2022-09-09T22:59:40.769 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis= 2022-09-09T22:59:40.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis=', 'Content-Length': '0'} 2022-09-09T22:59:40.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbb0672405c80a1d-00631bc55c-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.777 DEBUG:boto:b'jkcgbj-39999false' 2022-09-09T22:59:40.778 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.779 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.779 DEBUG:boto:path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.780 DEBUG:boto:auth_path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.780 DEBUG:boto:Method: GET 2022-09-09T22:59:40.781 DEBUG:boto:Path: /jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.781 DEBUG:boto:Data: 2022-09-09T22:59:40.782 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.782 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:40.783 DEBUG:boto:Port: 8001 2022-09-09T22:59:40.783 DEBUG:boto:Params: {} 2022-09-09T22:59:40.784 DEBUG:boto:Token: None 2022-09-09T22:59:40.785 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/?versions 2022-09-09T22:59:40.786 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis= 2022-09-09T22:59:40.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis=', 'Content-Length': '0'} 2022-09-09T22:59:40.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d130b416e43f3ab6-00631bc55c-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.793 DEBUG:boto:b'jkcgbj-39999false' 2022-09-09T22:59:40.794 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.794 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.795 DEBUG:boto:path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.795 DEBUG:boto:auth_path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.796 DEBUG:boto:Method: GET 2022-09-09T22:59:40.797 DEBUG:boto:Path: /jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.797 DEBUG:boto:Data: 2022-09-09T22:59:40.798 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.798 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:40.799 DEBUG:boto:Port: 8000 2022-09-09T22:59:40.800 DEBUG:boto:Params: {} 2022-09-09T22:59:40.800 DEBUG:boto:Token: None 2022-09-09T22:59:40.801 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/?versions 2022-09-09T22:59:40.801 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis= 2022-09-09T22:59:40.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis=', 'Content-Length': '0'} 2022-09-09T22:59:40.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb9c2c38c5c9132b-00631bc55c-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.809 DEBUG:boto:b'jkcgbj-39999false' 2022-09-09T22:59:40.810 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-39 zones={test-zone3, test-zone2} 2022-09-09T22:59:40.811 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-39 zones={test-zone1, test-zone3} 2022-09-09T22:59:40.811 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.812 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.813 DEBUG:boto:Method: HEAD 2022-09-09T22:59:40.814 DEBUG:boto:Path: /jkcgbj-39/ 2022-09-09T22:59:40.815 DEBUG:boto:Data: 2022-09-09T22:59:40.815 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.816 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:40.817 DEBUG:boto:Port: 8000 2022-09-09T22:59:40.818 DEBUG:boto:Params: {} 2022-09-09T22:59:40.819 DEBUG:boto:Token: None 2022-09-09T22:59:40.820 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/ 2022-09-09T22:59:40.821 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:glcR3o5yuCpAmcI1r/bOqrBYoeg= 2022-09-09T22:59:40.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:glcR3o5yuCpAmcI1r/bOqrBYoeg=', 'Content-Length': '0'} 2022-09-09T22:59:40.829 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e96cb1b2985d76aa-00631bc55c-13fd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.830 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.831 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.831 DEBUG:boto:Method: HEAD 2022-09-09T22:59:40.832 DEBUG:boto:Path: /jkcgbj-39/ 2022-09-09T22:59:40.832 DEBUG:boto:Data: 2022-09-09T22:59:40.833 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.834 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:40.834 DEBUG:boto:Port: 8001 2022-09-09T22:59:40.835 DEBUG:boto:Params: {} 2022-09-09T22:59:40.835 DEBUG:boto:Token: None 2022-09-09T22:59:40.836 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/ 2022-09-09T22:59:40.836 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:glcR3o5yuCpAmcI1r/bOqrBYoeg= 2022-09-09T22:59:40.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:glcR3o5yuCpAmcI1r/bOqrBYoeg=', 'Content-Length': '0'} 2022-09-09T22:59:40.840 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002af02c72502bde4a-00631bc55c-1403-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.841 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:40.842 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.842 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.843 DEBUG:boto:path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.843 DEBUG:boto:auth_path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.844 DEBUG:boto:Method: GET 2022-09-09T22:59:40.844 DEBUG:boto:Path: /jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.845 DEBUG:boto:Data: 2022-09-09T22:59:40.845 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.846 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:40.847 DEBUG:boto:Port: 8000 2022-09-09T22:59:40.847 DEBUG:boto:Params: {} 2022-09-09T22:59:40.848 DEBUG:boto:Token: None 2022-09-09T22:59:40.849 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/?versions 2022-09-09T22:59:40.849 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis= 2022-09-09T22:59:40.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis=', 'Content-Length': '0'} 2022-09-09T22:59:40.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b446689c23f540b-00631bc55c-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.856 DEBUG:boto:b'jkcgbj-39999false' 2022-09-09T22:59:40.857 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:40.857 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.858 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.858 DEBUG:boto:path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.859 DEBUG:boto:auth_path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.859 DEBUG:boto:Method: GET 2022-09-09T22:59:40.860 DEBUG:boto:Path: /jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.861 DEBUG:boto:Data: 2022-09-09T22:59:40.861 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.862 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:40.863 DEBUG:boto:Port: 8001 2022-09-09T22:59:40.863 DEBUG:boto:Params: {} 2022-09-09T22:59:40.864 DEBUG:boto:Token: None 2022-09-09T22:59:40.864 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/?versions 2022-09-09T22:59:40.865 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis= 2022-09-09T22:59:40.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis=', 'Content-Length': '0'} 2022-09-09T22:59:40.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024d081d5d6b67145-00631bc55c-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.872 DEBUG:boto:b'jkcgbj-39999false' 2022-09-09T22:59:40.874 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.875 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.875 DEBUG:boto:path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.876 DEBUG:boto:auth_path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.877 DEBUG:boto:Method: GET 2022-09-09T22:59:40.878 DEBUG:boto:Path: /jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.879 DEBUG:boto:Data: 2022-09-09T22:59:40.879 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.880 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:40.880 DEBUG:boto:Port: 8000 2022-09-09T22:59:40.881 DEBUG:boto:Params: {} 2022-09-09T22:59:40.882 DEBUG:boto:Token: None 2022-09-09T22:59:40.882 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/?versions 2022-09-09T22:59:40.883 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis= 2022-09-09T22:59:40.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis=', 'Content-Length': '0'} 2022-09-09T22:59:40.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ebb579a9ffcb983-00631bc55c-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.890 DEBUG:boto:b'jkcgbj-39999false' 2022-09-09T22:59:40.891 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.892 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.892 DEBUG:boto:path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.893 DEBUG:boto:auth_path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.893 DEBUG:boto:Method: GET 2022-09-09T22:59:40.894 DEBUG:boto:Path: /jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.895 DEBUG:boto:Data: 2022-09-09T22:59:40.895 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.896 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:40.896 DEBUG:boto:Port: 8001 2022-09-09T22:59:40.897 DEBUG:boto:Params: {} 2022-09-09T22:59:40.897 DEBUG:boto:Token: None 2022-09-09T22:59:40.898 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/?versions 2022-09-09T22:59:40.899 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis= 2022-09-09T22:59:40.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis=', 'Content-Length': '0'} 2022-09-09T22:59:40.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000121a8d88d21a58d2-00631bc55c-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.906 DEBUG:boto:b'jkcgbj-39999false' 2022-09-09T22:59:40.907 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-39 zones={test-zone1, test-zone3} 2022-09-09T22:59:40.908 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-39 zones={test-zone2, test-zone3} 2022-09-09T22:59:40.908 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.909 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.909 DEBUG:boto:Method: HEAD 2022-09-09T22:59:40.910 DEBUG:boto:Path: /jkcgbj-39/ 2022-09-09T22:59:40.910 DEBUG:boto:Data: 2022-09-09T22:59:40.911 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.911 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:40.912 DEBUG:boto:Port: 8000 2022-09-09T22:59:40.912 DEBUG:boto:Params: {} 2022-09-09T22:59:40.913 DEBUG:boto:Token: None 2022-09-09T22:59:40.914 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/ 2022-09-09T22:59:40.914 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:glcR3o5yuCpAmcI1r/bOqrBYoeg= 2022-09-09T22:59:40.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:glcR3o5yuCpAmcI1r/bOqrBYoeg=', 'Content-Length': '0'} 2022-09-09T22:59:40.919 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d928a80913c09cd6-00631bc55c-1549-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.920 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.921 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.921 DEBUG:boto:Method: HEAD 2022-09-09T22:59:40.922 DEBUG:boto:Path: /jkcgbj-39/ 2022-09-09T22:59:40.923 DEBUG:boto:Data: 2022-09-09T22:59:40.923 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.924 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:40.925 DEBUG:boto:Port: 8001 2022-09-09T22:59:40.925 DEBUG:boto:Params: {} 2022-09-09T22:59:40.926 DEBUG:boto:Token: None 2022-09-09T22:59:40.927 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/ 2022-09-09T22:59:40.927 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:glcR3o5yuCpAmcI1r/bOqrBYoeg= 2022-09-09T22:59:40.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:glcR3o5yuCpAmcI1r/bOqrBYoeg=', 'Content-Length': '0'} 2022-09-09T22:59:40.934 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b66f707ec2ca4a08-00631bc55c-1403-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.934 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:40.935 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.936 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.937 DEBUG:boto:path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.938 DEBUG:boto:auth_path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.939 DEBUG:boto:Method: GET 2022-09-09T22:59:40.939 DEBUG:boto:Path: /jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.940 DEBUG:boto:Data: 2022-09-09T22:59:40.940 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.941 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:40.942 DEBUG:boto:Port: 8000 2022-09-09T22:59:40.942 DEBUG:boto:Params: {} 2022-09-09T22:59:40.943 DEBUG:boto:Token: None 2022-09-09T22:59:40.943 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/?versions 2022-09-09T22:59:40.944 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis= 2022-09-09T22:59:40.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis=', 'Content-Length': '0'} 2022-09-09T22:59:40.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075532c6e79c92407-00631bc55c-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:40.951 DEBUG:boto:b'jkcgbj-39999false' 2022-09-09T22:59:40.952 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:40.952 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:40.953 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:40.953 DEBUG:boto:path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.954 DEBUG:boto:auth_path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.954 DEBUG:boto:Method: GET 2022-09-09T22:59:40.955 DEBUG:boto:Path: /jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:40.956 DEBUG:boto:Data: 2022-09-09T22:59:40.956 DEBUG:boto:Headers: {} 2022-09-09T22:59:40.957 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:40.957 DEBUG:boto:Port: 8001 2022-09-09T22:59:40.958 DEBUG:boto:Params: {} 2022-09-09T22:59:40.959 DEBUG:boto:Token: None 2022-09-09T22:59:40.960 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:40 GMT /jkcgbj-39/?versions 2022-09-09T22:59:40.961 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis= 2022-09-09T22:59:40.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:40 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qcYB0vobUtF7TtUFtIm5Rbn7jis=', 'Content-Length': '0'} 2022-09-09T22:59:40.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bd7cd703a82b4b3-00631bc55c-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.014 DEBUG:boto:b'jkcgbj-39999false' 2022-09-09T22:59:41.015 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:41.016 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:41.016 DEBUG:boto:path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:41.017 DEBUG:boto:auth_path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:41.017 DEBUG:boto:Method: GET 2022-09-09T22:59:41.018 DEBUG:boto:Path: /jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:41.018 DEBUG:boto:Data: 2022-09-09T22:59:41.019 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.019 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:41.020 DEBUG:boto:Port: 8000 2022-09-09T22:59:41.021 DEBUG:boto:Params: {} 2022-09-09T22:59:41.021 DEBUG:boto:Token: None 2022-09-09T22:59:41.022 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-39/?versions 2022-09-09T22:59:41.023 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:U4y4Caxhqc560XLoVhw+S8ESZaY= 2022-09-09T22:59:41.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:U4y4Caxhqc560XLoVhw+S8ESZaY=', 'Content-Length': '0'} 2022-09-09T22:59:41.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000495c6f5281b0b731-00631bc55d-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.030 DEBUG:boto:b'jkcgbj-39999false' 2022-09-09T22:59:41.031 DEBUG:boto:path=/jkcgbj-39/ 2022-09-09T22:59:41.032 DEBUG:boto:auth_path=/jkcgbj-39/ 2022-09-09T22:59:41.033 DEBUG:boto:path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:41.034 DEBUG:boto:auth_path=/jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:41.035 DEBUG:boto:Method: GET 2022-09-09T22:59:41.036 DEBUG:boto:Path: /jkcgbj-39/?versions&max-keys=999 2022-09-09T22:59:41.037 DEBUG:boto:Data: 2022-09-09T22:59:41.038 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.039 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:41.039 DEBUG:boto:Port: 8001 2022-09-09T22:59:41.040 DEBUG:boto:Params: {} 2022-09-09T22:59:41.041 DEBUG:boto:Token: None 2022-09-09T22:59:41.041 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-39/?versions 2022-09-09T22:59:41.042 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:U4y4Caxhqc560XLoVhw+S8ESZaY= 2022-09-09T22:59:41.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:U4y4Caxhqc560XLoVhw+S8ESZaY=', 'Content-Length': '0'} 2022-09-09T22:59:41.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000633aa3e030d0acc6-00631bc55d-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.049 DEBUG:boto:b'jkcgbj-39999false' 2022-09-09T22:59:41.050 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-39 zones={test-zone2, test-zone3} 2022-09-09T22:59:41.051 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-40 zones={test-zone2, test-zone1} 2022-09-09T22:59:41.052 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.053 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.054 DEBUG:boto:Method: HEAD 2022-09-09T22:59:41.054 DEBUG:boto:Path: /jkcgbj-40/ 2022-09-09T22:59:41.055 DEBUG:boto:Data: 2022-09-09T22:59:41.055 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.056 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:41.056 DEBUG:boto:Port: 8000 2022-09-09T22:59:41.057 DEBUG:boto:Params: {} 2022-09-09T22:59:41.058 DEBUG:boto:Token: None 2022-09-09T22:59:41.058 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/ 2022-09-09T22:59:41.059 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FrX2AJroD3tjf6JQY4qrUOt3ApU= 2022-09-09T22:59:41.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FrX2AJroD3tjf6JQY4qrUOt3ApU=', 'Content-Length': '0'} 2022-09-09T22:59:41.066 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000474eca00ee263707-00631bc55d-1549-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.067 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.067 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.068 DEBUG:boto:Method: HEAD 2022-09-09T22:59:41.068 DEBUG:boto:Path: /jkcgbj-40/ 2022-09-09T22:59:41.069 DEBUG:boto:Data: 2022-09-09T22:59:41.069 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.070 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:41.070 DEBUG:boto:Port: 8000 2022-09-09T22:59:41.071 DEBUG:boto:Params: {} 2022-09-09T22:59:41.072 DEBUG:boto:Token: None 2022-09-09T22:59:41.072 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/ 2022-09-09T22:59:41.073 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FrX2AJroD3tjf6JQY4qrUOt3ApU= 2022-09-09T22:59:41.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FrX2AJroD3tjf6JQY4qrUOt3ApU=', 'Content-Length': '0'} 2022-09-09T22:59:41.079 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c17e26c57c55754-00631bc55d-13fd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.080 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:41.080 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.081 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.082 DEBUG:boto:path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.082 DEBUG:boto:auth_path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.083 DEBUG:boto:Method: GET 2022-09-09T22:59:41.084 DEBUG:boto:Path: /jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.084 DEBUG:boto:Data: 2022-09-09T22:59:41.085 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.086 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:41.086 DEBUG:boto:Port: 8000 2022-09-09T22:59:41.087 DEBUG:boto:Params: {} 2022-09-09T22:59:41.088 DEBUG:boto:Token: None 2022-09-09T22:59:41.088 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/?versions 2022-09-09T22:59:41.089 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8= 2022-09-09T22:59:41.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8=', 'Content-Length': '0'} 2022-09-09T22:59:41.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f46125cc5a83dcb-00631bc55d-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.138 DEBUG:boto:b'jkcgbj-40999false' 2022-09-09T22:59:41.139 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:41.140 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.140 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.141 DEBUG:boto:path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.141 DEBUG:boto:auth_path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.142 DEBUG:boto:Method: GET 2022-09-09T22:59:41.143 DEBUG:boto:Path: /jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.143 DEBUG:boto:Data: 2022-09-09T22:59:41.144 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.145 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:41.146 DEBUG:boto:Port: 8000 2022-09-09T22:59:41.147 DEBUG:boto:Params: {} 2022-09-09T22:59:41.148 DEBUG:boto:Token: None 2022-09-09T22:59:41.149 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/?versions 2022-09-09T22:59:41.150 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8= 2022-09-09T22:59:41.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8=', 'Content-Length': '0'} 2022-09-09T22:59:41.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f53e546fc93d8611-00631bc55d-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.157 DEBUG:boto:b'jkcgbj-40999false' 2022-09-09T22:59:41.158 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.158 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.159 DEBUG:boto:path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.159 DEBUG:boto:auth_path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.160 DEBUG:boto:Method: GET 2022-09-09T22:59:41.161 DEBUG:boto:Path: /jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.164 DEBUG:boto:Data: 2022-09-09T22:59:41.165 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.165 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:41.166 DEBUG:boto:Port: 8000 2022-09-09T22:59:41.166 DEBUG:boto:Params: {} 2022-09-09T22:59:41.167 DEBUG:boto:Token: None 2022-09-09T22:59:41.168 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/?versions 2022-09-09T22:59:41.170 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8= 2022-09-09T22:59:41.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8=', 'Content-Length': '0'} 2022-09-09T22:59:41.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b093b40d1fd77511-00631bc55d-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.177 DEBUG:boto:b'jkcgbj-40999false' 2022-09-09T22:59:41.177 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.178 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.179 DEBUG:boto:path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.179 DEBUG:boto:auth_path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.180 DEBUG:boto:Method: GET 2022-09-09T22:59:41.180 DEBUG:boto:Path: /jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.181 DEBUG:boto:Data: 2022-09-09T22:59:41.181 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.184 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:41.184 DEBUG:boto:Port: 8000 2022-09-09T22:59:41.185 DEBUG:boto:Params: {} 2022-09-09T22:59:41.186 DEBUG:boto:Token: None 2022-09-09T22:59:41.186 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/?versions 2022-09-09T22:59:41.187 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8= 2022-09-09T22:59:41.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8=', 'Content-Length': '0'} 2022-09-09T22:59:41.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff2ffe07d2e15e34-00631bc55d-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.234 DEBUG:boto:b'jkcgbj-40999false' 2022-09-09T22:59:41.235 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-40 zones={test-zone2, test-zone1} 2022-09-09T22:59:41.236 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-40 zones={test-zone3, test-zone1} 2022-09-09T22:59:41.237 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.237 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.238 DEBUG:boto:Method: HEAD 2022-09-09T22:59:41.239 DEBUG:boto:Path: /jkcgbj-40/ 2022-09-09T22:59:41.239 DEBUG:boto:Data: 2022-09-09T22:59:41.240 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.240 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:41.241 DEBUG:boto:Port: 8001 2022-09-09T22:59:41.241 DEBUG:boto:Params: {} 2022-09-09T22:59:41.242 DEBUG:boto:Token: None 2022-09-09T22:59:41.242 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/ 2022-09-09T22:59:41.243 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FrX2AJroD3tjf6JQY4qrUOt3ApU= 2022-09-09T22:59:41.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FrX2AJroD3tjf6JQY4qrUOt3ApU=', 'Content-Length': '0'} 2022-09-09T22:59:41.251 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007e39040a298e5440-00631bc55d-1403-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.252 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.253 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.253 DEBUG:boto:Method: HEAD 2022-09-09T22:59:41.254 DEBUG:boto:Path: /jkcgbj-40/ 2022-09-09T22:59:41.256 DEBUG:boto:Data: 2022-09-09T22:59:41.256 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.257 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:41.257 DEBUG:boto:Port: 8000 2022-09-09T22:59:41.258 DEBUG:boto:Params: {} 2022-09-09T22:59:41.259 DEBUG:boto:Token: None 2022-09-09T22:59:41.260 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/ 2022-09-09T22:59:41.260 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FrX2AJroD3tjf6JQY4qrUOt3ApU= 2022-09-09T22:59:41.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FrX2AJroD3tjf6JQY4qrUOt3ApU=', 'Content-Length': '0'} 2022-09-09T22:59:41.265 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c392faee3ee7abe4-00631bc55d-13fd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.266 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:41.267 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.267 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.268 DEBUG:boto:path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.268 DEBUG:boto:auth_path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.269 DEBUG:boto:Method: GET 2022-09-09T22:59:41.269 DEBUG:boto:Path: /jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.270 DEBUG:boto:Data: 2022-09-09T22:59:41.270 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.271 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:41.271 DEBUG:boto:Port: 8001 2022-09-09T22:59:41.272 DEBUG:boto:Params: {} 2022-09-09T22:59:41.272 DEBUG:boto:Token: None 2022-09-09T22:59:41.273 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/?versions 2022-09-09T22:59:41.273 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8= 2022-09-09T22:59:41.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8=', 'Content-Length': '0'} 2022-09-09T22:59:41.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a5e48811e5bf91a-00631bc55d-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.281 DEBUG:boto:b'jkcgbj-40999false' 2022-09-09T22:59:41.282 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:41.282 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.283 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.284 DEBUG:boto:path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.285 DEBUG:boto:auth_path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.285 DEBUG:boto:Method: GET 2022-09-09T22:59:41.286 DEBUG:boto:Path: /jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.286 DEBUG:boto:Data: 2022-09-09T22:59:41.287 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.287 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:41.288 DEBUG:boto:Port: 8000 2022-09-09T22:59:41.288 DEBUG:boto:Params: {} 2022-09-09T22:59:41.289 DEBUG:boto:Token: None 2022-09-09T22:59:41.290 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/?versions 2022-09-09T22:59:41.290 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8= 2022-09-09T22:59:41.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8=', 'Content-Length': '0'} 2022-09-09T22:59:41.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ce472a040b3b05e-00631bc55d-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.298 DEBUG:boto:b'jkcgbj-40999false' 2022-09-09T22:59:41.298 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.300 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.300 DEBUG:boto:path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.301 DEBUG:boto:auth_path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.301 DEBUG:boto:Method: GET 2022-09-09T22:59:41.302 DEBUG:boto:Path: /jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.302 DEBUG:boto:Data: 2022-09-09T22:59:41.303 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.304 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:41.304 DEBUG:boto:Port: 8001 2022-09-09T22:59:41.305 DEBUG:boto:Params: {} 2022-09-09T22:59:41.305 DEBUG:boto:Token: None 2022-09-09T22:59:41.306 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/?versions 2022-09-09T22:59:41.306 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8= 2022-09-09T22:59:41.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8=', 'Content-Length': '0'} 2022-09-09T22:59:41.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c47b5a7beb19459-00631bc55d-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.354 DEBUG:boto:b'jkcgbj-40999false' 2022-09-09T22:59:41.355 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.356 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.356 DEBUG:boto:path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.357 DEBUG:boto:auth_path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.357 DEBUG:boto:Method: GET 2022-09-09T22:59:41.358 DEBUG:boto:Path: /jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.358 DEBUG:boto:Data: 2022-09-09T22:59:41.359 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.359 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:41.360 DEBUG:boto:Port: 8000 2022-09-09T22:59:41.361 DEBUG:boto:Params: {} 2022-09-09T22:59:41.361 DEBUG:boto:Token: None 2022-09-09T22:59:41.362 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/?versions 2022-09-09T22:59:41.363 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8= 2022-09-09T22:59:41.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8=', 'Content-Length': '0'} 2022-09-09T22:59:41.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd2efd6d3b471965-00631bc55d-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.370 DEBUG:boto:b'jkcgbj-40999false' 2022-09-09T22:59:41.371 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-40 zones={test-zone3, test-zone1} 2022-09-09T22:59:41.371 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-40 zones={test-zone1, test-zone2} 2022-09-09T22:59:41.372 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.373 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.373 DEBUG:boto:Method: HEAD 2022-09-09T22:59:41.374 DEBUG:boto:Path: /jkcgbj-40/ 2022-09-09T22:59:41.374 DEBUG:boto:Data: 2022-09-09T22:59:41.375 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.375 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:41.376 DEBUG:boto:Port: 8000 2022-09-09T22:59:41.377 DEBUG:boto:Params: {} 2022-09-09T22:59:41.377 DEBUG:boto:Token: None 2022-09-09T22:59:41.378 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/ 2022-09-09T22:59:41.379 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FrX2AJroD3tjf6JQY4qrUOt3ApU= 2022-09-09T22:59:41.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FrX2AJroD3tjf6JQY4qrUOt3ApU=', 'Content-Length': '0'} 2022-09-09T22:59:41.384 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000994aeefd27f7368f-00631bc55d-13fd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.385 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.386 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.386 DEBUG:boto:Method: HEAD 2022-09-09T22:59:41.387 DEBUG:boto:Path: /jkcgbj-40/ 2022-09-09T22:59:41.387 DEBUG:boto:Data: 2022-09-09T22:59:41.388 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.388 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:41.389 DEBUG:boto:Port: 8000 2022-09-09T22:59:41.389 DEBUG:boto:Params: {} 2022-09-09T22:59:41.390 DEBUG:boto:Token: None 2022-09-09T22:59:41.391 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/ 2022-09-09T22:59:41.392 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FrX2AJroD3tjf6JQY4qrUOt3ApU= 2022-09-09T22:59:41.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FrX2AJroD3tjf6JQY4qrUOt3ApU=', 'Content-Length': '0'} 2022-09-09T22:59:41.398 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008328c6eec797de84-00631bc55d-1549-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.398 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:41.399 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.399 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.400 DEBUG:boto:path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.400 DEBUG:boto:auth_path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.401 DEBUG:boto:Method: GET 2022-09-09T22:59:41.401 DEBUG:boto:Path: /jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.402 DEBUG:boto:Data: 2022-09-09T22:59:41.402 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.403 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:41.403 DEBUG:boto:Port: 8000 2022-09-09T22:59:41.404 DEBUG:boto:Params: {} 2022-09-09T22:59:41.404 DEBUG:boto:Token: None 2022-09-09T22:59:41.405 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/?versions 2022-09-09T22:59:41.406 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8= 2022-09-09T22:59:41.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8=', 'Content-Length': '0'} 2022-09-09T22:59:41.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dc4383a637f86c6-00631bc55d-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.458 DEBUG:boto:b'jkcgbj-40999false' 2022-09-09T22:59:41.460 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:41.461 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.461 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.462 DEBUG:boto:path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.462 DEBUG:boto:auth_path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.463 DEBUG:boto:Method: GET 2022-09-09T22:59:41.463 DEBUG:boto:Path: /jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.464 DEBUG:boto:Data: 2022-09-09T22:59:41.464 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.465 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:41.465 DEBUG:boto:Port: 8000 2022-09-09T22:59:41.466 DEBUG:boto:Params: {} 2022-09-09T22:59:41.467 DEBUG:boto:Token: None 2022-09-09T22:59:41.467 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/?versions 2022-09-09T22:59:41.468 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8= 2022-09-09T22:59:41.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8=', 'Content-Length': '0'} 2022-09-09T22:59:41.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006760c1db652d0e3b-00631bc55d-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.475 DEBUG:boto:b'jkcgbj-40999false' 2022-09-09T22:59:41.476 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.477 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.477 DEBUG:boto:path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.478 DEBUG:boto:auth_path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.478 DEBUG:boto:Method: GET 2022-09-09T22:59:41.479 DEBUG:boto:Path: /jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.479 DEBUG:boto:Data: 2022-09-09T22:59:41.480 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.481 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:41.481 DEBUG:boto:Port: 8000 2022-09-09T22:59:41.482 DEBUG:boto:Params: {} 2022-09-09T22:59:41.482 DEBUG:boto:Token: None 2022-09-09T22:59:41.483 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/?versions 2022-09-09T22:59:41.484 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8= 2022-09-09T22:59:41.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8=', 'Content-Length': '0'} 2022-09-09T22:59:41.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ba3e6853c590c38-00631bc55d-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.491 DEBUG:boto:b'jkcgbj-40999false' 2022-09-09T22:59:41.492 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.492 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.493 DEBUG:boto:path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.493 DEBUG:boto:auth_path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.494 DEBUG:boto:Method: GET 2022-09-09T22:59:41.495 DEBUG:boto:Path: /jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.495 DEBUG:boto:Data: 2022-09-09T22:59:41.496 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.496 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:41.497 DEBUG:boto:Port: 8000 2022-09-09T22:59:41.498 DEBUG:boto:Params: {} 2022-09-09T22:59:41.498 DEBUG:boto:Token: None 2022-09-09T22:59:41.499 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/?versions 2022-09-09T22:59:41.500 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8= 2022-09-09T22:59:41.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8=', 'Content-Length': '0'} 2022-09-09T22:59:41.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df946c409450039d-00631bc55d-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.506 DEBUG:boto:b'jkcgbj-40999false' 2022-09-09T22:59:41.507 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-40 zones={test-zone1, test-zone2} 2022-09-09T22:59:41.508 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-40 zones={test-zone3, test-zone2} 2022-09-09T22:59:41.508 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.509 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.510 DEBUG:boto:Method: HEAD 2022-09-09T22:59:41.510 DEBUG:boto:Path: /jkcgbj-40/ 2022-09-09T22:59:41.511 DEBUG:boto:Data: 2022-09-09T22:59:41.511 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.512 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:41.513 DEBUG:boto:Port: 8001 2022-09-09T22:59:41.513 DEBUG:boto:Params: {} 2022-09-09T22:59:41.514 DEBUG:boto:Token: None 2022-09-09T22:59:41.514 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/ 2022-09-09T22:59:41.515 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FrX2AJroD3tjf6JQY4qrUOt3ApU= 2022-09-09T22:59:41.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FrX2AJroD3tjf6JQY4qrUOt3ApU=', 'Content-Length': '0'} 2022-09-09T22:59:41.521 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b99cbf2fdd279f74-00631bc55d-1403-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.521 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.522 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.523 DEBUG:boto:Method: HEAD 2022-09-09T22:59:41.523 DEBUG:boto:Path: /jkcgbj-40/ 2022-09-09T22:59:41.524 DEBUG:boto:Data: 2022-09-09T22:59:41.524 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.525 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:41.525 DEBUG:boto:Port: 8000 2022-09-09T22:59:41.526 DEBUG:boto:Params: {} 2022-09-09T22:59:41.526 DEBUG:boto:Token: None 2022-09-09T22:59:41.527 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/ 2022-09-09T22:59:41.527 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FrX2AJroD3tjf6JQY4qrUOt3ApU= 2022-09-09T22:59:41.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FrX2AJroD3tjf6JQY4qrUOt3ApU=', 'Content-Length': '0'} 2022-09-09T22:59:41.533 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e88fa4f1baf102fd-00631bc55d-1549-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.534 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:41.535 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.535 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.536 DEBUG:boto:path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.536 DEBUG:boto:auth_path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.537 DEBUG:boto:Method: GET 2022-09-09T22:59:41.537 DEBUG:boto:Path: /jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.538 DEBUG:boto:Data: 2022-09-09T22:59:41.538 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.539 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:41.540 DEBUG:boto:Port: 8001 2022-09-09T22:59:41.540 DEBUG:boto:Params: {} 2022-09-09T22:59:41.541 DEBUG:boto:Token: None 2022-09-09T22:59:41.541 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/?versions 2022-09-09T22:59:41.542 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8= 2022-09-09T22:59:41.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8=', 'Content-Length': '0'} 2022-09-09T22:59:41.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d374c9c76306d38e-00631bc55d-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.550 DEBUG:boto:b'jkcgbj-40999false' 2022-09-09T22:59:41.551 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:41.552 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.553 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.554 DEBUG:boto:path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.554 DEBUG:boto:auth_path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.555 DEBUG:boto:Method: GET 2022-09-09T22:59:41.555 DEBUG:boto:Path: /jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.556 DEBUG:boto:Data: 2022-09-09T22:59:41.556 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.557 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:41.557 DEBUG:boto:Port: 8000 2022-09-09T22:59:41.558 DEBUG:boto:Params: {} 2022-09-09T22:59:41.559 DEBUG:boto:Token: None 2022-09-09T22:59:41.559 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/?versions 2022-09-09T22:59:41.560 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8= 2022-09-09T22:59:41.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8=', 'Content-Length': '0'} 2022-09-09T22:59:41.567 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b67eed51b4245816-00631bc55d-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.610 DEBUG:boto:b'jkcgbj-40999false' 2022-09-09T22:59:41.612 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.612 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.614 DEBUG:boto:path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.614 DEBUG:boto:auth_path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.615 DEBUG:boto:Method: GET 2022-09-09T22:59:41.616 DEBUG:boto:Path: /jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.618 DEBUG:boto:Data: 2022-09-09T22:59:41.619 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.619 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:41.620 DEBUG:boto:Port: 8001 2022-09-09T22:59:41.621 DEBUG:boto:Params: {} 2022-09-09T22:59:41.622 DEBUG:boto:Token: None 2022-09-09T22:59:41.622 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/?versions 2022-09-09T22:59:41.623 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8= 2022-09-09T22:59:41.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8=', 'Content-Length': '0'} 2022-09-09T22:59:41.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af109562973f56bb-00631bc55d-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.631 DEBUG:boto:b'jkcgbj-40999false' 2022-09-09T22:59:41.632 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.633 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.634 DEBUG:boto:path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.634 DEBUG:boto:auth_path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.635 DEBUG:boto:Method: GET 2022-09-09T22:59:41.635 DEBUG:boto:Path: /jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.636 DEBUG:boto:Data: 2022-09-09T22:59:41.636 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.637 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:41.637 DEBUG:boto:Port: 8000 2022-09-09T22:59:41.638 DEBUG:boto:Params: {} 2022-09-09T22:59:41.638 DEBUG:boto:Token: None 2022-09-09T22:59:41.639 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/?versions 2022-09-09T22:59:41.640 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8= 2022-09-09T22:59:41.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8=', 'Content-Length': '0'} 2022-09-09T22:59:41.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000996e319a3d3ecf15-00631bc55d-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.647 DEBUG:boto:b'jkcgbj-40999false' 2022-09-09T22:59:41.648 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-40 zones={test-zone3, test-zone2} 2022-09-09T22:59:41.648 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-40 zones={test-zone1, test-zone3} 2022-09-09T22:59:41.649 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.649 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.650 DEBUG:boto:Method: HEAD 2022-09-09T22:59:41.651 DEBUG:boto:Path: /jkcgbj-40/ 2022-09-09T22:59:41.651 DEBUG:boto:Data: 2022-09-09T22:59:41.652 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.652 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:41.653 DEBUG:boto:Port: 8000 2022-09-09T22:59:41.653 DEBUG:boto:Params: {} 2022-09-09T22:59:41.654 DEBUG:boto:Token: None 2022-09-09T22:59:41.654 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/ 2022-09-09T22:59:41.655 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FrX2AJroD3tjf6JQY4qrUOt3ApU= 2022-09-09T22:59:41.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FrX2AJroD3tjf6JQY4qrUOt3ApU=', 'Content-Length': '0'} 2022-09-09T22:59:41.659 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000daa3f0b11791d891-00631bc55d-13fd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.659 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.660 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.660 DEBUG:boto:Method: HEAD 2022-09-09T22:59:41.661 DEBUG:boto:Path: /jkcgbj-40/ 2022-09-09T22:59:41.661 DEBUG:boto:Data: 2022-09-09T22:59:41.662 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.662 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:41.663 DEBUG:boto:Port: 8001 2022-09-09T22:59:41.663 DEBUG:boto:Params: {} 2022-09-09T22:59:41.664 DEBUG:boto:Token: None 2022-09-09T22:59:41.664 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/ 2022-09-09T22:59:41.665 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FrX2AJroD3tjf6JQY4qrUOt3ApU= 2022-09-09T22:59:41.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FrX2AJroD3tjf6JQY4qrUOt3ApU=', 'Content-Length': '0'} 2022-09-09T22:59:41.673 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000659ca5d98827edd7-00631bc55d-1403-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.673 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:41.674 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.675 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.675 DEBUG:boto:path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.676 DEBUG:boto:auth_path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.676 DEBUG:boto:Method: GET 2022-09-09T22:59:41.677 DEBUG:boto:Path: /jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.677 DEBUG:boto:Data: 2022-09-09T22:59:41.678 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.678 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:41.679 DEBUG:boto:Port: 8000 2022-09-09T22:59:41.679 DEBUG:boto:Params: {} 2022-09-09T22:59:41.680 DEBUG:boto:Token: None 2022-09-09T22:59:41.681 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/?versions 2022-09-09T22:59:41.682 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8= 2022-09-09T22:59:41.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8=', 'Content-Length': '0'} 2022-09-09T22:59:41.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd6311faf0da7999-00631bc55d-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.687 DEBUG:boto:b'jkcgbj-40999false' 2022-09-09T22:59:41.687 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:41.688 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.688 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.689 DEBUG:boto:path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.689 DEBUG:boto:auth_path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.690 DEBUG:boto:Method: GET 2022-09-09T22:59:41.690 DEBUG:boto:Path: /jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.691 DEBUG:boto:Data: 2022-09-09T22:59:41.691 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.692 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:41.692 DEBUG:boto:Port: 8001 2022-09-09T22:59:41.693 DEBUG:boto:Params: {} 2022-09-09T22:59:41.693 DEBUG:boto:Token: None 2022-09-09T22:59:41.694 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/?versions 2022-09-09T22:59:41.695 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8= 2022-09-09T22:59:41.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8=', 'Content-Length': '0'} 2022-09-09T22:59:41.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cce6bb4ecc8ef480-00631bc55d-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.742 DEBUG:boto:b'jkcgbj-40999false' 2022-09-09T22:59:41.744 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.745 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.745 DEBUG:boto:path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.746 DEBUG:boto:auth_path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.747 DEBUG:boto:Method: GET 2022-09-09T22:59:41.748 DEBUG:boto:Path: /jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.749 DEBUG:boto:Data: 2022-09-09T22:59:41.750 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.750 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:41.751 DEBUG:boto:Port: 8000 2022-09-09T22:59:41.751 DEBUG:boto:Params: {} 2022-09-09T22:59:41.752 DEBUG:boto:Token: None 2022-09-09T22:59:41.753 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/?versions 2022-09-09T22:59:41.754 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8= 2022-09-09T22:59:41.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8=', 'Content-Length': '0'} 2022-09-09T22:59:41.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000629e879c2139b47e-00631bc55d-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.762 DEBUG:boto:b'jkcgbj-40999false' 2022-09-09T22:59:41.763 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.764 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.765 DEBUG:boto:path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.765 DEBUG:boto:auth_path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.770 DEBUG:boto:Method: GET 2022-09-09T22:59:41.771 DEBUG:boto:Path: /jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.771 DEBUG:boto:Data: 2022-09-09T22:59:41.772 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.772 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:41.773 DEBUG:boto:Port: 8001 2022-09-09T22:59:41.773 DEBUG:boto:Params: {} 2022-09-09T22:59:41.774 DEBUG:boto:Token: None 2022-09-09T22:59:41.775 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/?versions 2022-09-09T22:59:41.775 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8= 2022-09-09T22:59:41.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8=', 'Content-Length': '0'} 2022-09-09T22:59:41.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f30191b08c0539ea-00631bc55d-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.780 DEBUG:boto:b'jkcgbj-40999false' 2022-09-09T22:59:41.781 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-40 zones={test-zone1, test-zone3} 2022-09-09T22:59:41.781 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-40 zones={test-zone2, test-zone3} 2022-09-09T22:59:41.782 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.782 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.783 DEBUG:boto:Method: HEAD 2022-09-09T22:59:41.783 DEBUG:boto:Path: /jkcgbj-40/ 2022-09-09T22:59:41.784 DEBUG:boto:Data: 2022-09-09T22:59:41.784 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.785 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:41.785 DEBUG:boto:Port: 8000 2022-09-09T22:59:41.786 DEBUG:boto:Params: {} 2022-09-09T22:59:41.787 DEBUG:boto:Token: None 2022-09-09T22:59:41.787 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/ 2022-09-09T22:59:41.788 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FrX2AJroD3tjf6JQY4qrUOt3ApU= 2022-09-09T22:59:41.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FrX2AJroD3tjf6JQY4qrUOt3ApU=', 'Content-Length': '0'} 2022-09-09T22:59:41.794 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a29d777195be289-00631bc55d-1549-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.795 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.795 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.796 DEBUG:boto:Method: HEAD 2022-09-09T22:59:41.797 DEBUG:boto:Path: /jkcgbj-40/ 2022-09-09T22:59:41.797 DEBUG:boto:Data: 2022-09-09T22:59:41.798 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.798 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:41.799 DEBUG:boto:Port: 8001 2022-09-09T22:59:41.799 DEBUG:boto:Params: {} 2022-09-09T22:59:41.800 DEBUG:boto:Token: None 2022-09-09T22:59:41.800 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/ 2022-09-09T22:59:41.801 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:FrX2AJroD3tjf6JQY4qrUOt3ApU= 2022-09-09T22:59:41.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:FrX2AJroD3tjf6JQY4qrUOt3ApU=', 'Content-Length': '0'} 2022-09-09T22:59:41.807 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000399af798737f59f7-00631bc55d-1403-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.808 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:41.808 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.809 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.809 DEBUG:boto:path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.810 DEBUG:boto:auth_path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.810 DEBUG:boto:Method: GET 2022-09-09T22:59:41.811 DEBUG:boto:Path: /jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.811 DEBUG:boto:Data: 2022-09-09T22:59:41.812 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.812 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:41.813 DEBUG:boto:Port: 8000 2022-09-09T22:59:41.813 DEBUG:boto:Params: {} 2022-09-09T22:59:41.814 DEBUG:boto:Token: None 2022-09-09T22:59:41.815 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/?versions 2022-09-09T22:59:41.815 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8= 2022-09-09T22:59:41.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8=', 'Content-Length': '0'} 2022-09-09T22:59:41.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079e2673216422fb7-00631bc55d-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.824 DEBUG:boto:b'jkcgbj-40999false' 2022-09-09T22:59:41.825 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:41.826 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.827 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.828 DEBUG:boto:path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.829 DEBUG:boto:auth_path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.830 DEBUG:boto:Method: GET 2022-09-09T22:59:41.831 DEBUG:boto:Path: /jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.831 DEBUG:boto:Data: 2022-09-09T22:59:41.832 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.833 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:41.834 DEBUG:boto:Port: 8001 2022-09-09T22:59:41.835 DEBUG:boto:Params: {} 2022-09-09T22:59:41.836 DEBUG:boto:Token: None 2022-09-09T22:59:41.837 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/?versions 2022-09-09T22:59:41.838 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8= 2022-09-09T22:59:41.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8=', 'Content-Length': '0'} 2022-09-09T22:59:41.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045c004a6ebe41b6a-00631bc55d-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.890 DEBUG:boto:b'jkcgbj-40999false' 2022-09-09T22:59:41.891 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.892 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.892 DEBUG:boto:path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.893 DEBUG:boto:auth_path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.894 DEBUG:boto:Method: GET 2022-09-09T22:59:41.895 DEBUG:boto:Path: /jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.896 DEBUG:boto:Data: 2022-09-09T22:59:41.897 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.898 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:41.899 DEBUG:boto:Port: 8000 2022-09-09T22:59:41.900 DEBUG:boto:Params: {} 2022-09-09T22:59:41.901 DEBUG:boto:Token: None 2022-09-09T22:59:41.901 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/?versions 2022-09-09T22:59:41.902 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8= 2022-09-09T22:59:41.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8=', 'Content-Length': '0'} 2022-09-09T22:59:41.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a96ce975b50a1cf-00631bc55d-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.910 DEBUG:boto:b'jkcgbj-40999false' 2022-09-09T22:59:41.911 DEBUG:boto:path=/jkcgbj-40/ 2022-09-09T22:59:41.911 DEBUG:boto:auth_path=/jkcgbj-40/ 2022-09-09T22:59:41.912 DEBUG:boto:path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.912 DEBUG:boto:auth_path=/jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.913 DEBUG:boto:Method: GET 2022-09-09T22:59:41.913 DEBUG:boto:Path: /jkcgbj-40/?versions&max-keys=999 2022-09-09T22:59:41.914 DEBUG:boto:Data: 2022-09-09T22:59:41.915 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.916 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:41.917 DEBUG:boto:Port: 8001 2022-09-09T22:59:41.918 DEBUG:boto:Params: {} 2022-09-09T22:59:41.919 DEBUG:boto:Token: None 2022-09-09T22:59:41.920 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-40/?versions 2022-09-09T22:59:41.920 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8= 2022-09-09T22:59:41.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:YYV3BbDmJoPq6U9Eki5+/6kl4X8=', 'Content-Length': '0'} 2022-09-09T22:59:41.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007eb5b7ff1db0b892-00631bc55d-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.931 DEBUG:boto:b'jkcgbj-40999false' 2022-09-09T22:59:41.933 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-40 zones={test-zone2, test-zone3} 2022-09-09T22:59:41.934 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-41 zones={test-zone2, test-zone1} 2022-09-09T22:59:41.935 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:41.935 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:41.936 DEBUG:boto:Method: HEAD 2022-09-09T22:59:41.937 DEBUG:boto:Path: /jkcgbj-41/ 2022-09-09T22:59:41.938 DEBUG:boto:Data: 2022-09-09T22:59:41.939 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.941 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:41.942 DEBUG:boto:Port: 8000 2022-09-09T22:59:41.943 DEBUG:boto:Params: {} 2022-09-09T22:59:41.944 DEBUG:boto:Token: None 2022-09-09T22:59:41.945 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-41/ 2022-09-09T22:59:41.945 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:NAam+wdlYb1u62TSqcHgmgnQUMo= 2022-09-09T22:59:41.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:NAam+wdlYb1u62TSqcHgmgnQUMo=', 'Content-Length': '0'} 2022-09-09T22:59:41.953 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005fec5cce2dd1e82e-00631bc55d-1549-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.954 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:41.954 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:41.955 DEBUG:boto:Method: HEAD 2022-09-09T22:59:41.956 DEBUG:boto:Path: /jkcgbj-41/ 2022-09-09T22:59:41.957 DEBUG:boto:Data: 2022-09-09T22:59:41.958 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.959 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:41.960 DEBUG:boto:Port: 8000 2022-09-09T22:59:41.961 DEBUG:boto:Params: {} 2022-09-09T22:59:41.962 DEBUG:boto:Token: None 2022-09-09T22:59:41.962 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-41/ 2022-09-09T22:59:41.963 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:NAam+wdlYb1u62TSqcHgmgnQUMo= 2022-09-09T22:59:41.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:NAam+wdlYb1u62TSqcHgmgnQUMo=', 'Content-Length': '0'} 2022-09-09T22:59:41.970 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000678037fe87b91d79-00631bc55d-13fd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:41.971 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:41.972 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:41.973 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:41.974 DEBUG:boto:path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:41.975 DEBUG:boto:auth_path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:41.976 DEBUG:boto:Method: GET 2022-09-09T22:59:41.977 DEBUG:boto:Path: /jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:41.978 DEBUG:boto:Data: 2022-09-09T22:59:41.979 DEBUG:boto:Headers: {} 2022-09-09T22:59:41.980 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:41.981 DEBUG:boto:Port: 8000 2022-09-09T22:59:41.982 DEBUG:boto:Params: {} 2022-09-09T22:59:41.983 DEBUG:boto:Token: None 2022-09-09T22:59:41.984 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:41 GMT /jkcgbj-41/?versions 2022-09-09T22:59:41.985 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:QoyN8duSgMqyg8GZh9zOU4aM0/8= 2022-09-09T22:59:41.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:QoyN8duSgMqyg8GZh9zOU4aM0/8=', 'Content-Length': '0'} 2022-09-09T22:59:41.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9c0545f1cc9146c-00631bc55d-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.034 DEBUG:boto:b'jkcgbj-41999false' 2022-09-09T22:59:42.036 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:42.037 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.037 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.038 DEBUG:boto:path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.039 DEBUG:boto:auth_path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.039 DEBUG:boto:Method: GET 2022-09-09T22:59:42.040 DEBUG:boto:Path: /jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.040 DEBUG:boto:Data: 2022-09-09T22:59:42.041 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.041 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:42.042 DEBUG:boto:Port: 8000 2022-09-09T22:59:42.042 DEBUG:boto:Params: {} 2022-09-09T22:59:42.043 DEBUG:boto:Token: None 2022-09-09T22:59:42.043 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/?versions 2022-09-09T22:59:42.044 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM= 2022-09-09T22:59:42.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM=', 'Content-Length': '0'} 2022-09-09T22:59:42.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd49d345fe77ba2d-00631bc55e-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.051 DEBUG:boto:b'jkcgbj-41999false' 2022-09-09T22:59:42.052 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.052 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.053 DEBUG:boto:path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.054 DEBUG:boto:auth_path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.054 DEBUG:boto:Method: GET 2022-09-09T22:59:42.055 DEBUG:boto:Path: /jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.055 DEBUG:boto:Data: 2022-09-09T22:59:42.056 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.056 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:42.057 DEBUG:boto:Port: 8000 2022-09-09T22:59:42.057 DEBUG:boto:Params: {} 2022-09-09T22:59:42.058 DEBUG:boto:Token: None 2022-09-09T22:59:42.059 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/?versions 2022-09-09T22:59:42.059 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM= 2022-09-09T22:59:42.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM=', 'Content-Length': '0'} 2022-09-09T22:59:42.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f7912250d4ade76-00631bc55e-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.066 DEBUG:boto:b'jkcgbj-41999false' 2022-09-09T22:59:42.066 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.067 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.068 DEBUG:boto:path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.068 DEBUG:boto:auth_path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.069 DEBUG:boto:Method: GET 2022-09-09T22:59:42.069 DEBUG:boto:Path: /jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.070 DEBUG:boto:Data: 2022-09-09T22:59:42.070 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.071 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:42.071 DEBUG:boto:Port: 8000 2022-09-09T22:59:42.072 DEBUG:boto:Params: {} 2022-09-09T22:59:42.072 DEBUG:boto:Token: None 2022-09-09T22:59:42.073 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/?versions 2022-09-09T22:59:42.074 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM= 2022-09-09T22:59:42.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM=', 'Content-Length': '0'} 2022-09-09T22:59:42.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008351a5dbbd7b9630-00631bc55e-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.123 DEBUG:boto:b'jkcgbj-41999false' 2022-09-09T22:59:42.124 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-41 zones={test-zone2, test-zone1} 2022-09-09T22:59:42.124 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-41 zones={test-zone3, test-zone1} 2022-09-09T22:59:42.125 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.125 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.126 DEBUG:boto:Method: HEAD 2022-09-09T22:59:42.126 DEBUG:boto:Path: /jkcgbj-41/ 2022-09-09T22:59:42.127 DEBUG:boto:Data: 2022-09-09T22:59:42.127 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.128 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:42.128 DEBUG:boto:Port: 8001 2022-09-09T22:59:42.129 DEBUG:boto:Params: {} 2022-09-09T22:59:42.130 DEBUG:boto:Token: None 2022-09-09T22:59:42.130 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/ 2022-09-09T22:59:42.131 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+TXqnEP5Q36tEMwSXjgZ1kZOuIg= 2022-09-09T22:59:42.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+TXqnEP5Q36tEMwSXjgZ1kZOuIg=', 'Content-Length': '0'} 2022-09-09T22:59:42.138 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007aba0c3b2e0bc1d4-00631bc55e-1403-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.138 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.140 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.140 DEBUG:boto:Method: HEAD 2022-09-09T22:59:42.141 DEBUG:boto:Path: /jkcgbj-41/ 2022-09-09T22:59:42.142 DEBUG:boto:Data: 2022-09-09T22:59:42.142 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.143 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:42.143 DEBUG:boto:Port: 8000 2022-09-09T22:59:42.144 DEBUG:boto:Params: {} 2022-09-09T22:59:42.145 DEBUG:boto:Token: None 2022-09-09T22:59:42.145 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/ 2022-09-09T22:59:42.146 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+TXqnEP5Q36tEMwSXjgZ1kZOuIg= 2022-09-09T22:59:42.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+TXqnEP5Q36tEMwSXjgZ1kZOuIg=', 'Content-Length': '0'} 2022-09-09T22:59:42.151 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000199b1090404d172f-00631bc55e-13fd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.152 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:42.153 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.154 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.155 DEBUG:boto:path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.156 DEBUG:boto:auth_path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.157 DEBUG:boto:Method: GET 2022-09-09T22:59:42.158 DEBUG:boto:Path: /jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.159 DEBUG:boto:Data: 2022-09-09T22:59:42.159 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.160 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:42.160 DEBUG:boto:Port: 8001 2022-09-09T22:59:42.161 DEBUG:boto:Params: {} 2022-09-09T22:59:42.161 DEBUG:boto:Token: None 2022-09-09T22:59:42.162 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/?versions 2022-09-09T22:59:42.163 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM= 2022-09-09T22:59:42.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM=', 'Content-Length': '0'} 2022-09-09T22:59:42.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000185f123931cf0bc5-00631bc55e-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.170 DEBUG:boto:b'jkcgbj-41999false' 2022-09-09T22:59:42.171 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:42.171 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.172 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.172 DEBUG:boto:path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.173 DEBUG:boto:auth_path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.173 DEBUG:boto:Method: GET 2022-09-09T22:59:42.174 DEBUG:boto:Path: /jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.175 DEBUG:boto:Data: 2022-09-09T22:59:42.175 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.176 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:42.176 DEBUG:boto:Port: 8000 2022-09-09T22:59:42.177 DEBUG:boto:Params: {} 2022-09-09T22:59:42.177 DEBUG:boto:Token: None 2022-09-09T22:59:42.178 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/?versions 2022-09-09T22:59:42.178 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM= 2022-09-09T22:59:42.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM=', 'Content-Length': '0'} 2022-09-09T22:59:42.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee4a20464ff52e70-00631bc55e-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.185 DEBUG:boto:b'jkcgbj-41999false' 2022-09-09T22:59:42.186 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.186 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.187 DEBUG:boto:path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.188 DEBUG:boto:auth_path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.188 DEBUG:boto:Method: GET 2022-09-09T22:59:42.189 DEBUG:boto:Path: /jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.190 DEBUG:boto:Data: 2022-09-09T22:59:42.190 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.191 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:42.191 DEBUG:boto:Port: 8001 2022-09-09T22:59:42.192 DEBUG:boto:Params: {} 2022-09-09T22:59:42.192 DEBUG:boto:Token: None 2022-09-09T22:59:42.193 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/?versions 2022-09-09T22:59:42.193 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM= 2022-09-09T22:59:42.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM=', 'Content-Length': '0'} 2022-09-09T22:59:42.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e50161f2e78924a1-00631bc55e-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.242 DEBUG:boto:b'jkcgbj-41999false' 2022-09-09T22:59:42.244 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.245 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.245 DEBUG:boto:path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.247 DEBUG:boto:auth_path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.247 DEBUG:boto:Method: GET 2022-09-09T22:59:42.250 DEBUG:boto:Path: /jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.251 DEBUG:boto:Data: 2022-09-09T22:59:42.252 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.252 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:42.253 DEBUG:boto:Port: 8000 2022-09-09T22:59:42.253 DEBUG:boto:Params: {} 2022-09-09T22:59:42.254 DEBUG:boto:Token: None 2022-09-09T22:59:42.255 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/?versions 2022-09-09T22:59:42.255 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM= 2022-09-09T22:59:42.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM=', 'Content-Length': '0'} 2022-09-09T22:59:42.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca5bf87d8ec16cd5-00631bc55e-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.264 DEBUG:boto:b'jkcgbj-41999false' 2022-09-09T22:59:42.264 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-41 zones={test-zone3, test-zone1} 2022-09-09T22:59:42.265 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-41 zones={test-zone1, test-zone2} 2022-09-09T22:59:42.265 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.266 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.267 DEBUG:boto:Method: HEAD 2022-09-09T22:59:42.267 DEBUG:boto:Path: /jkcgbj-41/ 2022-09-09T22:59:42.268 DEBUG:boto:Data: 2022-09-09T22:59:42.268 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.269 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:42.269 DEBUG:boto:Port: 8000 2022-09-09T22:59:42.270 DEBUG:boto:Params: {} 2022-09-09T22:59:42.274 DEBUG:boto:Token: None 2022-09-09T22:59:42.275 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/ 2022-09-09T22:59:42.276 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+TXqnEP5Q36tEMwSXjgZ1kZOuIg= 2022-09-09T22:59:42.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+TXqnEP5Q36tEMwSXjgZ1kZOuIg=', 'Content-Length': '0'} 2022-09-09T22:59:42.290 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b08b2b723f79add-00631bc55e-13fd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.292 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.293 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.294 DEBUG:boto:Method: HEAD 2022-09-09T22:59:42.294 DEBUG:boto:Path: /jkcgbj-41/ 2022-09-09T22:59:42.295 DEBUG:boto:Data: 2022-09-09T22:59:42.295 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.296 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:42.296 DEBUG:boto:Port: 8000 2022-09-09T22:59:42.297 DEBUG:boto:Params: {} 2022-09-09T22:59:42.297 DEBUG:boto:Token: None 2022-09-09T22:59:42.298 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/ 2022-09-09T22:59:42.299 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+TXqnEP5Q36tEMwSXjgZ1kZOuIg= 2022-09-09T22:59:42.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+TXqnEP5Q36tEMwSXjgZ1kZOuIg=', 'Content-Length': '0'} 2022-09-09T22:59:42.305 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1b04dff74e3c257-00631bc55e-1549-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.306 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:42.307 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.308 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.309 DEBUG:boto:path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.310 DEBUG:boto:auth_path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.311 DEBUG:boto:Method: GET 2022-09-09T22:59:42.312 DEBUG:boto:Path: /jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.313 DEBUG:boto:Data: 2022-09-09T22:59:42.314 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.315 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:42.316 DEBUG:boto:Port: 8000 2022-09-09T22:59:42.317 DEBUG:boto:Params: {} 2022-09-09T22:59:42.318 DEBUG:boto:Token: None 2022-09-09T22:59:42.319 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/?versions 2022-09-09T22:59:42.320 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM= 2022-09-09T22:59:42.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM=', 'Content-Length': '0'} 2022-09-09T22:59:42.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003aa08d891ad705ae-00631bc55e-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.370 DEBUG:boto:b'jkcgbj-41999false' 2022-09-09T22:59:42.372 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:42.373 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.373 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.374 DEBUG:boto:path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.375 DEBUG:boto:auth_path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.376 DEBUG:boto:Method: GET 2022-09-09T22:59:42.377 DEBUG:boto:Path: /jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.377 DEBUG:boto:Data: 2022-09-09T22:59:42.379 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.379 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:42.380 DEBUG:boto:Port: 8000 2022-09-09T22:59:42.381 DEBUG:boto:Params: {} 2022-09-09T22:59:42.381 DEBUG:boto:Token: None 2022-09-09T22:59:42.382 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/?versions 2022-09-09T22:59:42.382 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM= 2022-09-09T22:59:42.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM=', 'Content-Length': '0'} 2022-09-09T22:59:42.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000266781881134b400-00631bc55e-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.390 DEBUG:boto:b'jkcgbj-41999false' 2022-09-09T22:59:42.391 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.392 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.393 DEBUG:boto:path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.394 DEBUG:boto:auth_path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.395 DEBUG:boto:Method: GET 2022-09-09T22:59:42.396 DEBUG:boto:Path: /jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.397 DEBUG:boto:Data: 2022-09-09T22:59:42.397 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.398 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:42.399 DEBUG:boto:Port: 8000 2022-09-09T22:59:42.399 DEBUG:boto:Params: {} 2022-09-09T22:59:42.400 DEBUG:boto:Token: None 2022-09-09T22:59:42.400 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/?versions 2022-09-09T22:59:42.401 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM= 2022-09-09T22:59:42.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM=', 'Content-Length': '0'} 2022-09-09T22:59:42.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003e5050051c2b7a4-00631bc55e-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.409 DEBUG:boto:b'jkcgbj-41999false' 2022-09-09T22:59:42.410 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.410 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.411 DEBUG:boto:path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.411 DEBUG:boto:auth_path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.412 DEBUG:boto:Method: GET 2022-09-09T22:59:42.412 DEBUG:boto:Path: /jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.413 DEBUG:boto:Data: 2022-09-09T22:59:42.413 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.414 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:42.414 DEBUG:boto:Port: 8000 2022-09-09T22:59:42.415 DEBUG:boto:Params: {} 2022-09-09T22:59:42.416 DEBUG:boto:Token: None 2022-09-09T22:59:42.416 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/?versions 2022-09-09T22:59:42.417 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM= 2022-09-09T22:59:42.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM=', 'Content-Length': '0'} 2022-09-09T22:59:42.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050bcd3dc77006ca6-00631bc55e-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.424 DEBUG:boto:b'jkcgbj-41999false' 2022-09-09T22:59:42.425 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-41 zones={test-zone1, test-zone2} 2022-09-09T22:59:42.425 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-41 zones={test-zone3, test-zone2} 2022-09-09T22:59:42.426 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.427 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.427 DEBUG:boto:Method: HEAD 2022-09-09T22:59:42.428 DEBUG:boto:Path: /jkcgbj-41/ 2022-09-09T22:59:42.428 DEBUG:boto:Data: 2022-09-09T22:59:42.429 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.430 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:42.431 DEBUG:boto:Port: 8001 2022-09-09T22:59:42.432 DEBUG:boto:Params: {} 2022-09-09T22:59:42.433 DEBUG:boto:Token: None 2022-09-09T22:59:42.434 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/ 2022-09-09T22:59:42.435 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+TXqnEP5Q36tEMwSXjgZ1kZOuIg= 2022-09-09T22:59:42.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+TXqnEP5Q36tEMwSXjgZ1kZOuIg=', 'Content-Length': '0'} 2022-09-09T22:59:42.441 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a83527235266f2d7-00631bc55e-1403-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.442 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.443 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.444 DEBUG:boto:Method: HEAD 2022-09-09T22:59:42.444 DEBUG:boto:Path: /jkcgbj-41/ 2022-09-09T22:59:42.445 DEBUG:boto:Data: 2022-09-09T22:59:42.445 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.446 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:42.446 DEBUG:boto:Port: 8000 2022-09-09T22:59:42.447 DEBUG:boto:Params: {} 2022-09-09T22:59:42.447 DEBUG:boto:Token: None 2022-09-09T22:59:42.448 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/ 2022-09-09T22:59:42.449 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+TXqnEP5Q36tEMwSXjgZ1kZOuIg= 2022-09-09T22:59:42.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+TXqnEP5Q36tEMwSXjgZ1kZOuIg=', 'Content-Length': '0'} 2022-09-09T22:59:42.454 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc3bbb273a0cc24f-00631bc55e-1549-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.455 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:42.455 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.456 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.456 DEBUG:boto:path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.457 DEBUG:boto:auth_path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.457 DEBUG:boto:Method: GET 2022-09-09T22:59:42.458 DEBUG:boto:Path: /jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.459 DEBUG:boto:Data: 2022-09-09T22:59:42.459 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.460 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:42.460 DEBUG:boto:Port: 8001 2022-09-09T22:59:42.461 DEBUG:boto:Params: {} 2022-09-09T22:59:42.461 DEBUG:boto:Token: None 2022-09-09T22:59:42.462 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/?versions 2022-09-09T22:59:42.463 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM= 2022-09-09T22:59:42.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM=', 'Content-Length': '0'} 2022-09-09T22:59:42.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001755e053309dec39-00631bc55e-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.470 DEBUG:boto:b'jkcgbj-41999false' 2022-09-09T22:59:42.471 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:42.472 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.472 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.473 DEBUG:boto:path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.474 DEBUG:boto:auth_path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.474 DEBUG:boto:Method: GET 2022-09-09T22:59:42.475 DEBUG:boto:Path: /jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.476 DEBUG:boto:Data: 2022-09-09T22:59:42.477 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.478 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:42.479 DEBUG:boto:Port: 8000 2022-09-09T22:59:42.480 DEBUG:boto:Params: {} 2022-09-09T22:59:42.481 DEBUG:boto:Token: None 2022-09-09T22:59:42.482 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/?versions 2022-09-09T22:59:42.483 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM= 2022-09-09T22:59:42.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM=', 'Content-Length': '0'} 2022-09-09T22:59:42.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087b2c44bb421b70e-00631bc55e-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.530 DEBUG:boto:b'jkcgbj-41999false' 2022-09-09T22:59:42.532 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.533 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.534 DEBUG:boto:path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.535 DEBUG:boto:auth_path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.535 DEBUG:boto:Method: GET 2022-09-09T22:59:42.536 DEBUG:boto:Path: /jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.537 DEBUG:boto:Data: 2022-09-09T22:59:42.538 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.539 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:42.540 DEBUG:boto:Port: 8001 2022-09-09T22:59:42.541 DEBUG:boto:Params: {} 2022-09-09T22:59:42.542 DEBUG:boto:Token: None 2022-09-09T22:59:42.543 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/?versions 2022-09-09T22:59:42.544 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM= 2022-09-09T22:59:42.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM=', 'Content-Length': '0'} 2022-09-09T22:59:42.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f564fb1e9bb50029-00631bc55e-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.553 DEBUG:boto:b'jkcgbj-41999false' 2022-09-09T22:59:42.554 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.555 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.555 DEBUG:boto:path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.556 DEBUG:boto:auth_path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.556 DEBUG:boto:Method: GET 2022-09-09T22:59:42.557 DEBUG:boto:Path: /jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.557 DEBUG:boto:Data: 2022-09-09T22:59:42.558 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.558 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:42.559 DEBUG:boto:Port: 8000 2022-09-09T22:59:42.559 DEBUG:boto:Params: {} 2022-09-09T22:59:42.560 DEBUG:boto:Token: None 2022-09-09T22:59:42.561 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/?versions 2022-09-09T22:59:42.561 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM= 2022-09-09T22:59:42.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM=', 'Content-Length': '0'} 2022-09-09T22:59:42.567 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007879eeedb8849e9e-00631bc55e-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.568 DEBUG:boto:b'jkcgbj-41999false' 2022-09-09T22:59:42.569 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-41 zones={test-zone3, test-zone2} 2022-09-09T22:59:42.570 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-41 zones={test-zone1, test-zone3} 2022-09-09T22:59:42.570 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.571 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.571 DEBUG:boto:Method: HEAD 2022-09-09T22:59:42.572 DEBUG:boto:Path: /jkcgbj-41/ 2022-09-09T22:59:42.572 DEBUG:boto:Data: 2022-09-09T22:59:42.573 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.573 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:42.574 DEBUG:boto:Port: 8000 2022-09-09T22:59:42.574 DEBUG:boto:Params: {} 2022-09-09T22:59:42.575 DEBUG:boto:Token: None 2022-09-09T22:59:42.576 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/ 2022-09-09T22:59:42.576 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+TXqnEP5Q36tEMwSXjgZ1kZOuIg= 2022-09-09T22:59:42.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+TXqnEP5Q36tEMwSXjgZ1kZOuIg=', 'Content-Length': '0'} 2022-09-09T22:59:42.582 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d10251a09a7420c5-00631bc55e-13fd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.583 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.584 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.584 DEBUG:boto:Method: HEAD 2022-09-09T22:59:42.585 DEBUG:boto:Path: /jkcgbj-41/ 2022-09-09T22:59:42.585 DEBUG:boto:Data: 2022-09-09T22:59:42.586 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.586 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:42.587 DEBUG:boto:Port: 8001 2022-09-09T22:59:42.588 DEBUG:boto:Params: {} 2022-09-09T22:59:42.589 DEBUG:boto:Token: None 2022-09-09T22:59:42.590 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/ 2022-09-09T22:59:42.590 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+TXqnEP5Q36tEMwSXjgZ1kZOuIg= 2022-09-09T22:59:42.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+TXqnEP5Q36tEMwSXjgZ1kZOuIg=', 'Content-Length': '0'} 2022-09-09T22:59:42.596 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000583e2dfbd3829ff4-00631bc55e-1403-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.597 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:42.597 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.598 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.598 DEBUG:boto:path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.599 DEBUG:boto:auth_path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.599 DEBUG:boto:Method: GET 2022-09-09T22:59:42.600 DEBUG:boto:Path: /jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.600 DEBUG:boto:Data: 2022-09-09T22:59:42.601 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.602 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:42.602 DEBUG:boto:Port: 8000 2022-09-09T22:59:42.603 DEBUG:boto:Params: {} 2022-09-09T22:59:42.604 DEBUG:boto:Token: None 2022-09-09T22:59:42.605 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/?versions 2022-09-09T22:59:42.606 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM= 2022-09-09T22:59:42.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM=', 'Content-Length': '0'} 2022-09-09T22:59:42.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096ad79ead38ae507-00631bc55e-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.615 DEBUG:boto:b'jkcgbj-41999false' 2022-09-09T22:59:42.617 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:42.618 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.619 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.619 DEBUG:boto:path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.620 DEBUG:boto:auth_path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.621 DEBUG:boto:Method: GET 2022-09-09T22:59:42.622 DEBUG:boto:Path: /jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.622 DEBUG:boto:Data: 2022-09-09T22:59:42.623 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.624 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:42.625 DEBUG:boto:Port: 8001 2022-09-09T22:59:42.625 DEBUG:boto:Params: {} 2022-09-09T22:59:42.626 DEBUG:boto:Token: None 2022-09-09T22:59:42.627 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/?versions 2022-09-09T22:59:42.628 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM= 2022-09-09T22:59:42.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM=', 'Content-Length': '0'} 2022-09-09T22:59:42.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000866734fa29f393ab-00631bc55e-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.635 DEBUG:boto:b'jkcgbj-41999false' 2022-09-09T22:59:42.637 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.638 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.639 DEBUG:boto:path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.640 DEBUG:boto:auth_path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.641 DEBUG:boto:Method: GET 2022-09-09T22:59:42.642 DEBUG:boto:Path: /jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.643 DEBUG:boto:Data: 2022-09-09T22:59:42.644 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.645 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:42.646 DEBUG:boto:Port: 8000 2022-09-09T22:59:42.647 DEBUG:boto:Params: {} 2022-09-09T22:59:42.648 DEBUG:boto:Token: None 2022-09-09T22:59:42.650 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/?versions 2022-09-09T22:59:42.651 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM= 2022-09-09T22:59:42.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM=', 'Content-Length': '0'} 2022-09-09T22:59:42.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a223cc45afe35017-00631bc55e-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.706 DEBUG:boto:b'jkcgbj-41999false' 2022-09-09T22:59:42.707 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.708 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.708 DEBUG:boto:path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.709 DEBUG:boto:auth_path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.709 DEBUG:boto:Method: GET 2022-09-09T22:59:42.710 DEBUG:boto:Path: /jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.710 DEBUG:boto:Data: 2022-09-09T22:59:42.711 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.711 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:42.712 DEBUG:boto:Port: 8001 2022-09-09T22:59:42.712 DEBUG:boto:Params: {} 2022-09-09T22:59:42.713 DEBUG:boto:Token: None 2022-09-09T22:59:42.713 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/?versions 2022-09-09T22:59:42.715 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM= 2022-09-09T22:59:42.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM=', 'Content-Length': '0'} 2022-09-09T22:59:42.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006adaef0e6b9f3b8a-00631bc55e-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.723 DEBUG:boto:b'jkcgbj-41999false' 2022-09-09T22:59:42.724 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-41 zones={test-zone1, test-zone3} 2022-09-09T22:59:42.725 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-41 zones={test-zone2, test-zone3} 2022-09-09T22:59:42.726 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.727 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.727 DEBUG:boto:Method: HEAD 2022-09-09T22:59:42.728 DEBUG:boto:Path: /jkcgbj-41/ 2022-09-09T22:59:42.729 DEBUG:boto:Data: 2022-09-09T22:59:42.731 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.732 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:42.733 DEBUG:boto:Port: 8000 2022-09-09T22:59:42.734 DEBUG:boto:Params: {} 2022-09-09T22:59:42.735 DEBUG:boto:Token: None 2022-09-09T22:59:42.736 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/ 2022-09-09T22:59:42.737 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+TXqnEP5Q36tEMwSXjgZ1kZOuIg= 2022-09-09T22:59:42.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+TXqnEP5Q36tEMwSXjgZ1kZOuIg=', 'Content-Length': '0'} 2022-09-09T22:59:42.744 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f55e658abb434900-00631bc55e-1549-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.745 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.745 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.746 DEBUG:boto:Method: HEAD 2022-09-09T22:59:42.747 DEBUG:boto:Path: /jkcgbj-41/ 2022-09-09T22:59:42.748 DEBUG:boto:Data: 2022-09-09T22:59:42.749 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.750 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:42.751 DEBUG:boto:Port: 8001 2022-09-09T22:59:42.752 DEBUG:boto:Params: {} 2022-09-09T22:59:42.753 DEBUG:boto:Token: None 2022-09-09T22:59:42.753 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/ 2022-09-09T22:59:42.754 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+TXqnEP5Q36tEMwSXjgZ1kZOuIg= 2022-09-09T22:59:42.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+TXqnEP5Q36tEMwSXjgZ1kZOuIg=', 'Content-Length': '0'} 2022-09-09T22:59:42.760 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b5c5879e9f9c80b-00631bc55e-1403-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.762 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:42.763 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.763 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.764 DEBUG:boto:path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.765 DEBUG:boto:auth_path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.766 DEBUG:boto:Method: GET 2022-09-09T22:59:42.766 DEBUG:boto:Path: /jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.767 DEBUG:boto:Data: 2022-09-09T22:59:42.767 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.768 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:42.768 DEBUG:boto:Port: 8000 2022-09-09T22:59:42.769 DEBUG:boto:Params: {} 2022-09-09T22:59:42.769 DEBUG:boto:Token: None 2022-09-09T22:59:42.770 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/?versions 2022-09-09T22:59:42.771 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM= 2022-09-09T22:59:42.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM=', 'Content-Length': '0'} 2022-09-09T22:59:42.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fe0807cd8a6e68b-00631bc55e-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.779 DEBUG:boto:b'jkcgbj-41999false' 2022-09-09T22:59:42.780 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:42.781 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.782 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.782 DEBUG:boto:path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.783 DEBUG:boto:auth_path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.783 DEBUG:boto:Method: GET 2022-09-09T22:59:42.784 DEBUG:boto:Path: /jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.784 DEBUG:boto:Data: 2022-09-09T22:59:42.785 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.785 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:42.786 DEBUG:boto:Port: 8001 2022-09-09T22:59:42.786 DEBUG:boto:Params: {} 2022-09-09T22:59:42.787 DEBUG:boto:Token: None 2022-09-09T22:59:42.788 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/?versions 2022-09-09T22:59:42.788 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM= 2022-09-09T22:59:42.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM=', 'Content-Length': '0'} 2022-09-09T22:59:42.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008598805927c4f94d-00631bc55e-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.835 DEBUG:boto:b'jkcgbj-41999false' 2022-09-09T22:59:42.837 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.838 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.839 DEBUG:boto:path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.839 DEBUG:boto:auth_path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.840 DEBUG:boto:Method: GET 2022-09-09T22:59:42.841 DEBUG:boto:Path: /jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.842 DEBUG:boto:Data: 2022-09-09T22:59:42.843 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.844 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:42.845 DEBUG:boto:Port: 8000 2022-09-09T22:59:42.846 DEBUG:boto:Params: {} 2022-09-09T22:59:42.847 DEBUG:boto:Token: None 2022-09-09T22:59:42.848 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/?versions 2022-09-09T22:59:42.849 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM= 2022-09-09T22:59:42.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM=', 'Content-Length': '0'} 2022-09-09T22:59:42.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045c1b0d8477cf495-00631bc55e-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.856 DEBUG:boto:b'jkcgbj-41999false' 2022-09-09T22:59:42.857 DEBUG:boto:path=/jkcgbj-41/ 2022-09-09T22:59:42.858 DEBUG:boto:auth_path=/jkcgbj-41/ 2022-09-09T22:59:42.858 DEBUG:boto:path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.859 DEBUG:boto:auth_path=/jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.860 DEBUG:boto:Method: GET 2022-09-09T22:59:42.860 DEBUG:boto:Path: /jkcgbj-41/?versions&max-keys=999 2022-09-09T22:59:42.861 DEBUG:boto:Data: 2022-09-09T22:59:42.862 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.862 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:42.863 DEBUG:boto:Port: 8001 2022-09-09T22:59:42.864 DEBUG:boto:Params: {} 2022-09-09T22:59:42.864 DEBUG:boto:Token: None 2022-09-09T22:59:42.865 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-41/?versions 2022-09-09T22:59:42.866 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM= 2022-09-09T22:59:42.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KBelKe8loW79GEMY71Tajru5YyM=', 'Content-Length': '0'} 2022-09-09T22:59:42.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000444f8f9af57d2078-00631bc55e-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.873 DEBUG:boto:b'jkcgbj-41999false' 2022-09-09T22:59:42.874 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-41 zones={test-zone2, test-zone3} 2022-09-09T22:59:42.875 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-42 zones={test-zone2, test-zone1} 2022-09-09T22:59:42.875 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:42.876 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:42.876 DEBUG:boto:Method: HEAD 2022-09-09T22:59:42.877 DEBUG:boto:Path: /jkcgbj-42/ 2022-09-09T22:59:42.878 DEBUG:boto:Data: 2022-09-09T22:59:42.879 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.879 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:42.880 DEBUG:boto:Port: 8000 2022-09-09T22:59:42.880 DEBUG:boto:Params: {} 2022-09-09T22:59:42.881 DEBUG:boto:Token: None 2022-09-09T22:59:42.882 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-42/ 2022-09-09T22:59:42.882 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Tyno8nIqSKEbECy2fIOETVrBc2I= 2022-09-09T22:59:42.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Tyno8nIqSKEbECy2fIOETVrBc2I=', 'Content-Length': '0'} 2022-09-09T22:59:42.888 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000099ac3ac50dde351d-00631bc55e-1549-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.889 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:42.890 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:42.890 DEBUG:boto:Method: HEAD 2022-09-09T22:59:42.891 DEBUG:boto:Path: /jkcgbj-42/ 2022-09-09T22:59:42.891 DEBUG:boto:Data: 2022-09-09T22:59:42.892 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.892 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:42.893 DEBUG:boto:Port: 8000 2022-09-09T22:59:42.893 DEBUG:boto:Params: {} 2022-09-09T22:59:42.894 DEBUG:boto:Token: None 2022-09-09T22:59:42.894 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-42/ 2022-09-09T22:59:42.895 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Tyno8nIqSKEbECy2fIOETVrBc2I= 2022-09-09T22:59:42.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Tyno8nIqSKEbECy2fIOETVrBc2I=', 'Content-Length': '0'} 2022-09-09T22:59:42.901 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000149d1bb15a962c9a-00631bc55e-13fd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.903 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:42.904 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:42.905 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:42.906 DEBUG:boto:path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:42.907 DEBUG:boto:auth_path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:42.909 DEBUG:boto:Method: GET 2022-09-09T22:59:42.910 DEBUG:boto:Path: /jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:42.911 DEBUG:boto:Data: 2022-09-09T22:59:42.912 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.912 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:42.913 DEBUG:boto:Port: 8000 2022-09-09T22:59:42.913 DEBUG:boto:Params: {} 2022-09-09T22:59:42.914 DEBUG:boto:Token: None 2022-09-09T22:59:42.915 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-42/?versions 2022-09-09T22:59:42.916 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:O3fpPwWBCIh5wAzvs+jF2PCHe9M= 2022-09-09T22:59:42.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:O3fpPwWBCIh5wAzvs+jF2PCHe9M=', 'Content-Length': '0'} 2022-09-09T22:59:42.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045230021e5307bb1-00631bc55e-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.966 DEBUG:boto:b'jkcgbj-42999false' 2022-09-09T22:59:42.968 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:42.969 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:42.970 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:42.971 DEBUG:boto:path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:42.972 DEBUG:boto:auth_path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:42.973 DEBUG:boto:Method: GET 2022-09-09T22:59:42.974 DEBUG:boto:Path: /jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:42.975 DEBUG:boto:Data: 2022-09-09T22:59:42.976 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.977 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:42.978 DEBUG:boto:Port: 8000 2022-09-09T22:59:42.979 DEBUG:boto:Params: {} 2022-09-09T22:59:42.979 DEBUG:boto:Token: None 2022-09-09T22:59:42.980 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-42/?versions 2022-09-09T22:59:42.981 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:O3fpPwWBCIh5wAzvs+jF2PCHe9M= 2022-09-09T22:59:42.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:O3fpPwWBCIh5wAzvs+jF2PCHe9M=', 'Content-Length': '0'} 2022-09-09T22:59:42.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099fbb294cf26478f-00631bc55e-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:42.988 DEBUG:boto:b'jkcgbj-42999false' 2022-09-09T22:59:42.988 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:42.989 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:42.989 DEBUG:boto:path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:42.990 DEBUG:boto:auth_path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:42.991 DEBUG:boto:Method: GET 2022-09-09T22:59:42.991 DEBUG:boto:Path: /jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:42.992 DEBUG:boto:Data: 2022-09-09T22:59:42.992 DEBUG:boto:Headers: {} 2022-09-09T22:59:42.993 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:42.993 DEBUG:boto:Port: 8000 2022-09-09T22:59:42.994 DEBUG:boto:Params: {} 2022-09-09T22:59:42.994 DEBUG:boto:Token: None 2022-09-09T22:59:42.995 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:42 GMT /jkcgbj-42/?versions 2022-09-09T22:59:42.996 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:O3fpPwWBCIh5wAzvs+jF2PCHe9M= 2022-09-09T22:59:42.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:42 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:O3fpPwWBCIh5wAzvs+jF2PCHe9M=', 'Content-Length': '0'} 2022-09-09T22:59:43.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e8f77913fa8658e-00631bc55e-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.003 DEBUG:boto:b'jkcgbj-42999false' 2022-09-09T22:59:43.004 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:43.004 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:43.005 DEBUG:boto:path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.006 DEBUG:boto:auth_path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.006 DEBUG:boto:Method: GET 2022-09-09T22:59:43.007 DEBUG:boto:Path: /jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.008 DEBUG:boto:Data: 2022-09-09T22:59:43.008 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.009 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:43.010 DEBUG:boto:Port: 8000 2022-09-09T22:59:43.010 DEBUG:boto:Params: {} 2022-09-09T22:59:43.011 DEBUG:boto:Token: None 2022-09-09T22:59:43.011 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-42/?versions 2022-09-09T22:59:43.017 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo= 2022-09-09T22:59:43.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo=', 'Content-Length': '0'} 2022-09-09T22:59:43.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca6e9e6b9f0e2db4-00631bc55f-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.025 DEBUG:boto:b'jkcgbj-42999false' 2022-09-09T22:59:43.026 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-42 zones={test-zone2, test-zone1} 2022-09-09T22:59:43.027 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-42 zones={test-zone3, test-zone1} 2022-09-09T22:59:43.028 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:43.028 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:43.029 DEBUG:boto:Method: HEAD 2022-09-09T22:59:43.030 DEBUG:boto:Path: /jkcgbj-42/ 2022-09-09T22:59:43.031 DEBUG:boto:Data: 2022-09-09T22:59:43.032 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.033 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:43.034 DEBUG:boto:Port: 8001 2022-09-09T22:59:43.035 DEBUG:boto:Params: {} 2022-09-09T22:59:43.036 DEBUG:boto:Token: None 2022-09-09T22:59:43.037 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-42/ 2022-09-09T22:59:43.037 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LH7B7/QDu5ZlrZ8Eg1dQ9icD9XY= 2022-09-09T22:59:43.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LH7B7/QDu5ZlrZ8Eg1dQ9icD9XY=', 'Content-Length': '0'} 2022-09-09T22:59:43.043 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088fd15479b0e1539-00631bc55f-1403-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.044 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:43.045 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:43.046 DEBUG:boto:Method: HEAD 2022-09-09T22:59:43.047 DEBUG:boto:Path: /jkcgbj-42/ 2022-09-09T22:59:43.048 DEBUG:boto:Data: 2022-09-09T22:59:43.048 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.049 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:43.050 DEBUG:boto:Port: 8000 2022-09-09T22:59:43.050 DEBUG:boto:Params: {} 2022-09-09T22:59:43.051 DEBUG:boto:Token: None 2022-09-09T22:59:43.052 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-42/ 2022-09-09T22:59:43.052 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LH7B7/QDu5ZlrZ8Eg1dQ9icD9XY= 2022-09-09T22:59:43.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LH7B7/QDu5ZlrZ8Eg1dQ9icD9XY=', 'Content-Length': '0'} 2022-09-09T22:59:43.060 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000191d94af65a324a4-00631bc55f-13fd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.061 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:43.062 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:43.063 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:43.064 DEBUG:boto:path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.064 DEBUG:boto:auth_path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.065 DEBUG:boto:Method: GET 2022-09-09T22:59:43.066 DEBUG:boto:Path: /jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.067 DEBUG:boto:Data: 2022-09-09T22:59:43.068 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.068 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:43.069 DEBUG:boto:Port: 8001 2022-09-09T22:59:43.069 DEBUG:boto:Params: {} 2022-09-09T22:59:43.070 DEBUG:boto:Token: None 2022-09-09T22:59:43.071 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-42/?versions 2022-09-09T22:59:43.073 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo= 2022-09-09T22:59:43.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo=', 'Content-Length': '0'} 2022-09-09T22:59:43.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014ad6f61ba21af30-00631bc55f-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.084 DEBUG:boto:b'jkcgbj-42999false' 2022-09-09T22:59:43.086 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:43.087 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:43.088 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:43.089 DEBUG:boto:path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.090 DEBUG:boto:auth_path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.091 DEBUG:boto:Method: GET 2022-09-09T22:59:43.092 DEBUG:boto:Path: /jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.093 DEBUG:boto:Data: 2022-09-09T22:59:43.094 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.095 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:43.097 DEBUG:boto:Port: 8000 2022-09-09T22:59:43.098 DEBUG:boto:Params: {} 2022-09-09T22:59:43.099 DEBUG:boto:Token: None 2022-09-09T22:59:43.100 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-42/?versions 2022-09-09T22:59:43.101 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo= 2022-09-09T22:59:43.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo=', 'Content-Length': '0'} 2022-09-09T22:59:43.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb16fed5b910ad7d-00631bc55f-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.154 DEBUG:boto:b'jkcgbj-42999false' 2022-09-09T22:59:43.155 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:43.155 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:43.156 DEBUG:boto:path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.157 DEBUG:boto:auth_path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.157 DEBUG:boto:Method: GET 2022-09-09T22:59:43.158 DEBUG:boto:Path: /jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.159 DEBUG:boto:Data: 2022-09-09T22:59:43.159 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.160 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:43.160 DEBUG:boto:Port: 8001 2022-09-09T22:59:43.161 DEBUG:boto:Params: {} 2022-09-09T22:59:43.161 DEBUG:boto:Token: None 2022-09-09T22:59:43.162 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-42/?versions 2022-09-09T22:59:43.163 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo= 2022-09-09T22:59:43.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo=', 'Content-Length': '0'} 2022-09-09T22:59:43.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089f64b08e999a4a8-00631bc55f-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.169 DEBUG:boto:b'jkcgbj-42999false' 2022-09-09T22:59:43.170 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:43.170 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:43.171 DEBUG:boto:path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.172 DEBUG:boto:auth_path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.172 DEBUG:boto:Method: GET 2022-09-09T22:59:43.173 DEBUG:boto:Path: /jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.173 DEBUG:boto:Data: 2022-09-09T22:59:43.174 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.174 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:43.175 DEBUG:boto:Port: 8000 2022-09-09T22:59:43.175 DEBUG:boto:Params: {} 2022-09-09T22:59:43.176 DEBUG:boto:Token: None 2022-09-09T22:59:43.177 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-42/?versions 2022-09-09T22:59:43.177 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo= 2022-09-09T22:59:43.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo=', 'Content-Length': '0'} 2022-09-09T22:59:43.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c1b456694683071-00631bc55f-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.184 DEBUG:boto:b'jkcgbj-42999false' 2022-09-09T22:59:43.185 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-42 zones={test-zone3, test-zone1} 2022-09-09T22:59:43.186 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-42 zones={test-zone1, test-zone2} 2022-09-09T22:59:43.186 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:43.187 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:43.187 DEBUG:boto:Method: HEAD 2022-09-09T22:59:43.188 DEBUG:boto:Path: /jkcgbj-42/ 2022-09-09T22:59:43.189 DEBUG:boto:Data: 2022-09-09T22:59:43.189 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.190 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:43.190 DEBUG:boto:Port: 8000 2022-09-09T22:59:43.191 DEBUG:boto:Params: {} 2022-09-09T22:59:43.191 DEBUG:boto:Token: None 2022-09-09T22:59:43.192 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-42/ 2022-09-09T22:59:43.193 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LH7B7/QDu5ZlrZ8Eg1dQ9icD9XY= 2022-09-09T22:59:43.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LH7B7/QDu5ZlrZ8Eg1dQ9icD9XY=', 'Content-Length': '0'} 2022-09-09T22:59:43.198 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc1ec76ba0f4b75f-00631bc55f-13fd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.199 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:43.199 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:43.200 DEBUG:boto:Method: HEAD 2022-09-09T22:59:43.200 DEBUG:boto:Path: /jkcgbj-42/ 2022-09-09T22:59:43.201 DEBUG:boto:Data: 2022-09-09T22:59:43.201 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.202 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:43.202 DEBUG:boto:Port: 8000 2022-09-09T22:59:43.203 DEBUG:boto:Params: {} 2022-09-09T22:59:43.204 DEBUG:boto:Token: None 2022-09-09T22:59:43.204 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-42/ 2022-09-09T22:59:43.205 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LH7B7/QDu5ZlrZ8Eg1dQ9icD9XY= 2022-09-09T22:59:43.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LH7B7/QDu5ZlrZ8Eg1dQ9icD9XY=', 'Content-Length': '0'} 2022-09-09T22:59:43.210 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a1856577f37e2de8-00631bc55f-1549-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.211 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:43.212 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:43.213 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:43.214 DEBUG:boto:path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.215 DEBUG:boto:auth_path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.215 DEBUG:boto:Method: GET 2022-09-09T22:59:43.216 DEBUG:boto:Path: /jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.217 DEBUG:boto:Data: 2022-09-09T22:59:43.218 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.218 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:43.219 DEBUG:boto:Port: 8000 2022-09-09T22:59:43.219 DEBUG:boto:Params: {} 2022-09-09T22:59:43.220 DEBUG:boto:Token: None 2022-09-09T22:59:43.221 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-42/?versions 2022-09-09T22:59:43.221 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo= 2022-09-09T22:59:43.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo=', 'Content-Length': '0'} 2022-09-09T22:59:43.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf4fc9dba15ebf05-00631bc55f-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.270 DEBUG:boto:b'jkcgbj-42999false' 2022-09-09T22:59:43.271 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:43.271 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:43.272 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:43.273 DEBUG:boto:path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.273 DEBUG:boto:auth_path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.274 DEBUG:boto:Method: GET 2022-09-09T22:59:43.275 DEBUG:boto:Path: /jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.275 DEBUG:boto:Data: 2022-09-09T22:59:43.276 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.276 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:43.277 DEBUG:boto:Port: 8000 2022-09-09T22:59:43.277 DEBUG:boto:Params: {} 2022-09-09T22:59:43.278 DEBUG:boto:Token: None 2022-09-09T22:59:43.279 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-42/?versions 2022-09-09T22:59:43.280 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo= 2022-09-09T22:59:43.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo=', 'Content-Length': '0'} 2022-09-09T22:59:43.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e58359390ea3b308-00631bc55f-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.287 DEBUG:boto:b'jkcgbj-42999false' 2022-09-09T22:59:43.287 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:43.288 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:43.288 DEBUG:boto:path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.289 DEBUG:boto:auth_path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.289 DEBUG:boto:Method: GET 2022-09-09T22:59:43.290 DEBUG:boto:Path: /jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.290 DEBUG:boto:Data: 2022-09-09T22:59:43.291 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.291 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:43.292 DEBUG:boto:Port: 8000 2022-09-09T22:59:43.293 DEBUG:boto:Params: {} 2022-09-09T22:59:43.293 DEBUG:boto:Token: None 2022-09-09T22:59:43.294 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-42/?versions 2022-09-09T22:59:43.295 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo= 2022-09-09T22:59:43.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo=', 'Content-Length': '0'} 2022-09-09T22:59:43.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035117e5a4ab78f64-00631bc55f-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.302 DEBUG:boto:b'jkcgbj-42999false' 2022-09-09T22:59:43.302 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:43.303 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:43.304 DEBUG:boto:path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.305 DEBUG:boto:auth_path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.306 DEBUG:boto:Method: GET 2022-09-09T22:59:43.307 DEBUG:boto:Path: /jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.307 DEBUG:boto:Data: 2022-09-09T22:59:43.308 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.309 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:43.310 DEBUG:boto:Port: 8000 2022-09-09T22:59:43.310 DEBUG:boto:Params: {} 2022-09-09T22:59:43.311 DEBUG:boto:Token: None 2022-09-09T22:59:43.312 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-42/?versions 2022-09-09T22:59:43.312 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo= 2022-09-09T22:59:43.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo=', 'Content-Length': '0'} 2022-09-09T22:59:43.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e43aee5521021fcf-00631bc55f-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.320 DEBUG:boto:b'jkcgbj-42999false' 2022-09-09T22:59:43.321 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-42 zones={test-zone1, test-zone2} 2022-09-09T22:59:43.322 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-42 zones={test-zone3, test-zone2} 2022-09-09T22:59:43.322 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:43.323 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:43.323 DEBUG:boto:Method: HEAD 2022-09-09T22:59:43.324 DEBUG:boto:Path: /jkcgbj-42/ 2022-09-09T22:59:43.324 DEBUG:boto:Data: 2022-09-09T22:59:43.325 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.325 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:43.326 DEBUG:boto:Port: 8001 2022-09-09T22:59:43.326 DEBUG:boto:Params: {} 2022-09-09T22:59:43.327 DEBUG:boto:Token: None 2022-09-09T22:59:43.327 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-42/ 2022-09-09T22:59:43.328 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LH7B7/QDu5ZlrZ8Eg1dQ9icD9XY= 2022-09-09T22:59:43.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LH7B7/QDu5ZlrZ8Eg1dQ9icD9XY=', 'Content-Length': '0'} 2022-09-09T22:59:43.334 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083cfbe73e8847aa1-00631bc55f-1403-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.335 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:43.336 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:43.337 DEBUG:boto:Method: HEAD 2022-09-09T22:59:43.338 DEBUG:boto:Path: /jkcgbj-42/ 2022-09-09T22:59:43.339 DEBUG:boto:Data: 2022-09-09T22:59:43.340 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.340 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:43.341 DEBUG:boto:Port: 8000 2022-09-09T22:59:43.341 DEBUG:boto:Params: {} 2022-09-09T22:59:43.342 DEBUG:boto:Token: None 2022-09-09T22:59:43.342 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-42/ 2022-09-09T22:59:43.343 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LH7B7/QDu5ZlrZ8Eg1dQ9icD9XY= 2022-09-09T22:59:43.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LH7B7/QDu5ZlrZ8Eg1dQ9icD9XY=', 'Content-Length': '0'} 2022-09-09T22:59:43.348 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af2fdff823756692-00631bc55f-1549-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.348 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:43.349 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:43.349 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:43.350 DEBUG:boto:path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.350 DEBUG:boto:auth_path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.351 DEBUG:boto:Method: GET 2022-09-09T22:59:43.351 DEBUG:boto:Path: /jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.352 DEBUG:boto:Data: 2022-09-09T22:59:43.352 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.353 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:43.353 DEBUG:boto:Port: 8001 2022-09-09T22:59:43.354 DEBUG:boto:Params: {} 2022-09-09T22:59:43.355 DEBUG:boto:Token: None 2022-09-09T22:59:43.355 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-42/?versions 2022-09-09T22:59:43.356 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo= 2022-09-09T22:59:43.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo=', 'Content-Length': '0'} 2022-09-09T22:59:43.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cae979e520e0cbd1-00631bc55f-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.363 DEBUG:boto:b'jkcgbj-42999false' 2022-09-09T22:59:43.364 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:43.364 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:43.365 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:43.365 DEBUG:boto:path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.366 DEBUG:boto:auth_path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.366 DEBUG:boto:Method: GET 2022-09-09T22:59:43.367 DEBUG:boto:Path: /jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.367 DEBUG:boto:Data: 2022-09-09T22:59:43.368 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.368 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:43.369 DEBUG:boto:Port: 8000 2022-09-09T22:59:43.369 DEBUG:boto:Params: {} 2022-09-09T22:59:43.370 DEBUG:boto:Token: None 2022-09-09T22:59:43.371 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-42/?versions 2022-09-09T22:59:43.371 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo= 2022-09-09T22:59:43.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo=', 'Content-Length': '0'} 2022-09-09T22:59:43.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003350c965ac69dfc5-00631bc55f-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.418 DEBUG:boto:b'jkcgbj-42999false' 2022-09-09T22:59:43.420 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:43.421 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:43.422 DEBUG:boto:path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.423 DEBUG:boto:auth_path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.424 DEBUG:boto:Method: GET 2022-09-09T22:59:43.425 DEBUG:boto:Path: /jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.425 DEBUG:boto:Data: 2022-09-09T22:59:43.426 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.427 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:43.427 DEBUG:boto:Port: 8001 2022-09-09T22:59:43.428 DEBUG:boto:Params: {} 2022-09-09T22:59:43.429 DEBUG:boto:Token: None 2022-09-09T22:59:43.429 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-42/?versions 2022-09-09T22:59:43.430 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo= 2022-09-09T22:59:43.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo=', 'Content-Length': '0'} 2022-09-09T22:59:43.436 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e1d7348a683c9c5-00631bc55f-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.437 DEBUG:boto:b'jkcgbj-42999false' 2022-09-09T22:59:43.437 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:43.438 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:43.439 DEBUG:boto:path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.440 DEBUG:boto:auth_path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.441 DEBUG:boto:Method: GET 2022-09-09T22:59:43.442 DEBUG:boto:Path: /jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.442 DEBUG:boto:Data: 2022-09-09T22:59:43.443 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.443 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:43.444 DEBUG:boto:Port: 8000 2022-09-09T22:59:43.444 DEBUG:boto:Params: {} 2022-09-09T22:59:43.445 DEBUG:boto:Token: None 2022-09-09T22:59:43.445 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-42/?versions 2022-09-09T22:59:43.446 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo= 2022-09-09T22:59:43.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo=', 'Content-Length': '0'} 2022-09-09T22:59:43.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000159d209aa71ad3fa-00631bc55f-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.459 DEBUG:boto:b'jkcgbj-42999false' 2022-09-09T22:59:43.460 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-42 zones={test-zone3, test-zone2} 2022-09-09T22:59:43.460 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-42 zones={test-zone1, test-zone3} 2022-09-09T22:59:43.461 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:43.462 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:43.463 DEBUG:boto:Method: HEAD 2022-09-09T22:59:43.464 DEBUG:boto:Path: /jkcgbj-42/ 2022-09-09T22:59:43.464 DEBUG:boto:Data: 2022-09-09T22:59:43.465 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.466 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:43.467 DEBUG:boto:Port: 8000 2022-09-09T22:59:43.468 DEBUG:boto:Params: {} 2022-09-09T22:59:43.469 DEBUG:boto:Token: None 2022-09-09T22:59:43.470 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-42/ 2022-09-09T22:59:43.471 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LH7B7/QDu5ZlrZ8Eg1dQ9icD9XY= 2022-09-09T22:59:43.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LH7B7/QDu5ZlrZ8Eg1dQ9icD9XY=', 'Content-Length': '0'} 2022-09-09T22:59:43.477 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002c25a86c08142db-00631bc55f-13fd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.477 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:43.478 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:43.478 DEBUG:boto:Method: HEAD 2022-09-09T22:59:43.479 DEBUG:boto:Path: /jkcgbj-42/ 2022-09-09T22:59:43.480 DEBUG:boto:Data: 2022-09-09T22:59:43.480 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.481 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:43.481 DEBUG:boto:Port: 8001 2022-09-09T22:59:43.482 DEBUG:boto:Params: {} 2022-09-09T22:59:43.482 DEBUG:boto:Token: None 2022-09-09T22:59:43.483 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-42/ 2022-09-09T22:59:43.484 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LH7B7/QDu5ZlrZ8Eg1dQ9icD9XY= 2022-09-09T22:59:43.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LH7B7/QDu5ZlrZ8Eg1dQ9icD9XY=', 'Content-Length': '0'} 2022-09-09T22:59:43.489 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000540ade89db085be-00631bc55f-1403-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.490 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:43.491 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:43.492 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:43.492 DEBUG:boto:path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.493 DEBUG:boto:auth_path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.493 DEBUG:boto:Method: GET 2022-09-09T22:59:43.494 DEBUG:boto:Path: /jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.494 DEBUG:boto:Data: 2022-09-09T22:59:43.495 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.495 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:43.496 DEBUG:boto:Port: 8000 2022-09-09T22:59:43.496 DEBUG:boto:Params: {} 2022-09-09T22:59:43.497 DEBUG:boto:Token: None 2022-09-09T22:59:43.497 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-42/?versions 2022-09-09T22:59:43.498 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo= 2022-09-09T22:59:43.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo=', 'Content-Length': '0'} 2022-09-09T22:59:43.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e690dac6c5f96fb-00631bc55f-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.505 DEBUG:boto:b'jkcgbj-42999false' 2022-09-09T22:59:43.506 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:43.507 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:43.507 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:43.508 DEBUG:boto:path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.509 DEBUG:boto:auth_path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.509 DEBUG:boto:Method: GET 2022-09-09T22:59:43.510 DEBUG:boto:Path: /jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.510 DEBUG:boto:Data: 2022-09-09T22:59:43.511 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.511 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:43.512 DEBUG:boto:Port: 8001 2022-09-09T22:59:43.512 DEBUG:boto:Params: {} 2022-09-09T22:59:43.513 DEBUG:boto:Token: None 2022-09-09T22:59:43.513 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-42/?versions 2022-09-09T22:59:43.514 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo= 2022-09-09T22:59:43.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo=', 'Content-Length': '0'} 2022-09-09T22:59:43.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030e2598b7540fa43-00631bc55f-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.562 DEBUG:boto:b'jkcgbj-42999false' 2022-09-09T22:59:43.563 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:43.564 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:43.564 DEBUG:boto:path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.565 DEBUG:boto:auth_path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.565 DEBUG:boto:Method: GET 2022-09-09T22:59:43.566 DEBUG:boto:Path: /jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.567 DEBUG:boto:Data: 2022-09-09T22:59:43.569 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.570 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:43.571 DEBUG:boto:Port: 8000 2022-09-09T22:59:43.571 DEBUG:boto:Params: {} 2022-09-09T22:59:43.572 DEBUG:boto:Token: None 2022-09-09T22:59:43.573 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-42/?versions 2022-09-09T22:59:43.575 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo= 2022-09-09T22:59:43.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo=', 'Content-Length': '0'} 2022-09-09T22:59:43.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006654b5b2ed78cff3-00631bc55f-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.583 DEBUG:boto:b'jkcgbj-42999false' 2022-09-09T22:59:43.585 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:43.586 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:43.587 DEBUG:boto:path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.588 DEBUG:boto:auth_path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.588 DEBUG:boto:Method: GET 2022-09-09T22:59:43.589 DEBUG:boto:Path: /jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.590 DEBUG:boto:Data: 2022-09-09T22:59:43.591 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.592 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:43.593 DEBUG:boto:Port: 8001 2022-09-09T22:59:43.594 DEBUG:boto:Params: {} 2022-09-09T22:59:43.595 DEBUG:boto:Token: None 2022-09-09T22:59:43.596 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-42/?versions 2022-09-09T22:59:43.597 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo= 2022-09-09T22:59:43.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo=', 'Content-Length': '0'} 2022-09-09T22:59:43.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1ae334a409d5a9a-00631bc55f-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.606 DEBUG:boto:b'jkcgbj-42999false' 2022-09-09T22:59:43.607 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-42 zones={test-zone1, test-zone3} 2022-09-09T22:59:43.608 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-42 zones={test-zone2, test-zone3} 2022-09-09T22:59:43.609 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:43.609 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:43.610 DEBUG:boto:Method: HEAD 2022-09-09T22:59:43.610 DEBUG:boto:Path: /jkcgbj-42/ 2022-09-09T22:59:43.611 DEBUG:boto:Data: 2022-09-09T22:59:43.611 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.612 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:43.612 DEBUG:boto:Port: 8000 2022-09-09T22:59:43.613 DEBUG:boto:Params: {} 2022-09-09T22:59:43.614 DEBUG:boto:Token: None 2022-09-09T22:59:43.615 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-42/ 2022-09-09T22:59:43.616 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LH7B7/QDu5ZlrZ8Eg1dQ9icD9XY= 2022-09-09T22:59:43.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LH7B7/QDu5ZlrZ8Eg1dQ9icD9XY=', 'Content-Length': '0'} 2022-09-09T22:59:43.621 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ce2630f9223ed08-00631bc55f-1549-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.622 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:43.622 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:43.623 DEBUG:boto:Method: HEAD 2022-09-09T22:59:43.624 DEBUG:boto:Path: /jkcgbj-42/ 2022-09-09T22:59:43.624 DEBUG:boto:Data: 2022-09-09T22:59:43.624 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.625 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:43.625 DEBUG:boto:Port: 8001 2022-09-09T22:59:43.626 DEBUG:boto:Params: {} 2022-09-09T22:59:43.627 DEBUG:boto:Token: None 2022-09-09T22:59:43.627 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-42/ 2022-09-09T22:59:43.628 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:LH7B7/QDu5ZlrZ8Eg1dQ9icD9XY= 2022-09-09T22:59:43.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:LH7B7/QDu5ZlrZ8Eg1dQ9icD9XY=', 'Content-Length': '0'} 2022-09-09T22:59:43.633 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f4708e764176b26-00631bc55f-1403-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.634 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:43.635 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:43.635 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:43.636 DEBUG:boto:path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.636 DEBUG:boto:auth_path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.637 DEBUG:boto:Method: GET 2022-09-09T22:59:43.637 DEBUG:boto:Path: /jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.638 DEBUG:boto:Data: 2022-09-09T22:59:43.638 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.639 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:43.639 DEBUG:boto:Port: 8000 2022-09-09T22:59:43.640 DEBUG:boto:Params: {} 2022-09-09T22:59:43.641 DEBUG:boto:Token: None 2022-09-09T22:59:43.642 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-42/?versions 2022-09-09T22:59:43.642 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo= 2022-09-09T22:59:43.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo=', 'Content-Length': '0'} 2022-09-09T22:59:43.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000978476b4953002-00631bc55f-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.659 DEBUG:boto:b'jkcgbj-42999false' 2022-09-09T22:59:43.660 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:43.660 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:43.661 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:43.661 DEBUG:boto:path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.662 DEBUG:boto:auth_path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.662 DEBUG:boto:Method: GET 2022-09-09T22:59:43.663 DEBUG:boto:Path: /jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.664 DEBUG:boto:Data: 2022-09-09T22:59:43.664 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.665 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:43.665 DEBUG:boto:Port: 8001 2022-09-09T22:59:43.666 DEBUG:boto:Params: {} 2022-09-09T22:59:43.666 DEBUG:boto:Token: None 2022-09-09T22:59:43.667 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-42/?versions 2022-09-09T22:59:43.668 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo= 2022-09-09T22:59:43.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo=', 'Content-Length': '0'} 2022-09-09T22:59:43.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042871486da8e0d36-00631bc55f-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.718 DEBUG:boto:b'jkcgbj-42999false' 2022-09-09T22:59:43.720 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:43.721 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:43.721 DEBUG:boto:path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.722 DEBUG:boto:auth_path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.723 DEBUG:boto:Method: GET 2022-09-09T22:59:43.724 DEBUG:boto:Path: /jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.725 DEBUG:boto:Data: 2022-09-09T22:59:43.726 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.727 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:43.728 DEBUG:boto:Port: 8000 2022-09-09T22:59:43.729 DEBUG:boto:Params: {} 2022-09-09T22:59:43.730 DEBUG:boto:Token: None 2022-09-09T22:59:43.731 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-42/?versions 2022-09-09T22:59:43.732 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo= 2022-09-09T22:59:43.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo=', 'Content-Length': '0'} 2022-09-09T22:59:43.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000305c765c4b4c4124-00631bc55f-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.739 DEBUG:boto:b'jkcgbj-42999false' 2022-09-09T22:59:43.741 DEBUG:boto:path=/jkcgbj-42/ 2022-09-09T22:59:43.741 DEBUG:boto:auth_path=/jkcgbj-42/ 2022-09-09T22:59:43.742 DEBUG:boto:path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.742 DEBUG:boto:auth_path=/jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.743 DEBUG:boto:Method: GET 2022-09-09T22:59:43.743 DEBUG:boto:Path: /jkcgbj-42/?versions&max-keys=999 2022-09-09T22:59:43.744 DEBUG:boto:Data: 2022-09-09T22:59:43.744 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.745 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:43.745 DEBUG:boto:Port: 8001 2022-09-09T22:59:43.746 DEBUG:boto:Params: {} 2022-09-09T22:59:43.747 DEBUG:boto:Token: None 2022-09-09T22:59:43.747 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-42/?versions 2022-09-09T22:59:43.748 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo= 2022-09-09T22:59:43.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dqlCiJHohNgFO4ScTIGkVtKGwEo=', 'Content-Length': '0'} 2022-09-09T22:59:43.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028233c11a1ecfe06-00631bc55f-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.755 DEBUG:boto:b'jkcgbj-42999false' 2022-09-09T22:59:43.756 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-42 zones={test-zone2, test-zone3} 2022-09-09T22:59:43.757 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-43 zones={test-zone2, test-zone1} 2022-09-09T22:59:43.758 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:43.758 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:43.759 DEBUG:boto:Method: HEAD 2022-09-09T22:59:43.759 DEBUG:boto:Path: /jkcgbj-43/ 2022-09-09T22:59:43.760 DEBUG:boto:Data: 2022-09-09T22:59:43.760 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.761 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:43.761 DEBUG:boto:Port: 8000 2022-09-09T22:59:43.762 DEBUG:boto:Params: {} 2022-09-09T22:59:43.763 DEBUG:boto:Token: None 2022-09-09T22:59:43.763 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-43/ 2022-09-09T22:59:43.764 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:W4sq9cmZN1IqF93Ee5Q1iXbmLlk= 2022-09-09T22:59:43.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:W4sq9cmZN1IqF93Ee5Q1iXbmLlk=', 'Content-Length': '0'} 2022-09-09T22:59:43.769 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ab1ee14344352314-00631bc55f-1549-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.770 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:43.770 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:43.771 DEBUG:boto:Method: HEAD 2022-09-09T22:59:43.771 DEBUG:boto:Path: /jkcgbj-43/ 2022-09-09T22:59:43.772 DEBUG:boto:Data: 2022-09-09T22:59:43.772 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.773 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:43.774 DEBUG:boto:Port: 8000 2022-09-09T22:59:43.774 DEBUG:boto:Params: {} 2022-09-09T22:59:43.775 DEBUG:boto:Token: None 2022-09-09T22:59:43.776 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-43/ 2022-09-09T22:59:43.777 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:W4sq9cmZN1IqF93Ee5Q1iXbmLlk= 2022-09-09T22:59:43.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:W4sq9cmZN1IqF93Ee5Q1iXbmLlk=', 'Content-Length': '0'} 2022-09-09T22:59:43.786 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000028cb9c52fbda5622-00631bc55f-13fd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.788 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:43.788 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:43.790 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:43.790 DEBUG:boto:path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:43.791 DEBUG:boto:auth_path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:43.792 DEBUG:boto:Method: GET 2022-09-09T22:59:43.793 DEBUG:boto:Path: /jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:43.794 DEBUG:boto:Data: 2022-09-09T22:59:43.795 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.795 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:43.796 DEBUG:boto:Port: 8000 2022-09-09T22:59:43.797 DEBUG:boto:Params: {} 2022-09-09T22:59:43.798 DEBUG:boto:Token: None 2022-09-09T22:59:43.799 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-43/?versions 2022-09-09T22:59:43.800 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OKJ/ETzPj2KdjslQGNOzwj7BlmQ= 2022-09-09T22:59:43.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OKJ/ETzPj2KdjslQGNOzwj7BlmQ=', 'Content-Length': '0'} 2022-09-09T22:59:43.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002480dc787e15628-00631bc55f-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.850 DEBUG:boto:b'jkcgbj-43999false' 2022-09-09T22:59:43.852 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:43.853 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:43.854 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:43.855 DEBUG:boto:path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:43.856 DEBUG:boto:auth_path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:43.856 DEBUG:boto:Method: GET 2022-09-09T22:59:43.857 DEBUG:boto:Path: /jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:43.857 DEBUG:boto:Data: 2022-09-09T22:59:43.858 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.858 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:43.859 DEBUG:boto:Port: 8000 2022-09-09T22:59:43.859 DEBUG:boto:Params: {} 2022-09-09T22:59:43.860 DEBUG:boto:Token: None 2022-09-09T22:59:43.861 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-43/?versions 2022-09-09T22:59:43.862 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OKJ/ETzPj2KdjslQGNOzwj7BlmQ= 2022-09-09T22:59:43.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OKJ/ETzPj2KdjslQGNOzwj7BlmQ=', 'Content-Length': '0'} 2022-09-09T22:59:43.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a8b66eca1f774cc-00631bc55f-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.870 DEBUG:boto:b'jkcgbj-43999false' 2022-09-09T22:59:43.871 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:43.872 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:43.872 DEBUG:boto:path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:43.873 DEBUG:boto:auth_path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:43.874 DEBUG:boto:Method: GET 2022-09-09T22:59:43.874 DEBUG:boto:Path: /jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:43.875 DEBUG:boto:Data: 2022-09-09T22:59:43.876 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.877 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:43.878 DEBUG:boto:Port: 8000 2022-09-09T22:59:43.879 DEBUG:boto:Params: {} 2022-09-09T22:59:43.880 DEBUG:boto:Token: None 2022-09-09T22:59:43.881 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-43/?versions 2022-09-09T22:59:43.882 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OKJ/ETzPj2KdjslQGNOzwj7BlmQ= 2022-09-09T22:59:43.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OKJ/ETzPj2KdjslQGNOzwj7BlmQ=', 'Content-Length': '0'} 2022-09-09T22:59:43.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006960f90315763833-00631bc55f-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.891 DEBUG:boto:b'jkcgbj-43999false' 2022-09-09T22:59:43.892 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:43.893 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:43.894 DEBUG:boto:path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:43.895 DEBUG:boto:auth_path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:43.896 DEBUG:boto:Method: GET 2022-09-09T22:59:43.897 DEBUG:boto:Path: /jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:43.898 DEBUG:boto:Data: 2022-09-09T22:59:43.899 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.900 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:43.901 DEBUG:boto:Port: 8000 2022-09-09T22:59:43.902 DEBUG:boto:Params: {} 2022-09-09T22:59:43.903 DEBUG:boto:Token: None 2022-09-09T22:59:43.903 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-43/?versions 2022-09-09T22:59:43.905 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OKJ/ETzPj2KdjslQGNOzwj7BlmQ= 2022-09-09T22:59:43.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OKJ/ETzPj2KdjslQGNOzwj7BlmQ=', 'Content-Length': '0'} 2022-09-09T22:59:43.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000988bb38b9544b0d6-00631bc55f-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.912 DEBUG:boto:b'jkcgbj-43999false' 2022-09-09T22:59:43.913 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-43 zones={test-zone2, test-zone1} 2022-09-09T22:59:43.913 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-43 zones={test-zone3, test-zone1} 2022-09-09T22:59:43.914 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:43.914 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:43.915 DEBUG:boto:Method: HEAD 2022-09-09T22:59:43.915 DEBUG:boto:Path: /jkcgbj-43/ 2022-09-09T22:59:43.916 DEBUG:boto:Data: 2022-09-09T22:59:43.916 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.917 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:43.917 DEBUG:boto:Port: 8001 2022-09-09T22:59:43.918 DEBUG:boto:Params: {} 2022-09-09T22:59:43.918 DEBUG:boto:Token: None 2022-09-09T22:59:43.919 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-43/ 2022-09-09T22:59:43.920 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:W4sq9cmZN1IqF93Ee5Q1iXbmLlk= 2022-09-09T22:59:43.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:W4sq9cmZN1IqF93Ee5Q1iXbmLlk=', 'Content-Length': '0'} 2022-09-09T22:59:43.925 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e481e3d08523484b-00631bc55f-1403-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.927 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:43.928 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:43.928 DEBUG:boto:Method: HEAD 2022-09-09T22:59:43.929 DEBUG:boto:Path: /jkcgbj-43/ 2022-09-09T22:59:43.930 DEBUG:boto:Data: 2022-09-09T22:59:43.931 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.932 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:43.932 DEBUG:boto:Port: 8000 2022-09-09T22:59:43.933 DEBUG:boto:Params: {} 2022-09-09T22:59:43.934 DEBUG:boto:Token: None 2022-09-09T22:59:43.935 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-43/ 2022-09-09T22:59:43.936 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:W4sq9cmZN1IqF93Ee5Q1iXbmLlk= 2022-09-09T22:59:43.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:W4sq9cmZN1IqF93Ee5Q1iXbmLlk=', 'Content-Length': '0'} 2022-09-09T22:59:43.942 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000069e64377711a6fb-00631bc55f-13fd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.943 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:43.943 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:43.944 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:43.944 DEBUG:boto:path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:43.945 DEBUG:boto:auth_path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:43.945 DEBUG:boto:Method: GET 2022-09-09T22:59:43.946 DEBUG:boto:Path: /jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:43.947 DEBUG:boto:Data: 2022-09-09T22:59:43.948 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.949 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:43.950 DEBUG:boto:Port: 8001 2022-09-09T22:59:43.951 DEBUG:boto:Params: {} 2022-09-09T22:59:43.952 DEBUG:boto:Token: None 2022-09-09T22:59:43.953 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-43/?versions 2022-09-09T22:59:43.954 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OKJ/ETzPj2KdjslQGNOzwj7BlmQ= 2022-09-09T22:59:43.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OKJ/ETzPj2KdjslQGNOzwj7BlmQ=', 'Content-Length': '0'} 2022-09-09T22:59:43.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ad88a5d31674bb8-00631bc55f-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:43.963 DEBUG:boto:b'jkcgbj-43999false' 2022-09-09T22:59:43.964 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:43.965 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:43.966 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:43.967 DEBUG:boto:path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:43.968 DEBUG:boto:auth_path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:43.969 DEBUG:boto:Method: GET 2022-09-09T22:59:43.970 DEBUG:boto:Path: /jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:43.971 DEBUG:boto:Data: 2022-09-09T22:59:43.972 DEBUG:boto:Headers: {} 2022-09-09T22:59:43.973 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:43.974 DEBUG:boto:Port: 8000 2022-09-09T22:59:43.975 DEBUG:boto:Params: {} 2022-09-09T22:59:43.976 DEBUG:boto:Token: None 2022-09-09T22:59:43.977 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:43 GMT /jkcgbj-43/?versions 2022-09-09T22:59:43.978 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OKJ/ETzPj2KdjslQGNOzwj7BlmQ= 2022-09-09T22:59:43.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:43 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OKJ/ETzPj2KdjslQGNOzwj7BlmQ=', 'Content-Length': '0'} 2022-09-09T22:59:43.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0877e95dcf41b9f-00631bc55f-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.030 DEBUG:boto:b'jkcgbj-43999false' 2022-09-09T22:59:44.031 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:44.032 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:44.032 DEBUG:boto:path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.033 DEBUG:boto:auth_path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.033 DEBUG:boto:Method: GET 2022-09-09T22:59:44.034 DEBUG:boto:Path: /jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.034 DEBUG:boto:Data: 2022-09-09T22:59:44.035 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.035 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:44.036 DEBUG:boto:Port: 8001 2022-09-09T22:59:44.036 DEBUG:boto:Params: {} 2022-09-09T22:59:44.037 DEBUG:boto:Token: None 2022-09-09T22:59:44.038 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-43/?versions 2022-09-09T22:59:44.038 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw= 2022-09-09T22:59:44.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw=', 'Content-Length': '0'} 2022-09-09T22:59:44.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003c463fb15ee9d11-00631bc560-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.045 DEBUG:boto:b'jkcgbj-43999false' 2022-09-09T22:59:44.046 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:44.046 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:44.047 DEBUG:boto:path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.047 DEBUG:boto:auth_path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.048 DEBUG:boto:Method: GET 2022-09-09T22:59:44.048 DEBUG:boto:Path: /jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.049 DEBUG:boto:Data: 2022-09-09T22:59:44.049 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.050 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:44.051 DEBUG:boto:Port: 8000 2022-09-09T22:59:44.051 DEBUG:boto:Params: {} 2022-09-09T22:59:44.052 DEBUG:boto:Token: None 2022-09-09T22:59:44.052 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-43/?versions 2022-09-09T22:59:44.053 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw= 2022-09-09T22:59:44.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw=', 'Content-Length': '0'} 2022-09-09T22:59:44.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000019a950c90f4be68-00631bc560-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.059 DEBUG:boto:b'jkcgbj-43999false' 2022-09-09T22:59:44.060 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-43 zones={test-zone3, test-zone1} 2022-09-09T22:59:44.061 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-43 zones={test-zone1, test-zone2} 2022-09-09T22:59:44.061 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:44.062 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:44.062 DEBUG:boto:Method: HEAD 2022-09-09T22:59:44.063 DEBUG:boto:Path: /jkcgbj-43/ 2022-09-09T22:59:44.063 DEBUG:boto:Data: 2022-09-09T22:59:44.064 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.065 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:44.065 DEBUG:boto:Port: 8000 2022-09-09T22:59:44.066 DEBUG:boto:Params: {} 2022-09-09T22:59:44.066 DEBUG:boto:Token: None 2022-09-09T22:59:44.068 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-43/ 2022-09-09T22:59:44.069 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jCBnZAUyNuRBhW+f2c8iIIEOwrM= 2022-09-09T22:59:44.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jCBnZAUyNuRBhW+f2c8iIIEOwrM=', 'Content-Length': '0'} 2022-09-09T22:59:44.074 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf2caf194087cd38-00631bc560-13fd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.075 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:44.076 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:44.076 DEBUG:boto:Method: HEAD 2022-09-09T22:59:44.077 DEBUG:boto:Path: /jkcgbj-43/ 2022-09-09T22:59:44.078 DEBUG:boto:Data: 2022-09-09T22:59:44.078 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.079 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:44.079 DEBUG:boto:Port: 8000 2022-09-09T22:59:44.080 DEBUG:boto:Params: {} 2022-09-09T22:59:44.081 DEBUG:boto:Token: None 2022-09-09T22:59:44.082 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-43/ 2022-09-09T22:59:44.083 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jCBnZAUyNuRBhW+f2c8iIIEOwrM= 2022-09-09T22:59:44.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jCBnZAUyNuRBhW+f2c8iIIEOwrM=', 'Content-Length': '0'} 2022-09-09T22:59:44.090 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ec46216caf63f35a-00631bc560-1549-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.091 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:44.092 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:44.093 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:44.093 DEBUG:boto:path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.096 DEBUG:boto:auth_path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.096 DEBUG:boto:Method: GET 2022-09-09T22:59:44.097 DEBUG:boto:Path: /jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.097 DEBUG:boto:Data: 2022-09-09T22:59:44.098 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.098 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:44.099 DEBUG:boto:Port: 8000 2022-09-09T22:59:44.099 DEBUG:boto:Params: {} 2022-09-09T22:59:44.100 DEBUG:boto:Token: None 2022-09-09T22:59:44.101 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-43/?versions 2022-09-09T22:59:44.101 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw= 2022-09-09T22:59:44.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw=', 'Content-Length': '0'} 2022-09-09T22:59:44.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd17556fd97a75a5-00631bc560-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.150 DEBUG:boto:b'jkcgbj-43999false' 2022-09-09T22:59:44.151 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:44.152 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:44.153 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:44.154 DEBUG:boto:path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.155 DEBUG:boto:auth_path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.155 DEBUG:boto:Method: GET 2022-09-09T22:59:44.156 DEBUG:boto:Path: /jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.157 DEBUG:boto:Data: 2022-09-09T22:59:44.158 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.159 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:44.160 DEBUG:boto:Port: 8000 2022-09-09T22:59:44.161 DEBUG:boto:Params: {} 2022-09-09T22:59:44.162 DEBUG:boto:Token: None 2022-09-09T22:59:44.163 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-43/?versions 2022-09-09T22:59:44.164 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw= 2022-09-09T22:59:44.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw=', 'Content-Length': '0'} 2022-09-09T22:59:44.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ad171954dac6e00-00631bc560-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.175 DEBUG:boto:b'jkcgbj-43999false' 2022-09-09T22:59:44.176 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:44.177 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:44.177 DEBUG:boto:path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.178 DEBUG:boto:auth_path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.178 DEBUG:boto:Method: GET 2022-09-09T22:59:44.179 DEBUG:boto:Path: /jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.179 DEBUG:boto:Data: 2022-09-09T22:59:44.180 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.181 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:44.181 DEBUG:boto:Port: 8000 2022-09-09T22:59:44.182 DEBUG:boto:Params: {} 2022-09-09T22:59:44.182 DEBUG:boto:Token: None 2022-09-09T22:59:44.183 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-43/?versions 2022-09-09T22:59:44.184 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw= 2022-09-09T22:59:44.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw=', 'Content-Length': '0'} 2022-09-09T22:59:44.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe3cbba11195b81a-00631bc560-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.191 DEBUG:boto:b'jkcgbj-43999false' 2022-09-09T22:59:44.192 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:44.193 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:44.194 DEBUG:boto:path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.194 DEBUG:boto:auth_path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.195 DEBUG:boto:Method: GET 2022-09-09T22:59:44.196 DEBUG:boto:Path: /jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.197 DEBUG:boto:Data: 2022-09-09T22:59:44.198 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.198 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:44.199 DEBUG:boto:Port: 8000 2022-09-09T22:59:44.199 DEBUG:boto:Params: {} 2022-09-09T22:59:44.200 DEBUG:boto:Token: None 2022-09-09T22:59:44.200 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-43/?versions 2022-09-09T22:59:44.201 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw= 2022-09-09T22:59:44.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw=', 'Content-Length': '0'} 2022-09-09T22:59:44.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000da3eaa052724389-00631bc560-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.207 DEBUG:boto:b'jkcgbj-43999false' 2022-09-09T22:59:44.208 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-43 zones={test-zone1, test-zone2} 2022-09-09T22:59:44.208 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-43 zones={test-zone3, test-zone2} 2022-09-09T22:59:44.209 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:44.209 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:44.210 DEBUG:boto:Method: HEAD 2022-09-09T22:59:44.211 DEBUG:boto:Path: /jkcgbj-43/ 2022-09-09T22:59:44.211 DEBUG:boto:Data: 2022-09-09T22:59:44.212 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.212 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:44.213 DEBUG:boto:Port: 8001 2022-09-09T22:59:44.213 DEBUG:boto:Params: {} 2022-09-09T22:59:44.214 DEBUG:boto:Token: None 2022-09-09T22:59:44.214 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-43/ 2022-09-09T22:59:44.215 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jCBnZAUyNuRBhW+f2c8iIIEOwrM= 2022-09-09T22:59:44.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jCBnZAUyNuRBhW+f2c8iIIEOwrM=', 'Content-Length': '0'} 2022-09-09T22:59:44.220 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000038293ef6d725554d-00631bc560-1403-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.222 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:44.223 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:44.224 DEBUG:boto:Method: HEAD 2022-09-09T22:59:44.225 DEBUG:boto:Path: /jkcgbj-43/ 2022-09-09T22:59:44.225 DEBUG:boto:Data: 2022-09-09T22:59:44.226 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.227 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:44.228 DEBUG:boto:Port: 8000 2022-09-09T22:59:44.229 DEBUG:boto:Params: {} 2022-09-09T22:59:44.230 DEBUG:boto:Token: None 2022-09-09T22:59:44.231 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-43/ 2022-09-09T22:59:44.232 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jCBnZAUyNuRBhW+f2c8iIIEOwrM= 2022-09-09T22:59:44.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jCBnZAUyNuRBhW+f2c8iIIEOwrM=', 'Content-Length': '0'} 2022-09-09T22:59:44.237 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a39d3c5d4159bee5-00631bc560-1549-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.238 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:44.238 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:44.239 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:44.239 DEBUG:boto:path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.240 DEBUG:boto:auth_path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.240 DEBUG:boto:Method: GET 2022-09-09T22:59:44.241 DEBUG:boto:Path: /jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.241 DEBUG:boto:Data: 2022-09-09T22:59:44.242 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.242 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:44.243 DEBUG:boto:Port: 8001 2022-09-09T22:59:44.243 DEBUG:boto:Params: {} 2022-09-09T22:59:44.244 DEBUG:boto:Token: None 2022-09-09T22:59:44.245 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-43/?versions 2022-09-09T22:59:44.246 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw= 2022-09-09T22:59:44.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw=', 'Content-Length': '0'} 2022-09-09T22:59:44.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a98c3b58c4ea2c2e-00631bc560-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.294 DEBUG:boto:b'jkcgbj-43999false' 2022-09-09T22:59:44.295 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:44.296 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:44.296 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:44.297 DEBUG:boto:path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.298 DEBUG:boto:auth_path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.299 DEBUG:boto:Method: GET 2022-09-09T22:59:44.300 DEBUG:boto:Path: /jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.300 DEBUG:boto:Data: 2022-09-09T22:59:44.301 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.302 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:44.303 DEBUG:boto:Port: 8000 2022-09-09T22:59:44.304 DEBUG:boto:Params: {} 2022-09-09T22:59:44.305 DEBUG:boto:Token: None 2022-09-09T22:59:44.306 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-43/?versions 2022-09-09T22:59:44.307 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw= 2022-09-09T22:59:44.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw=', 'Content-Length': '0'} 2022-09-09T22:59:44.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001583f39b3dcd552b-00631bc560-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.315 DEBUG:boto:b'jkcgbj-43999false' 2022-09-09T22:59:44.316 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:44.317 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:44.317 DEBUG:boto:path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.318 DEBUG:boto:auth_path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.318 DEBUG:boto:Method: GET 2022-09-09T22:59:44.319 DEBUG:boto:Path: /jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.320 DEBUG:boto:Data: 2022-09-09T22:59:44.320 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.321 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:44.321 DEBUG:boto:Port: 8001 2022-09-09T22:59:44.322 DEBUG:boto:Params: {} 2022-09-09T22:59:44.322 DEBUG:boto:Token: None 2022-09-09T22:59:44.332 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-43/?versions 2022-09-09T22:59:44.332 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw= 2022-09-09T22:59:44.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw=', 'Content-Length': '0'} 2022-09-09T22:59:44.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082f17e39ed683af9-00631bc560-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.341 DEBUG:boto:b'jkcgbj-43999false' 2022-09-09T22:59:44.342 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:44.342 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:44.343 DEBUG:boto:path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.343 DEBUG:boto:auth_path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.344 DEBUG:boto:Method: GET 2022-09-09T22:59:44.344 DEBUG:boto:Path: /jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.345 DEBUG:boto:Data: 2022-09-09T22:59:44.345 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.346 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:44.346 DEBUG:boto:Port: 8000 2022-09-09T22:59:44.347 DEBUG:boto:Params: {} 2022-09-09T22:59:44.348 DEBUG:boto:Token: None 2022-09-09T22:59:44.348 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-43/?versions 2022-09-09T22:59:44.349 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw= 2022-09-09T22:59:44.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw=', 'Content-Length': '0'} 2022-09-09T22:59:44.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000448cb5875191daee-00631bc560-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.356 DEBUG:boto:b'jkcgbj-43999false' 2022-09-09T22:59:44.356 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-43 zones={test-zone3, test-zone2} 2022-09-09T22:59:44.357 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-43 zones={test-zone1, test-zone3} 2022-09-09T22:59:44.357 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:44.358 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:44.359 DEBUG:boto:Method: HEAD 2022-09-09T22:59:44.359 DEBUG:boto:Path: /jkcgbj-43/ 2022-09-09T22:59:44.360 DEBUG:boto:Data: 2022-09-09T22:59:44.361 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.362 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:44.363 DEBUG:boto:Port: 8000 2022-09-09T22:59:44.364 DEBUG:boto:Params: {} 2022-09-09T22:59:44.365 DEBUG:boto:Token: None 2022-09-09T22:59:44.366 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-43/ 2022-09-09T22:59:44.367 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jCBnZAUyNuRBhW+f2c8iIIEOwrM= 2022-09-09T22:59:44.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jCBnZAUyNuRBhW+f2c8iIIEOwrM=', 'Content-Length': '0'} 2022-09-09T22:59:44.374 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f3051b8fdca0e96-00631bc560-13fd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.376 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:44.376 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:44.377 DEBUG:boto:Method: HEAD 2022-09-09T22:59:44.378 DEBUG:boto:Path: /jkcgbj-43/ 2022-09-09T22:59:44.379 DEBUG:boto:Data: 2022-09-09T22:59:44.380 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.381 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:44.382 DEBUG:boto:Port: 8001 2022-09-09T22:59:44.382 DEBUG:boto:Params: {} 2022-09-09T22:59:44.383 DEBUG:boto:Token: None 2022-09-09T22:59:44.383 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-43/ 2022-09-09T22:59:44.384 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jCBnZAUyNuRBhW+f2c8iIIEOwrM= 2022-09-09T22:59:44.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jCBnZAUyNuRBhW+f2c8iIIEOwrM=', 'Content-Length': '0'} 2022-09-09T22:59:44.389 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ce90e890dce8d149-00631bc560-1403-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.390 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:44.391 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:44.391 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:44.392 DEBUG:boto:path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.392 DEBUG:boto:auth_path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.393 DEBUG:boto:Method: GET 2022-09-09T22:59:44.393 DEBUG:boto:Path: /jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.394 DEBUG:boto:Data: 2022-09-09T22:59:44.394 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.395 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:44.395 DEBUG:boto:Port: 8000 2022-09-09T22:59:44.396 DEBUG:boto:Params: {} 2022-09-09T22:59:44.396 DEBUG:boto:Token: None 2022-09-09T22:59:44.397 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-43/?versions 2022-09-09T22:59:44.398 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw= 2022-09-09T22:59:44.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw=', 'Content-Length': '0'} 2022-09-09T22:59:44.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccf58289e0c7bc9d-00631bc560-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.405 DEBUG:boto:b'jkcgbj-43999false' 2022-09-09T22:59:44.406 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:44.406 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:44.407 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:44.407 DEBUG:boto:path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.408 DEBUG:boto:auth_path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.409 DEBUG:boto:Method: GET 2022-09-09T22:59:44.409 DEBUG:boto:Path: /jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.410 DEBUG:boto:Data: 2022-09-09T22:59:44.410 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.413 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:44.414 DEBUG:boto:Port: 8001 2022-09-09T22:59:44.415 DEBUG:boto:Params: {} 2022-09-09T22:59:44.416 DEBUG:boto:Token: None 2022-09-09T22:59:44.416 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-43/?versions 2022-09-09T22:59:44.418 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw= 2022-09-09T22:59:44.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw=', 'Content-Length': '0'} 2022-09-09T22:59:44.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b35a2254f7a230ba-00631bc560-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.427 DEBUG:boto:b'jkcgbj-43999false' 2022-09-09T22:59:44.428 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:44.428 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:44.429 DEBUG:boto:path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.429 DEBUG:boto:auth_path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.430 DEBUG:boto:Method: GET 2022-09-09T22:59:44.430 DEBUG:boto:Path: /jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.431 DEBUG:boto:Data: 2022-09-09T22:59:44.431 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.432 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:44.432 DEBUG:boto:Port: 8000 2022-09-09T22:59:44.433 DEBUG:boto:Params: {} 2022-09-09T22:59:44.434 DEBUG:boto:Token: None 2022-09-09T22:59:44.435 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-43/?versions 2022-09-09T22:59:44.435 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw= 2022-09-09T22:59:44.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw=', 'Content-Length': '0'} 2022-09-09T22:59:44.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a473d573cfb883c2-00631bc560-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.482 DEBUG:boto:b'jkcgbj-43999false' 2022-09-09T22:59:44.484 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:44.484 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:44.485 DEBUG:boto:path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.486 DEBUG:boto:auth_path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.487 DEBUG:boto:Method: GET 2022-09-09T22:59:44.488 DEBUG:boto:Path: /jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.489 DEBUG:boto:Data: 2022-09-09T22:59:44.490 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.491 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:44.492 DEBUG:boto:Port: 8001 2022-09-09T22:59:44.493 DEBUG:boto:Params: {} 2022-09-09T22:59:44.494 DEBUG:boto:Token: None 2022-09-09T22:59:44.495 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-43/?versions 2022-09-09T22:59:44.496 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw= 2022-09-09T22:59:44.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw=', 'Content-Length': '0'} 2022-09-09T22:59:44.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e249f1574006a78-00631bc560-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.506 DEBUG:boto:b'jkcgbj-43999false' 2022-09-09T22:59:44.507 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-43 zones={test-zone1, test-zone3} 2022-09-09T22:59:44.508 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-43 zones={test-zone2, test-zone3} 2022-09-09T22:59:44.509 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:44.510 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:44.510 DEBUG:boto:Method: HEAD 2022-09-09T22:59:44.511 DEBUG:boto:Path: /jkcgbj-43/ 2022-09-09T22:59:44.512 DEBUG:boto:Data: 2022-09-09T22:59:44.513 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.513 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:44.514 DEBUG:boto:Port: 8000 2022-09-09T22:59:44.514 DEBUG:boto:Params: {} 2022-09-09T22:59:44.515 DEBUG:boto:Token: None 2022-09-09T22:59:44.516 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-43/ 2022-09-09T22:59:44.516 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jCBnZAUyNuRBhW+f2c8iIIEOwrM= 2022-09-09T22:59:44.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jCBnZAUyNuRBhW+f2c8iIIEOwrM=', 'Content-Length': '0'} 2022-09-09T22:59:44.522 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000059b4c731f0318d10-00631bc560-1549-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.522 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:44.523 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:44.524 DEBUG:boto:Method: HEAD 2022-09-09T22:59:44.524 DEBUG:boto:Path: /jkcgbj-43/ 2022-09-09T22:59:44.525 DEBUG:boto:Data: 2022-09-09T22:59:44.526 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.527 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:44.528 DEBUG:boto:Port: 8001 2022-09-09T22:59:44.529 DEBUG:boto:Params: {} 2022-09-09T22:59:44.530 DEBUG:boto:Token: None 2022-09-09T22:59:44.531 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-43/ 2022-09-09T22:59:44.532 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jCBnZAUyNuRBhW+f2c8iIIEOwrM= 2022-09-09T22:59:44.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jCBnZAUyNuRBhW+f2c8iIIEOwrM=', 'Content-Length': '0'} 2022-09-09T22:59:44.538 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002c9af4682fe0263-00631bc560-1403-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.539 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:44.540 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:44.540 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:44.541 DEBUG:boto:path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.541 DEBUG:boto:auth_path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.542 DEBUG:boto:Method: GET 2022-09-09T22:59:44.543 DEBUG:boto:Path: /jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.543 DEBUG:boto:Data: 2022-09-09T22:59:44.544 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.544 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:44.545 DEBUG:boto:Port: 8000 2022-09-09T22:59:44.546 DEBUG:boto:Params: {} 2022-09-09T22:59:44.547 DEBUG:boto:Token: None 2022-09-09T22:59:44.548 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-43/?versions 2022-09-09T22:59:44.548 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw= 2022-09-09T22:59:44.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw=', 'Content-Length': '0'} 2022-09-09T22:59:44.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bda37b4b7971e22-00631bc560-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.557 DEBUG:boto:b'jkcgbj-43999false' 2022-09-09T22:59:44.558 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:44.559 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:44.559 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:44.560 DEBUG:boto:path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.560 DEBUG:boto:auth_path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.561 DEBUG:boto:Method: GET 2022-09-09T22:59:44.561 DEBUG:boto:Path: /jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.562 DEBUG:boto:Data: 2022-09-09T22:59:44.562 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.563 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:44.563 DEBUG:boto:Port: 8001 2022-09-09T22:59:44.564 DEBUG:boto:Params: {} 2022-09-09T22:59:44.564 DEBUG:boto:Token: None 2022-09-09T22:59:44.565 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-43/?versions 2022-09-09T22:59:44.565 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw= 2022-09-09T22:59:44.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw=', 'Content-Length': '0'} 2022-09-09T22:59:44.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000643dfb88b1a33e05-00631bc560-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.618 DEBUG:boto:b'jkcgbj-43999false' 2022-09-09T22:59:44.619 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:44.620 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:44.621 DEBUG:boto:path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.621 DEBUG:boto:auth_path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.622 DEBUG:boto:Method: GET 2022-09-09T22:59:44.622 DEBUG:boto:Path: /jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.623 DEBUG:boto:Data: 2022-09-09T22:59:44.623 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.624 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:44.624 DEBUG:boto:Port: 8000 2022-09-09T22:59:44.625 DEBUG:boto:Params: {} 2022-09-09T22:59:44.626 DEBUG:boto:Token: None 2022-09-09T22:59:44.626 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-43/?versions 2022-09-09T22:59:44.627 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw= 2022-09-09T22:59:44.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw=', 'Content-Length': '0'} 2022-09-09T22:59:44.632 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb6c07c7553919c9-00631bc560-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.632 DEBUG:boto:b'jkcgbj-43999false' 2022-09-09T22:59:44.633 DEBUG:boto:path=/jkcgbj-43/ 2022-09-09T22:59:44.634 DEBUG:boto:auth_path=/jkcgbj-43/ 2022-09-09T22:59:44.634 DEBUG:boto:path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.635 DEBUG:boto:auth_path=/jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.635 DEBUG:boto:Method: GET 2022-09-09T22:59:44.636 DEBUG:boto:Path: /jkcgbj-43/?versions&max-keys=999 2022-09-09T22:59:44.637 DEBUG:boto:Data: 2022-09-09T22:59:44.637 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.638 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:44.638 DEBUG:boto:Port: 8001 2022-09-09T22:59:44.639 DEBUG:boto:Params: {} 2022-09-09T22:59:44.639 DEBUG:boto:Token: None 2022-09-09T22:59:44.640 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-43/?versions 2022-09-09T22:59:44.640 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw= 2022-09-09T22:59:44.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:CrITFrA6rMjoXd2oCAyPZi1mVLw=', 'Content-Length': '0'} 2022-09-09T22:59:44.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005645904588f0a3ad-00631bc560-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.651 DEBUG:boto:b'jkcgbj-43999false' 2022-09-09T22:59:44.652 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-43 zones={test-zone2, test-zone3} 2022-09-09T22:59:44.654 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-44 zones={test-zone2, test-zone1} 2022-09-09T22:59:44.654 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:44.655 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:44.655 DEBUG:boto:Method: HEAD 2022-09-09T22:59:44.656 DEBUG:boto:Path: /jkcgbj-44/ 2022-09-09T22:59:44.657 DEBUG:boto:Data: 2022-09-09T22:59:44.657 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.658 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:44.658 DEBUG:boto:Port: 8000 2022-09-09T22:59:44.659 DEBUG:boto:Params: {} 2022-09-09T22:59:44.659 DEBUG:boto:Token: None 2022-09-09T22:59:44.660 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-44/ 2022-09-09T22:59:44.661 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Ygo5jWtxjEplxkZToPMA/dFcPrg= 2022-09-09T22:59:44.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Ygo5jWtxjEplxkZToPMA/dFcPrg=', 'Content-Length': '0'} 2022-09-09T22:59:44.666 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b3f5a9d8bb090b0-00631bc560-1549-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.667 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:44.667 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:44.668 DEBUG:boto:Method: HEAD 2022-09-09T22:59:44.668 DEBUG:boto:Path: /jkcgbj-44/ 2022-09-09T22:59:44.669 DEBUG:boto:Data: 2022-09-09T22:59:44.669 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.670 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:44.671 DEBUG:boto:Port: 8000 2022-09-09T22:59:44.671 DEBUG:boto:Params: {} 2022-09-09T22:59:44.672 DEBUG:boto:Token: None 2022-09-09T22:59:44.672 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-44/ 2022-09-09T22:59:44.673 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Ygo5jWtxjEplxkZToPMA/dFcPrg= 2022-09-09T22:59:44.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Ygo5jWtxjEplxkZToPMA/dFcPrg=', 'Content-Length': '0'} 2022-09-09T22:59:44.680 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006bf39707dab47749-00631bc560-13fd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.682 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:44.683 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:44.683 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:44.684 DEBUG:boto:path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:44.684 DEBUG:boto:auth_path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:44.685 DEBUG:boto:Method: GET 2022-09-09T22:59:44.685 DEBUG:boto:Path: /jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:44.686 DEBUG:boto:Data: 2022-09-09T22:59:44.687 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.687 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:44.688 DEBUG:boto:Port: 8000 2022-09-09T22:59:44.688 DEBUG:boto:Params: {} 2022-09-09T22:59:44.689 DEBUG:boto:Token: None 2022-09-09T22:59:44.690 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-44/?versions 2022-09-09T22:59:44.691 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:RKS34XfHh0/20an+PZf9dYDYTeE= 2022-09-09T22:59:44.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:RKS34XfHh0/20an+PZf9dYDYTeE=', 'Content-Length': '0'} 2022-09-09T22:59:44.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002db4ab98aa0955c2-00631bc560-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.742 DEBUG:boto:b'jkcgbj-44999false' 2022-09-09T22:59:44.744 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:44.745 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:44.746 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:44.747 DEBUG:boto:path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:44.748 DEBUG:boto:auth_path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:44.749 DEBUG:boto:Method: GET 2022-09-09T22:59:44.750 DEBUG:boto:Path: /jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:44.751 DEBUG:boto:Data: 2022-09-09T22:59:44.752 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.753 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:44.754 DEBUG:boto:Port: 8000 2022-09-09T22:59:44.755 DEBUG:boto:Params: {} 2022-09-09T22:59:44.756 DEBUG:boto:Token: None 2022-09-09T22:59:44.757 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-44/?versions 2022-09-09T22:59:44.758 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:RKS34XfHh0/20an+PZf9dYDYTeE= 2022-09-09T22:59:44.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:RKS34XfHh0/20an+PZf9dYDYTeE=', 'Content-Length': '0'} 2022-09-09T22:59:44.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063d62639f178696a-00631bc560-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.766 DEBUG:boto:b'jkcgbj-44999false' 2022-09-09T22:59:44.768 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:44.769 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:44.771 DEBUG:boto:path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:44.772 DEBUG:boto:auth_path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:44.773 DEBUG:boto:Method: GET 2022-09-09T22:59:44.774 DEBUG:boto:Path: /jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:44.775 DEBUG:boto:Data: 2022-09-09T22:59:44.776 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.777 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:44.778 DEBUG:boto:Port: 8000 2022-09-09T22:59:44.779 DEBUG:boto:Params: {} 2022-09-09T22:59:44.780 DEBUG:boto:Token: None 2022-09-09T22:59:44.781 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-44/?versions 2022-09-09T22:59:44.782 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:RKS34XfHh0/20an+PZf9dYDYTeE= 2022-09-09T22:59:44.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:RKS34XfHh0/20an+PZf9dYDYTeE=', 'Content-Length': '0'} 2022-09-09T22:59:44.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025ba1715219b1510-00631bc560-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.790 DEBUG:boto:b'jkcgbj-44999false' 2022-09-09T22:59:44.792 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:44.792 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:44.793 DEBUG:boto:path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:44.794 DEBUG:boto:auth_path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:44.795 DEBUG:boto:Method: GET 2022-09-09T22:59:44.796 DEBUG:boto:Path: /jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:44.797 DEBUG:boto:Data: 2022-09-09T22:59:44.798 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.799 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:44.800 DEBUG:boto:Port: 8000 2022-09-09T22:59:44.801 DEBUG:boto:Params: {} 2022-09-09T22:59:44.801 DEBUG:boto:Token: None 2022-09-09T22:59:44.803 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-44/?versions 2022-09-09T22:59:44.803 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:RKS34XfHh0/20an+PZf9dYDYTeE= 2022-09-09T22:59:44.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:RKS34XfHh0/20an+PZf9dYDYTeE=', 'Content-Length': '0'} 2022-09-09T22:59:44.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cfd2edb902a8f64-00631bc560-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.811 DEBUG:boto:b'jkcgbj-44999false' 2022-09-09T22:59:44.812 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-44 zones={test-zone2, test-zone1} 2022-09-09T22:59:44.813 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-44 zones={test-zone3, test-zone1} 2022-09-09T22:59:44.813 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:44.814 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:44.814 DEBUG:boto:Method: HEAD 2022-09-09T22:59:44.815 DEBUG:boto:Path: /jkcgbj-44/ 2022-09-09T22:59:44.816 DEBUG:boto:Data: 2022-09-09T22:59:44.816 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.817 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:44.817 DEBUG:boto:Port: 8001 2022-09-09T22:59:44.818 DEBUG:boto:Params: {} 2022-09-09T22:59:44.819 DEBUG:boto:Token: None 2022-09-09T22:59:44.819 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-44/ 2022-09-09T22:59:44.820 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Ygo5jWtxjEplxkZToPMA/dFcPrg= 2022-09-09T22:59:44.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Ygo5jWtxjEplxkZToPMA/dFcPrg=', 'Content-Length': '0'} 2022-09-09T22:59:44.825 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069ebe96956a3e83d-00631bc560-1403-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.826 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:44.827 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:44.827 DEBUG:boto:Method: HEAD 2022-09-09T22:59:44.828 DEBUG:boto:Path: /jkcgbj-44/ 2022-09-09T22:59:44.828 DEBUG:boto:Data: 2022-09-09T22:59:44.829 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.829 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:44.830 DEBUG:boto:Port: 8000 2022-09-09T22:59:44.830 DEBUG:boto:Params: {} 2022-09-09T22:59:44.831 DEBUG:boto:Token: None 2022-09-09T22:59:44.832 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-44/ 2022-09-09T22:59:44.832 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Ygo5jWtxjEplxkZToPMA/dFcPrg= 2022-09-09T22:59:44.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Ygo5jWtxjEplxkZToPMA/dFcPrg=', 'Content-Length': '0'} 2022-09-09T22:59:44.842 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b02c7a592f79aa9f-00631bc560-13fd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.843 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:44.844 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:44.845 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:44.846 DEBUG:boto:path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:44.846 DEBUG:boto:auth_path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:44.848 DEBUG:boto:Method: GET 2022-09-09T22:59:44.849 DEBUG:boto:Path: /jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:44.850 DEBUG:boto:Data: 2022-09-09T22:59:44.852 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.852 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:44.853 DEBUG:boto:Port: 8001 2022-09-09T22:59:44.853 DEBUG:boto:Params: {} 2022-09-09T22:59:44.854 DEBUG:boto:Token: None 2022-09-09T22:59:44.855 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-44/?versions 2022-09-09T22:59:44.855 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:RKS34XfHh0/20an+PZf9dYDYTeE= 2022-09-09T22:59:44.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:RKS34XfHh0/20an+PZf9dYDYTeE=', 'Content-Length': '0'} 2022-09-09T22:59:44.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc515e7c12e1d3d8-00631bc560-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.863 DEBUG:boto:b'jkcgbj-44999false' 2022-09-09T22:59:44.863 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:44.864 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:44.865 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:44.865 DEBUG:boto:path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:44.866 DEBUG:boto:auth_path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:44.866 DEBUG:boto:Method: GET 2022-09-09T22:59:44.867 DEBUG:boto:Path: /jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:44.867 DEBUG:boto:Data: 2022-09-09T22:59:44.868 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.868 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:44.869 DEBUG:boto:Port: 8000 2022-09-09T22:59:44.869 DEBUG:boto:Params: {} 2022-09-09T22:59:44.870 DEBUG:boto:Token: None 2022-09-09T22:59:44.871 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-44/?versions 2022-09-09T22:59:44.872 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:RKS34XfHh0/20an+PZf9dYDYTeE= 2022-09-09T22:59:44.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:RKS34XfHh0/20an+PZf9dYDYTeE=', 'Content-Length': '0'} 2022-09-09T22:59:44.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026d78ea8084fdfb3-00631bc560-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.880 DEBUG:boto:b'jkcgbj-44999false' 2022-09-09T22:59:44.881 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:44.881 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:44.882 DEBUG:boto:path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:44.883 DEBUG:boto:auth_path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:44.883 DEBUG:boto:Method: GET 2022-09-09T22:59:44.884 DEBUG:boto:Path: /jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:44.884 DEBUG:boto:Data: 2022-09-09T22:59:44.885 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.885 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:44.886 DEBUG:boto:Port: 8001 2022-09-09T22:59:44.887 DEBUG:boto:Params: {} 2022-09-09T22:59:44.888 DEBUG:boto:Token: None 2022-09-09T22:59:44.889 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-44/?versions 2022-09-09T22:59:44.889 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:RKS34XfHh0/20an+PZf9dYDYTeE= 2022-09-09T22:59:44.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:RKS34XfHh0/20an+PZf9dYDYTeE=', 'Content-Length': '0'} 2022-09-09T22:59:44.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068d425a4dc97c2d1-00631bc560-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.938 DEBUG:boto:b'jkcgbj-44999false' 2022-09-09T22:59:44.940 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:44.940 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:44.941 DEBUG:boto:path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:44.942 DEBUG:boto:auth_path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:44.942 DEBUG:boto:Method: GET 2022-09-09T22:59:44.943 DEBUG:boto:Path: /jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:44.943 DEBUG:boto:Data: 2022-09-09T22:59:44.944 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.944 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:44.945 DEBUG:boto:Port: 8000 2022-09-09T22:59:44.945 DEBUG:boto:Params: {} 2022-09-09T22:59:44.946 DEBUG:boto:Token: None 2022-09-09T22:59:44.947 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-44/?versions 2022-09-09T22:59:44.947 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:RKS34XfHh0/20an+PZf9dYDYTeE= 2022-09-09T22:59:44.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:RKS34XfHh0/20an+PZf9dYDYTeE=', 'Content-Length': '0'} 2022-09-09T22:59:44.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b11b8de4cb95f60d-00631bc560-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.955 DEBUG:boto:b'jkcgbj-44999false' 2022-09-09T22:59:44.962 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-44 zones={test-zone3, test-zone1} 2022-09-09T22:59:44.963 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-44 zones={test-zone1, test-zone2} 2022-09-09T22:59:44.963 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:44.964 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:44.964 DEBUG:boto:Method: HEAD 2022-09-09T22:59:44.965 DEBUG:boto:Path: /jkcgbj-44/ 2022-09-09T22:59:44.966 DEBUG:boto:Data: 2022-09-09T22:59:44.966 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.967 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:44.967 DEBUG:boto:Port: 8000 2022-09-09T22:59:44.968 DEBUG:boto:Params: {} 2022-09-09T22:59:44.968 DEBUG:boto:Token: None 2022-09-09T22:59:44.969 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-44/ 2022-09-09T22:59:44.969 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Ygo5jWtxjEplxkZToPMA/dFcPrg= 2022-09-09T22:59:44.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Ygo5jWtxjEplxkZToPMA/dFcPrg=', 'Content-Length': '0'} 2022-09-09T22:59:44.975 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e01b47a8bda8394-00631bc560-13fd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.976 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:44.976 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:44.977 DEBUG:boto:Method: HEAD 2022-09-09T22:59:44.977 DEBUG:boto:Path: /jkcgbj-44/ 2022-09-09T22:59:44.978 DEBUG:boto:Data: 2022-09-09T22:59:44.978 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.979 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:44.979 DEBUG:boto:Port: 8000 2022-09-09T22:59:44.980 DEBUG:boto:Params: {} 2022-09-09T22:59:44.980 DEBUG:boto:Token: None 2022-09-09T22:59:44.981 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-44/ 2022-09-09T22:59:44.982 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Ygo5jWtxjEplxkZToPMA/dFcPrg= 2022-09-09T22:59:44.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Ygo5jWtxjEplxkZToPMA/dFcPrg=', 'Content-Length': '0'} 2022-09-09T22:59:44.988 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000090f55daf75e2d3f-00631bc560-1549-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:44.988 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:44.989 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:44.989 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:44.990 DEBUG:boto:path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:44.990 DEBUG:boto:auth_path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:44.991 DEBUG:boto:Method: GET 2022-09-09T22:59:44.992 DEBUG:boto:Path: /jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:44.992 DEBUG:boto:Data: 2022-09-09T22:59:44.993 DEBUG:boto:Headers: {} 2022-09-09T22:59:44.993 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:44.994 DEBUG:boto:Port: 8000 2022-09-09T22:59:44.994 DEBUG:boto:Params: {} 2022-09-09T22:59:44.995 DEBUG:boto:Token: None 2022-09-09T22:59:44.996 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:44 GMT /jkcgbj-44/?versions 2022-09-09T22:59:44.996 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:RKS34XfHh0/20an+PZf9dYDYTeE= 2022-09-09T22:59:44.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:RKS34XfHh0/20an+PZf9dYDYTeE=', 'Content-Length': '0'} 2022-09-09T22:59:45.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f71558255854398-00631bc560-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.050 DEBUG:boto:b'jkcgbj-44999false' 2022-09-09T22:59:45.051 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:45.052 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:45.052 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:45.053 DEBUG:boto:path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.053 DEBUG:boto:auth_path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.054 DEBUG:boto:Method: GET 2022-09-09T22:59:45.055 DEBUG:boto:Path: /jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.056 DEBUG:boto:Data: 2022-09-09T22:59:45.056 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.057 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:45.058 DEBUG:boto:Port: 8000 2022-09-09T22:59:45.059 DEBUG:boto:Params: {} 2022-09-09T22:59:45.060 DEBUG:boto:Token: None 2022-09-09T22:59:45.060 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-44/?versions 2022-09-09T22:59:45.061 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:f0X/P8HR5Lkj/kzvX4gbUnkmi+4= 2022-09-09T22:59:45.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:f0X/P8HR5Lkj/kzvX4gbUnkmi+4=', 'Content-Length': '0'} 2022-09-09T22:59:45.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e59b3a2a4da60bca-00631bc561-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.068 DEBUG:boto:b'jkcgbj-44999false' 2022-09-09T22:59:45.069 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:45.070 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:45.070 DEBUG:boto:path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.071 DEBUG:boto:auth_path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.071 DEBUG:boto:Method: GET 2022-09-09T22:59:45.072 DEBUG:boto:Path: /jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.072 DEBUG:boto:Data: 2022-09-09T22:59:45.073 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.074 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:45.074 DEBUG:boto:Port: 8000 2022-09-09T22:59:45.075 DEBUG:boto:Params: {} 2022-09-09T22:59:45.076 DEBUG:boto:Token: None 2022-09-09T22:59:45.076 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-44/?versions 2022-09-09T22:59:45.077 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:f0X/P8HR5Lkj/kzvX4gbUnkmi+4= 2022-09-09T22:59:45.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:f0X/P8HR5Lkj/kzvX4gbUnkmi+4=', 'Content-Length': '0'} 2022-09-09T22:59:45.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cc5094b24b7b814-00631bc561-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.083 DEBUG:boto:b'jkcgbj-44999false' 2022-09-09T22:59:45.084 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:45.085 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:45.085 DEBUG:boto:path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.086 DEBUG:boto:auth_path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.086 DEBUG:boto:Method: GET 2022-09-09T22:59:45.087 DEBUG:boto:Path: /jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.087 DEBUG:boto:Data: 2022-09-09T22:59:45.088 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.088 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:45.089 DEBUG:boto:Port: 8000 2022-09-09T22:59:45.089 DEBUG:boto:Params: {} 2022-09-09T22:59:45.090 DEBUG:boto:Token: None 2022-09-09T22:59:45.091 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-44/?versions 2022-09-09T22:59:45.092 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:f0X/P8HR5Lkj/kzvX4gbUnkmi+4= 2022-09-09T22:59:45.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:f0X/P8HR5Lkj/kzvX4gbUnkmi+4=', 'Content-Length': '0'} 2022-09-09T22:59:45.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000810b39c18bcf0657-00631bc561-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.099 DEBUG:boto:b'jkcgbj-44999false' 2022-09-09T22:59:45.100 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-44 zones={test-zone1, test-zone2} 2022-09-09T22:59:45.101 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-44 zones={test-zone3, test-zone2} 2022-09-09T22:59:45.101 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:45.101 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:45.102 DEBUG:boto:Method: HEAD 2022-09-09T22:59:45.103 DEBUG:boto:Path: /jkcgbj-44/ 2022-09-09T22:59:45.104 DEBUG:boto:Data: 2022-09-09T22:59:45.105 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.105 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:45.106 DEBUG:boto:Port: 8001 2022-09-09T22:59:45.107 DEBUG:boto:Params: {} 2022-09-09T22:59:45.108 DEBUG:boto:Token: None 2022-09-09T22:59:45.109 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-44/ 2022-09-09T22:59:45.110 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:unrmQSO+C+FaBgdPKpEJjRwWIr0= 2022-09-09T22:59:45.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:unrmQSO+C+FaBgdPKpEJjRwWIr0=', 'Content-Length': '0'} 2022-09-09T22:59:45.115 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000023c0017f164513ad-00631bc561-1403-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.116 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:45.116 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:45.117 DEBUG:boto:Method: HEAD 2022-09-09T22:59:45.118 DEBUG:boto:Path: /jkcgbj-44/ 2022-09-09T22:59:45.118 DEBUG:boto:Data: 2022-09-09T22:59:45.119 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.119 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:45.120 DEBUG:boto:Port: 8000 2022-09-09T22:59:45.120 DEBUG:boto:Params: {} 2022-09-09T22:59:45.121 DEBUG:boto:Token: None 2022-09-09T22:59:45.121 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-44/ 2022-09-09T22:59:45.122 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:unrmQSO+C+FaBgdPKpEJjRwWIr0= 2022-09-09T22:59:45.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:unrmQSO+C+FaBgdPKpEJjRwWIr0=', 'Content-Length': '0'} 2022-09-09T22:59:45.127 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f9447ad818cefe1-00631bc561-1549-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.127 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:45.128 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:45.128 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:45.129 DEBUG:boto:path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.129 DEBUG:boto:auth_path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.130 DEBUG:boto:Method: GET 2022-09-09T22:59:45.130 DEBUG:boto:Path: /jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.131 DEBUG:boto:Data: 2022-09-09T22:59:45.131 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.132 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:45.132 DEBUG:boto:Port: 8001 2022-09-09T22:59:45.133 DEBUG:boto:Params: {} 2022-09-09T22:59:45.133 DEBUG:boto:Token: None 2022-09-09T22:59:45.134 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-44/?versions 2022-09-09T22:59:45.135 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:f0X/P8HR5Lkj/kzvX4gbUnkmi+4= 2022-09-09T22:59:45.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:f0X/P8HR5Lkj/kzvX4gbUnkmi+4=', 'Content-Length': '0'} 2022-09-09T22:59:45.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043933eefe1c60ba3-00631bc561-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.144 DEBUG:boto:b'jkcgbj-44999false' 2022-09-09T22:59:45.145 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:45.146 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:45.147 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:45.149 DEBUG:boto:path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.149 DEBUG:boto:auth_path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.150 DEBUG:boto:Method: GET 2022-09-09T22:59:45.151 DEBUG:boto:Path: /jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.152 DEBUG:boto:Data: 2022-09-09T22:59:45.153 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.154 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:45.155 DEBUG:boto:Port: 8000 2022-09-09T22:59:45.156 DEBUG:boto:Params: {} 2022-09-09T22:59:45.157 DEBUG:boto:Token: None 2022-09-09T22:59:45.158 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-44/?versions 2022-09-09T22:59:45.159 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:f0X/P8HR5Lkj/kzvX4gbUnkmi+4= 2022-09-09T22:59:45.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:f0X/P8HR5Lkj/kzvX4gbUnkmi+4=', 'Content-Length': '0'} 2022-09-09T22:59:45.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009dd01e459da8ffa7-00631bc561-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.207 DEBUG:boto:b'jkcgbj-44999false' 2022-09-09T22:59:45.209 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:45.209 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:45.210 DEBUG:boto:path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.210 DEBUG:boto:auth_path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.211 DEBUG:boto:Method: GET 2022-09-09T22:59:45.212 DEBUG:boto:Path: /jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.212 DEBUG:boto:Data: 2022-09-09T22:59:45.213 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.214 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:45.214 DEBUG:boto:Port: 8001 2022-09-09T22:59:45.215 DEBUG:boto:Params: {} 2022-09-09T22:59:45.215 DEBUG:boto:Token: None 2022-09-09T22:59:45.216 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-44/?versions 2022-09-09T22:59:45.217 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:f0X/P8HR5Lkj/kzvX4gbUnkmi+4= 2022-09-09T22:59:45.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:f0X/P8HR5Lkj/kzvX4gbUnkmi+4=', 'Content-Length': '0'} 2022-09-09T22:59:45.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000678113c0f86d2b33-00631bc561-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.224 DEBUG:boto:b'jkcgbj-44999false' 2022-09-09T22:59:45.225 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:45.226 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:45.226 DEBUG:boto:path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.227 DEBUG:boto:auth_path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.228 DEBUG:boto:Method: GET 2022-09-09T22:59:45.228 DEBUG:boto:Path: /jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.229 DEBUG:boto:Data: 2022-09-09T22:59:45.229 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.230 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:45.230 DEBUG:boto:Port: 8000 2022-09-09T22:59:45.231 DEBUG:boto:Params: {} 2022-09-09T22:59:45.232 DEBUG:boto:Token: None 2022-09-09T22:59:45.232 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-44/?versions 2022-09-09T22:59:45.233 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:f0X/P8HR5Lkj/kzvX4gbUnkmi+4= 2022-09-09T22:59:45.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:f0X/P8HR5Lkj/kzvX4gbUnkmi+4=', 'Content-Length': '0'} 2022-09-09T22:59:45.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f780f7c28a2a86c-00631bc561-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.240 DEBUG:boto:b'jkcgbj-44999false' 2022-09-09T22:59:45.241 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-44 zones={test-zone3, test-zone2} 2022-09-09T22:59:45.242 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-44 zones={test-zone1, test-zone3} 2022-09-09T22:59:45.243 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:45.244 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:45.245 DEBUG:boto:Method: HEAD 2022-09-09T22:59:45.246 DEBUG:boto:Path: /jkcgbj-44/ 2022-09-09T22:59:45.247 DEBUG:boto:Data: 2022-09-09T22:59:45.248 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.249 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:45.249 DEBUG:boto:Port: 8000 2022-09-09T22:59:45.250 DEBUG:boto:Params: {} 2022-09-09T22:59:45.251 DEBUG:boto:Token: None 2022-09-09T22:59:45.252 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-44/ 2022-09-09T22:59:45.253 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:unrmQSO+C+FaBgdPKpEJjRwWIr0= 2022-09-09T22:59:45.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:unrmQSO+C+FaBgdPKpEJjRwWIr0=', 'Content-Length': '0'} 2022-09-09T22:59:45.259 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e28027e38d78acba-00631bc561-13fd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.260 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:45.261 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:45.262 DEBUG:boto:Method: HEAD 2022-09-09T22:59:45.263 DEBUG:boto:Path: /jkcgbj-44/ 2022-09-09T22:59:45.264 DEBUG:boto:Data: 2022-09-09T22:59:45.265 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.266 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:45.266 DEBUG:boto:Port: 8001 2022-09-09T22:59:45.267 DEBUG:boto:Params: {} 2022-09-09T22:59:45.269 DEBUG:boto:Token: None 2022-09-09T22:59:45.270 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-44/ 2022-09-09T22:59:45.270 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:unrmQSO+C+FaBgdPKpEJjRwWIr0= 2022-09-09T22:59:45.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:unrmQSO+C+FaBgdPKpEJjRwWIr0=', 'Content-Length': '0'} 2022-09-09T22:59:45.276 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009124b348bc5c8e86-00631bc561-1403-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.277 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:45.278 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:45.278 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:45.279 DEBUG:boto:path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.279 DEBUG:boto:auth_path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.280 DEBUG:boto:Method: GET 2022-09-09T22:59:45.280 DEBUG:boto:Path: /jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.281 DEBUG:boto:Data: 2022-09-09T22:59:45.281 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.282 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:45.282 DEBUG:boto:Port: 8000 2022-09-09T22:59:45.283 DEBUG:boto:Params: {} 2022-09-09T22:59:45.284 DEBUG:boto:Token: None 2022-09-09T22:59:45.285 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-44/?versions 2022-09-09T22:59:45.285 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:f0X/P8HR5Lkj/kzvX4gbUnkmi+4= 2022-09-09T22:59:45.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:f0X/P8HR5Lkj/kzvX4gbUnkmi+4=', 'Content-Length': '0'} 2022-09-09T22:59:45.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000761589aaa7ae0dcc-00631bc561-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.293 DEBUG:boto:b'jkcgbj-44999false' 2022-09-09T22:59:45.294 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:45.294 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:45.295 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:45.295 DEBUG:boto:path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.296 DEBUG:boto:auth_path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.296 DEBUG:boto:Method: GET 2022-09-09T22:59:45.297 DEBUG:boto:Path: /jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.298 DEBUG:boto:Data: 2022-09-09T22:59:45.299 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.300 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:45.301 DEBUG:boto:Port: 8001 2022-09-09T22:59:45.302 DEBUG:boto:Params: {} 2022-09-09T22:59:45.302 DEBUG:boto:Token: None 2022-09-09T22:59:45.303 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-44/?versions 2022-09-09T22:59:45.303 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:f0X/P8HR5Lkj/kzvX4gbUnkmi+4= 2022-09-09T22:59:45.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:f0X/P8HR5Lkj/kzvX4gbUnkmi+4=', 'Content-Length': '0'} 2022-09-09T22:59:45.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4c70e01fdb8cf61-00631bc561-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.311 DEBUG:boto:b'jkcgbj-44999false' 2022-09-09T22:59:45.312 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:45.313 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:45.313 DEBUG:boto:path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.314 DEBUG:boto:auth_path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.314 DEBUG:boto:Method: GET 2022-09-09T22:59:45.315 DEBUG:boto:Path: /jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.316 DEBUG:boto:Data: 2022-09-09T22:59:45.316 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.317 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:45.317 DEBUG:boto:Port: 8000 2022-09-09T22:59:45.318 DEBUG:boto:Params: {} 2022-09-09T22:59:45.318 DEBUG:boto:Token: None 2022-09-09T22:59:45.319 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-44/?versions 2022-09-09T22:59:45.319 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:f0X/P8HR5Lkj/kzvX4gbUnkmi+4= 2022-09-09T22:59:45.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:f0X/P8HR5Lkj/kzvX4gbUnkmi+4=', 'Content-Length': '0'} 2022-09-09T22:59:45.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002209950e3e0be641-00631bc561-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.366 DEBUG:boto:b'jkcgbj-44999false' 2022-09-09T22:59:45.367 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:45.368 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:45.368 DEBUG:boto:path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.369 DEBUG:boto:auth_path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.370 DEBUG:boto:Method: GET 2022-09-09T22:59:45.370 DEBUG:boto:Path: /jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.371 DEBUG:boto:Data: 2022-09-09T22:59:45.371 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.372 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:45.372 DEBUG:boto:Port: 8001 2022-09-09T22:59:45.373 DEBUG:boto:Params: {} 2022-09-09T22:59:45.374 DEBUG:boto:Token: None 2022-09-09T22:59:45.375 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-44/?versions 2022-09-09T22:59:45.376 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:f0X/P8HR5Lkj/kzvX4gbUnkmi+4= 2022-09-09T22:59:45.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:f0X/P8HR5Lkj/kzvX4gbUnkmi+4=', 'Content-Length': '0'} 2022-09-09T22:59:45.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015365cbef32ce39e-00631bc561-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.383 DEBUG:boto:b'jkcgbj-44999false' 2022-09-09T22:59:45.383 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-44 zones={test-zone1, test-zone3} 2022-09-09T22:59:45.384 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-44 zones={test-zone2, test-zone3} 2022-09-09T22:59:45.384 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:45.385 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:45.385 DEBUG:boto:Method: HEAD 2022-09-09T22:59:45.386 DEBUG:boto:Path: /jkcgbj-44/ 2022-09-09T22:59:45.387 DEBUG:boto:Data: 2022-09-09T22:59:45.387 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.388 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:45.389 DEBUG:boto:Port: 8000 2022-09-09T22:59:45.390 DEBUG:boto:Params: {} 2022-09-09T22:59:45.391 DEBUG:boto:Token: None 2022-09-09T22:59:45.392 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-44/ 2022-09-09T22:59:45.392 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:unrmQSO+C+FaBgdPKpEJjRwWIr0= 2022-09-09T22:59:45.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:unrmQSO+C+FaBgdPKpEJjRwWIr0=', 'Content-Length': '0'} 2022-09-09T22:59:45.398 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000061f0875164c73d1d-00631bc561-1549-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.399 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:45.400 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:45.401 DEBUG:boto:Method: HEAD 2022-09-09T22:59:45.402 DEBUG:boto:Path: /jkcgbj-44/ 2022-09-09T22:59:45.402 DEBUG:boto:Data: 2022-09-09T22:59:45.405 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.406 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:45.407 DEBUG:boto:Port: 8001 2022-09-09T22:59:45.408 DEBUG:boto:Params: {} 2022-09-09T22:59:45.409 DEBUG:boto:Token: None 2022-09-09T22:59:45.410 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-44/ 2022-09-09T22:59:45.411 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:unrmQSO+C+FaBgdPKpEJjRwWIr0= 2022-09-09T22:59:45.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:unrmQSO+C+FaBgdPKpEJjRwWIr0=', 'Content-Length': '0'} 2022-09-09T22:59:45.418 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ef52843667a9c9a-00631bc561-1403-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.419 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:45.419 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:45.420 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:45.420 DEBUG:boto:path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.421 DEBUG:boto:auth_path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.421 DEBUG:boto:Method: GET 2022-09-09T22:59:45.422 DEBUG:boto:Path: /jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.423 DEBUG:boto:Data: 2022-09-09T22:59:45.423 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.424 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:45.424 DEBUG:boto:Port: 8000 2022-09-09T22:59:45.425 DEBUG:boto:Params: {} 2022-09-09T22:59:45.426 DEBUG:boto:Token: None 2022-09-09T22:59:45.427 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-44/?versions 2022-09-09T22:59:45.428 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:f0X/P8HR5Lkj/kzvX4gbUnkmi+4= 2022-09-09T22:59:45.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:f0X/P8HR5Lkj/kzvX4gbUnkmi+4=', 'Content-Length': '0'} 2022-09-09T22:59:45.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003679b864088b3414-00631bc561-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.436 DEBUG:boto:b'jkcgbj-44999false' 2022-09-09T22:59:45.437 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:45.438 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:45.438 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:45.439 DEBUG:boto:path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.439 DEBUG:boto:auth_path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.440 DEBUG:boto:Method: GET 2022-09-09T22:59:45.440 DEBUG:boto:Path: /jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.441 DEBUG:boto:Data: 2022-09-09T22:59:45.442 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.443 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:45.443 DEBUG:boto:Port: 8001 2022-09-09T22:59:45.444 DEBUG:boto:Params: {} 2022-09-09T22:59:45.446 DEBUG:boto:Token: None 2022-09-09T22:59:45.447 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-44/?versions 2022-09-09T22:59:45.448 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:f0X/P8HR5Lkj/kzvX4gbUnkmi+4= 2022-09-09T22:59:45.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:f0X/P8HR5Lkj/kzvX4gbUnkmi+4=', 'Content-Length': '0'} 2022-09-09T22:59:45.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2b5b1a5826fbe6d-00631bc561-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.498 DEBUG:boto:b'jkcgbj-44999false' 2022-09-09T22:59:45.500 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:45.501 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:45.502 DEBUG:boto:path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.503 DEBUG:boto:auth_path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.504 DEBUG:boto:Method: GET 2022-09-09T22:59:45.505 DEBUG:boto:Path: /jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.505 DEBUG:boto:Data: 2022-09-09T22:59:45.506 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.506 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:45.507 DEBUG:boto:Port: 8000 2022-09-09T22:59:45.509 DEBUG:boto:Params: {} 2022-09-09T22:59:45.510 DEBUG:boto:Token: None 2022-09-09T22:59:45.511 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-44/?versions 2022-09-09T22:59:45.511 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:f0X/P8HR5Lkj/kzvX4gbUnkmi+4= 2022-09-09T22:59:45.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:f0X/P8HR5Lkj/kzvX4gbUnkmi+4=', 'Content-Length': '0'} 2022-09-09T22:59:45.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015dc97ecb3828995-00631bc561-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.519 DEBUG:boto:b'jkcgbj-44999false' 2022-09-09T22:59:45.520 DEBUG:boto:path=/jkcgbj-44/ 2022-09-09T22:59:45.521 DEBUG:boto:auth_path=/jkcgbj-44/ 2022-09-09T22:59:45.521 DEBUG:boto:path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.522 DEBUG:boto:auth_path=/jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.523 DEBUG:boto:Method: GET 2022-09-09T22:59:45.524 DEBUG:boto:Path: /jkcgbj-44/?versions&max-keys=999 2022-09-09T22:59:45.524 DEBUG:boto:Data: 2022-09-09T22:59:45.525 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.526 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:45.527 DEBUG:boto:Port: 8001 2022-09-09T22:59:45.528 DEBUG:boto:Params: {} 2022-09-09T22:59:45.528 DEBUG:boto:Token: None 2022-09-09T22:59:45.529 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-44/?versions 2022-09-09T22:59:45.530 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:f0X/P8HR5Lkj/kzvX4gbUnkmi+4= 2022-09-09T22:59:45.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:f0X/P8HR5Lkj/kzvX4gbUnkmi+4=', 'Content-Length': '0'} 2022-09-09T22:59:45.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c1c204084a3b938-00631bc561-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.539 DEBUG:boto:b'jkcgbj-44999false' 2022-09-09T22:59:45.541 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-44 zones={test-zone2, test-zone3} 2022-09-09T22:59:45.542 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-45 zones={test-zone2, test-zone1} 2022-09-09T22:59:45.543 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:45.544 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:45.545 DEBUG:boto:Method: HEAD 2022-09-09T22:59:45.545 DEBUG:boto:Path: /jkcgbj-45/ 2022-09-09T22:59:45.546 DEBUG:boto:Data: 2022-09-09T22:59:45.547 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.548 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:45.549 DEBUG:boto:Port: 8000 2022-09-09T22:59:45.550 DEBUG:boto:Params: {} 2022-09-09T22:59:45.551 DEBUG:boto:Token: None 2022-09-09T22:59:45.552 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-45/ 2022-09-09T22:59:45.553 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R6czooxbQFUQBAkcQOJxeqWIhX4= 2022-09-09T22:59:45.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R6czooxbQFUQBAkcQOJxeqWIhX4=', 'Content-Length': '0'} 2022-09-09T22:59:45.560 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007906c2867180d2c3-00631bc561-1549-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.561 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:45.562 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:45.562 DEBUG:boto:Method: HEAD 2022-09-09T22:59:45.563 DEBUG:boto:Path: /jkcgbj-45/ 2022-09-09T22:59:45.564 DEBUG:boto:Data: 2022-09-09T22:59:45.564 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.565 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:45.565 DEBUG:boto:Port: 8000 2022-09-09T22:59:45.566 DEBUG:boto:Params: {} 2022-09-09T22:59:45.566 DEBUG:boto:Token: None 2022-09-09T22:59:45.567 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-45/ 2022-09-09T22:59:45.567 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R6czooxbQFUQBAkcQOJxeqWIhX4= 2022-09-09T22:59:45.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R6czooxbQFUQBAkcQOJxeqWIhX4=', 'Content-Length': '0'} 2022-09-09T22:59:45.575 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ffa1f051c2ddf13-00631bc561-13fd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.576 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:45.577 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:45.578 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:45.579 DEBUG:boto:path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.580 DEBUG:boto:auth_path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.581 DEBUG:boto:Method: GET 2022-09-09T22:59:45.582 DEBUG:boto:Path: /jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.583 DEBUG:boto:Data: 2022-09-09T22:59:45.583 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.584 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:45.585 DEBUG:boto:Port: 8000 2022-09-09T22:59:45.585 DEBUG:boto:Params: {} 2022-09-09T22:59:45.586 DEBUG:boto:Token: None 2022-09-09T22:59:45.586 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-45/?versions 2022-09-09T22:59:45.587 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OG9yXzqaDJuUxSGO7Kk81s/3MPk= 2022-09-09T22:59:45.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OG9yXzqaDJuUxSGO7Kk81s/3MPk=', 'Content-Length': '0'} 2022-09-09T22:59:45.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000356db897972d9c64-00631bc561-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.634 DEBUG:boto:b'jkcgbj-45999false' 2022-09-09T22:59:45.636 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:45.637 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:45.638 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:45.639 DEBUG:boto:path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.639 DEBUG:boto:auth_path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.640 DEBUG:boto:Method: GET 2022-09-09T22:59:45.640 DEBUG:boto:Path: /jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.641 DEBUG:boto:Data: 2022-09-09T22:59:45.641 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.642 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:45.642 DEBUG:boto:Port: 8000 2022-09-09T22:59:45.643 DEBUG:boto:Params: {} 2022-09-09T22:59:45.644 DEBUG:boto:Token: None 2022-09-09T22:59:45.644 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-45/?versions 2022-09-09T22:59:45.645 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OG9yXzqaDJuUxSGO7Kk81s/3MPk= 2022-09-09T22:59:45.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OG9yXzqaDJuUxSGO7Kk81s/3MPk=', 'Content-Length': '0'} 2022-09-09T22:59:45.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fb937d0f31a779a-00631bc561-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.654 DEBUG:boto:b'jkcgbj-45999false' 2022-09-09T22:59:45.656 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:45.657 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:45.657 DEBUG:boto:path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.658 DEBUG:boto:auth_path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.658 DEBUG:boto:Method: GET 2022-09-09T22:59:45.659 DEBUG:boto:Path: /jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.659 DEBUG:boto:Data: 2022-09-09T22:59:45.660 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.660 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:45.661 DEBUG:boto:Port: 8000 2022-09-09T22:59:45.661 DEBUG:boto:Params: {} 2022-09-09T22:59:45.662 DEBUG:boto:Token: None 2022-09-09T22:59:45.663 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-45/?versions 2022-09-09T22:59:45.664 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OG9yXzqaDJuUxSGO7Kk81s/3MPk= 2022-09-09T22:59:45.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OG9yXzqaDJuUxSGO7Kk81s/3MPk=', 'Content-Length': '0'} 2022-09-09T22:59:45.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e3994d453ce75a3-00631bc561-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.671 DEBUG:boto:b'jkcgbj-45999false' 2022-09-09T22:59:45.671 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:45.672 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:45.672 DEBUG:boto:path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.673 DEBUG:boto:auth_path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.673 DEBUG:boto:Method: GET 2022-09-09T22:59:45.674 DEBUG:boto:Path: /jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.675 DEBUG:boto:Data: 2022-09-09T22:59:45.675 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.676 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:45.676 DEBUG:boto:Port: 8000 2022-09-09T22:59:45.677 DEBUG:boto:Params: {} 2022-09-09T22:59:45.677 DEBUG:boto:Token: None 2022-09-09T22:59:45.678 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-45/?versions 2022-09-09T22:59:45.679 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OG9yXzqaDJuUxSGO7Kk81s/3MPk= 2022-09-09T22:59:45.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OG9yXzqaDJuUxSGO7Kk81s/3MPk=', 'Content-Length': '0'} 2022-09-09T22:59:45.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccd73b9659b89f9b-00631bc561-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.687 DEBUG:boto:b'jkcgbj-45999false' 2022-09-09T22:59:45.688 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-45 zones={test-zone2, test-zone1} 2022-09-09T22:59:45.689 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-45 zones={test-zone3, test-zone1} 2022-09-09T22:59:45.690 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:45.690 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:45.691 DEBUG:boto:Method: HEAD 2022-09-09T22:59:45.692 DEBUG:boto:Path: /jkcgbj-45/ 2022-09-09T22:59:45.692 DEBUG:boto:Data: 2022-09-09T22:59:45.693 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.693 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:45.694 DEBUG:boto:Port: 8001 2022-09-09T22:59:45.695 DEBUG:boto:Params: {} 2022-09-09T22:59:45.695 DEBUG:boto:Token: None 2022-09-09T22:59:45.696 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-45/ 2022-09-09T22:59:45.697 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R6czooxbQFUQBAkcQOJxeqWIhX4= 2022-09-09T22:59:45.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R6czooxbQFUQBAkcQOJxeqWIhX4=', 'Content-Length': '0'} 2022-09-09T22:59:45.701 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c0e2c17bcab207e-00631bc561-1403-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.702 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:45.703 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:45.703 DEBUG:boto:Method: HEAD 2022-09-09T22:59:45.704 DEBUG:boto:Path: /jkcgbj-45/ 2022-09-09T22:59:45.704 DEBUG:boto:Data: 2022-09-09T22:59:45.705 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.705 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:45.706 DEBUG:boto:Port: 8000 2022-09-09T22:59:45.706 DEBUG:boto:Params: {} 2022-09-09T22:59:45.707 DEBUG:boto:Token: None 2022-09-09T22:59:45.708 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-45/ 2022-09-09T22:59:45.708 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R6czooxbQFUQBAkcQOJxeqWIhX4= 2022-09-09T22:59:45.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R6czooxbQFUQBAkcQOJxeqWIhX4=', 'Content-Length': '0'} 2022-09-09T22:59:45.714 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000747724b1a2392f83-00631bc561-13fd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.715 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:45.716 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:45.717 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:45.718 DEBUG:boto:path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.719 DEBUG:boto:auth_path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.719 DEBUG:boto:Method: GET 2022-09-09T22:59:45.720 DEBUG:boto:Path: /jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.721 DEBUG:boto:Data: 2022-09-09T22:59:45.722 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.723 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:45.723 DEBUG:boto:Port: 8001 2022-09-09T22:59:45.724 DEBUG:boto:Params: {} 2022-09-09T22:59:45.724 DEBUG:boto:Token: None 2022-09-09T22:59:45.725 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-45/?versions 2022-09-09T22:59:45.726 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OG9yXzqaDJuUxSGO7Kk81s/3MPk= 2022-09-09T22:59:45.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OG9yXzqaDJuUxSGO7Kk81s/3MPk=', 'Content-Length': '0'} 2022-09-09T22:59:45.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc7d026cb463ed29-00631bc561-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.734 DEBUG:boto:b'jkcgbj-45999false' 2022-09-09T22:59:45.735 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:45.735 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:45.736 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:45.736 DEBUG:boto:path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.737 DEBUG:boto:auth_path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.737 DEBUG:boto:Method: GET 2022-09-09T22:59:45.738 DEBUG:boto:Path: /jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.739 DEBUG:boto:Data: 2022-09-09T22:59:45.739 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.740 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:45.740 DEBUG:boto:Port: 8000 2022-09-09T22:59:45.741 DEBUG:boto:Params: {} 2022-09-09T22:59:45.741 DEBUG:boto:Token: None 2022-09-09T22:59:45.742 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-45/?versions 2022-09-09T22:59:45.742 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OG9yXzqaDJuUxSGO7Kk81s/3MPk= 2022-09-09T22:59:45.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OG9yXzqaDJuUxSGO7Kk81s/3MPk=', 'Content-Length': '0'} 2022-09-09T22:59:45.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004facde2e7dab5d43-00631bc561-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.790 DEBUG:boto:b'jkcgbj-45999false' 2022-09-09T22:59:45.791 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:45.792 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:45.792 DEBUG:boto:path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.793 DEBUG:boto:auth_path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.793 DEBUG:boto:Method: GET 2022-09-09T22:59:45.794 DEBUG:boto:Path: /jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.794 DEBUG:boto:Data: 2022-09-09T22:59:45.795 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.796 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:45.796 DEBUG:boto:Port: 8001 2022-09-09T22:59:45.797 DEBUG:boto:Params: {} 2022-09-09T22:59:45.797 DEBUG:boto:Token: None 2022-09-09T22:59:45.798 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-45/?versions 2022-09-09T22:59:45.799 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OG9yXzqaDJuUxSGO7Kk81s/3MPk= 2022-09-09T22:59:45.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OG9yXzqaDJuUxSGO7Kk81s/3MPk=', 'Content-Length': '0'} 2022-09-09T22:59:45.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb4780add9879d4c-00631bc561-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.808 DEBUG:boto:b'jkcgbj-45999false' 2022-09-09T22:59:45.809 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:45.810 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:45.811 DEBUG:boto:path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.812 DEBUG:boto:auth_path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.813 DEBUG:boto:Method: GET 2022-09-09T22:59:45.814 DEBUG:boto:Path: /jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.815 DEBUG:boto:Data: 2022-09-09T22:59:45.816 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.817 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:45.818 DEBUG:boto:Port: 8000 2022-09-09T22:59:45.819 DEBUG:boto:Params: {} 2022-09-09T22:59:45.819 DEBUG:boto:Token: None 2022-09-09T22:59:45.820 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-45/?versions 2022-09-09T22:59:45.821 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OG9yXzqaDJuUxSGO7Kk81s/3MPk= 2022-09-09T22:59:45.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OG9yXzqaDJuUxSGO7Kk81s/3MPk=', 'Content-Length': '0'} 2022-09-09T22:59:45.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f89fefd135f75838-00631bc561-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.828 DEBUG:boto:b'jkcgbj-45999false' 2022-09-09T22:59:45.829 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-45 zones={test-zone3, test-zone1} 2022-09-09T22:59:45.830 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-45 zones={test-zone1, test-zone2} 2022-09-09T22:59:45.830 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:45.831 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:45.832 DEBUG:boto:Method: HEAD 2022-09-09T22:59:45.833 DEBUG:boto:Path: /jkcgbj-45/ 2022-09-09T22:59:45.834 DEBUG:boto:Data: 2022-09-09T22:59:45.834 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.835 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:45.836 DEBUG:boto:Port: 8000 2022-09-09T22:59:45.836 DEBUG:boto:Params: {} 2022-09-09T22:59:45.837 DEBUG:boto:Token: None 2022-09-09T22:59:45.838 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-45/ 2022-09-09T22:59:45.838 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R6czooxbQFUQBAkcQOJxeqWIhX4= 2022-09-09T22:59:45.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R6czooxbQFUQBAkcQOJxeqWIhX4=', 'Content-Length': '0'} 2022-09-09T22:59:45.844 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c9b022afeead763e-00631bc561-13fd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.845 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:45.846 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:45.846 DEBUG:boto:Method: HEAD 2022-09-09T22:59:45.847 DEBUG:boto:Path: /jkcgbj-45/ 2022-09-09T22:59:45.847 DEBUG:boto:Data: 2022-09-09T22:59:45.848 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.848 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:45.849 DEBUG:boto:Port: 8000 2022-09-09T22:59:45.849 DEBUG:boto:Params: {} 2022-09-09T22:59:45.850 DEBUG:boto:Token: None 2022-09-09T22:59:45.851 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-45/ 2022-09-09T22:59:45.851 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R6czooxbQFUQBAkcQOJxeqWIhX4= 2022-09-09T22:59:45.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R6czooxbQFUQBAkcQOJxeqWIhX4=', 'Content-Length': '0'} 2022-09-09T22:59:45.857 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000648cbe8816305ab1-00631bc561-1549-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.858 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:45.859 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:45.859 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:45.860 DEBUG:boto:path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.860 DEBUG:boto:auth_path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.861 DEBUG:boto:Method: GET 2022-09-09T22:59:45.861 DEBUG:boto:Path: /jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.862 DEBUG:boto:Data: 2022-09-09T22:59:45.862 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.863 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:45.863 DEBUG:boto:Port: 8000 2022-09-09T22:59:45.864 DEBUG:boto:Params: {} 2022-09-09T22:59:45.864 DEBUG:boto:Token: None 2022-09-09T22:59:45.865 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-45/?versions 2022-09-09T22:59:45.866 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OG9yXzqaDJuUxSGO7Kk81s/3MPk= 2022-09-09T22:59:45.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OG9yXzqaDJuUxSGO7Kk81s/3MPk=', 'Content-Length': '0'} 2022-09-09T22:59:45.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6c382e280811957-00631bc561-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.914 DEBUG:boto:b'jkcgbj-45999false' 2022-09-09T22:59:45.915 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:45.915 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:45.916 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:45.917 DEBUG:boto:path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.917 DEBUG:boto:auth_path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.918 DEBUG:boto:Method: GET 2022-09-09T22:59:45.918 DEBUG:boto:Path: /jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.919 DEBUG:boto:Data: 2022-09-09T22:59:45.919 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.920 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:45.920 DEBUG:boto:Port: 8000 2022-09-09T22:59:45.921 DEBUG:boto:Params: {} 2022-09-09T22:59:45.921 DEBUG:boto:Token: None 2022-09-09T22:59:45.922 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-45/?versions 2022-09-09T22:59:45.922 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OG9yXzqaDJuUxSGO7Kk81s/3MPk= 2022-09-09T22:59:45.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OG9yXzqaDJuUxSGO7Kk81s/3MPk=', 'Content-Length': '0'} 2022-09-09T22:59:45.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003724e43ed176f312-00631bc561-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.931 DEBUG:boto:b'jkcgbj-45999false' 2022-09-09T22:59:45.932 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:45.933 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:45.935 DEBUG:boto:path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.935 DEBUG:boto:auth_path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.936 DEBUG:boto:Method: GET 2022-09-09T22:59:45.937 DEBUG:boto:Path: /jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.938 DEBUG:boto:Data: 2022-09-09T22:59:45.938 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.939 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:45.939 DEBUG:boto:Port: 8000 2022-09-09T22:59:45.940 DEBUG:boto:Params: {} 2022-09-09T22:59:45.940 DEBUG:boto:Token: None 2022-09-09T22:59:45.941 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-45/?versions 2022-09-09T22:59:45.942 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OG9yXzqaDJuUxSGO7Kk81s/3MPk= 2022-09-09T22:59:45.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OG9yXzqaDJuUxSGO7Kk81s/3MPk=', 'Content-Length': '0'} 2022-09-09T22:59:45.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000607d2a2bf0c41e49-00631bc561-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.949 DEBUG:boto:b'jkcgbj-45999false' 2022-09-09T22:59:45.950 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:45.951 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:45.951 DEBUG:boto:path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.952 DEBUG:boto:auth_path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.952 DEBUG:boto:Method: GET 2022-09-09T22:59:45.953 DEBUG:boto:Path: /jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:45.953 DEBUG:boto:Data: 2022-09-09T22:59:45.954 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.955 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:45.955 DEBUG:boto:Port: 8000 2022-09-09T22:59:45.956 DEBUG:boto:Params: {} 2022-09-09T22:59:45.957 DEBUG:boto:Token: None 2022-09-09T22:59:45.957 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-45/?versions 2022-09-09T22:59:45.958 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OG9yXzqaDJuUxSGO7Kk81s/3MPk= 2022-09-09T22:59:45.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OG9yXzqaDJuUxSGO7Kk81s/3MPk=', 'Content-Length': '0'} 2022-09-09T22:59:45.965 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff8376bd21c72381-00631bc561-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.965 DEBUG:boto:b'jkcgbj-45999false' 2022-09-09T22:59:45.966 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-45 zones={test-zone1, test-zone2} 2022-09-09T22:59:45.967 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-45 zones={test-zone3, test-zone2} 2022-09-09T22:59:45.967 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:45.968 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:45.969 DEBUG:boto:Method: HEAD 2022-09-09T22:59:45.970 DEBUG:boto:Path: /jkcgbj-45/ 2022-09-09T22:59:45.971 DEBUG:boto:Data: 2022-09-09T22:59:45.972 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.973 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:45.974 DEBUG:boto:Port: 8001 2022-09-09T22:59:45.975 DEBUG:boto:Params: {} 2022-09-09T22:59:45.976 DEBUG:boto:Token: None 2022-09-09T22:59:45.977 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-45/ 2022-09-09T22:59:45.978 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R6czooxbQFUQBAkcQOJxeqWIhX4= 2022-09-09T22:59:45.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R6czooxbQFUQBAkcQOJxeqWIhX4=', 'Content-Length': '0'} 2022-09-09T22:59:45.986 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005276924d03250c70-00631bc561-1403-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:45.987 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:45.988 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:45.989 DEBUG:boto:Method: HEAD 2022-09-09T22:59:45.990 DEBUG:boto:Path: /jkcgbj-45/ 2022-09-09T22:59:45.991 DEBUG:boto:Data: 2022-09-09T22:59:45.992 DEBUG:boto:Headers: {} 2022-09-09T22:59:45.993 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:45.994 DEBUG:boto:Port: 8000 2022-09-09T22:59:45.994 DEBUG:boto:Params: {} 2022-09-09T22:59:45.995 DEBUG:boto:Token: None 2022-09-09T22:59:45.995 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:45 GMT /jkcgbj-45/ 2022-09-09T22:59:45.996 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:R6czooxbQFUQBAkcQOJxeqWIhX4= 2022-09-09T22:59:45.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:45 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:R6czooxbQFUQBAkcQOJxeqWIhX4=', 'Content-Length': '0'} 2022-09-09T22:59:46.002 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e2081c2dfcc76948-00631bc561-1549-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:46.003 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:46.004 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:46.004 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:46.005 DEBUG:boto:path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.005 DEBUG:boto:auth_path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.006 DEBUG:boto:Method: GET 2022-09-09T22:59:46.006 DEBUG:boto:Path: /jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.007 DEBUG:boto:Data: 2022-09-09T22:59:46.007 DEBUG:boto:Headers: {} 2022-09-09T22:59:46.008 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:46.008 DEBUG:boto:Port: 8001 2022-09-09T22:59:46.009 DEBUG:boto:Params: {} 2022-09-09T22:59:46.010 DEBUG:boto:Token: None 2022-09-09T22:59:46.010 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:46 GMT /jkcgbj-45/?versions 2022-09-09T22:59:46.011 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:3ppJO10Pq7FeciaEMBDNKE02AfY= 2022-09-09T22:59:46.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:46 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:3ppJO10Pq7FeciaEMBDNKE02AfY=', 'Content-Length': '0'} 2022-09-09T22:59:46.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f9b6da04c078606-00631bc562-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:46.019 DEBUG:boto:b'jkcgbj-45999false' 2022-09-09T22:59:46.020 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:46.021 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:46.021 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:46.022 DEBUG:boto:path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.022 DEBUG:boto:auth_path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.023 DEBUG:boto:Method: GET 2022-09-09T22:59:46.024 DEBUG:boto:Path: /jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.024 DEBUG:boto:Data: 2022-09-09T22:59:46.025 DEBUG:boto:Headers: {} 2022-09-09T22:59:46.025 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:46.026 DEBUG:boto:Port: 8000 2022-09-09T22:59:46.026 DEBUG:boto:Params: {} 2022-09-09T22:59:46.027 DEBUG:boto:Token: None 2022-09-09T22:59:46.028 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:46 GMT /jkcgbj-45/?versions 2022-09-09T22:59:46.029 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:3ppJO10Pq7FeciaEMBDNKE02AfY= 2022-09-09T22:59:46.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:46 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:3ppJO10Pq7FeciaEMBDNKE02AfY=', 'Content-Length': '0'} 2022-09-09T22:59:46.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000073dd2ae3aae395f-00631bc562-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:46.078 DEBUG:boto:b'jkcgbj-45999false' 2022-09-09T22:59:46.079 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:46.080 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:46.080 DEBUG:boto:path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.081 DEBUG:boto:auth_path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.081 DEBUG:boto:Method: GET 2022-09-09T22:59:46.082 DEBUG:boto:Path: /jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.082 DEBUG:boto:Data: 2022-09-09T22:59:46.083 DEBUG:boto:Headers: {} 2022-09-09T22:59:46.083 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:46.084 DEBUG:boto:Port: 8001 2022-09-09T22:59:46.085 DEBUG:boto:Params: {} 2022-09-09T22:59:46.085 DEBUG:boto:Token: None 2022-09-09T22:59:46.086 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:46 GMT /jkcgbj-45/?versions 2022-09-09T22:59:46.086 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:3ppJO10Pq7FeciaEMBDNKE02AfY= 2022-09-09T22:59:46.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:46 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:3ppJO10Pq7FeciaEMBDNKE02AfY=', 'Content-Length': '0'} 2022-09-09T22:59:46.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c9352cb85f45fdf-00631bc562-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:46.095 DEBUG:boto:b'jkcgbj-45999false' 2022-09-09T22:59:46.097 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:46.098 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:46.099 DEBUG:boto:path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.100 DEBUG:boto:auth_path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.101 DEBUG:boto:Method: GET 2022-09-09T22:59:46.102 DEBUG:boto:Path: /jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.103 DEBUG:boto:Data: 2022-09-09T22:59:46.104 DEBUG:boto:Headers: {} 2022-09-09T22:59:46.105 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:46.106 DEBUG:boto:Port: 8000 2022-09-09T22:59:46.107 DEBUG:boto:Params: {} 2022-09-09T22:59:46.108 DEBUG:boto:Token: None 2022-09-09T22:59:46.109 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:46 GMT /jkcgbj-45/?versions 2022-09-09T22:59:46.110 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:3ppJO10Pq7FeciaEMBDNKE02AfY= 2022-09-09T22:59:46.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:46 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:3ppJO10Pq7FeciaEMBDNKE02AfY=', 'Content-Length': '0'} 2022-09-09T22:59:46.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000136236a45c5a4e95-00631bc562-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:46.119 DEBUG:boto:b'jkcgbj-45999false' 2022-09-09T22:59:46.120 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-45 zones={test-zone3, test-zone2} 2022-09-09T22:59:46.120 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-45 zones={test-zone1, test-zone3} 2022-09-09T22:59:46.121 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:46.121 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:46.122 DEBUG:boto:Method: HEAD 2022-09-09T22:59:46.122 DEBUG:boto:Path: /jkcgbj-45/ 2022-09-09T22:59:46.123 DEBUG:boto:Data: 2022-09-09T22:59:46.123 DEBUG:boto:Headers: {} 2022-09-09T22:59:46.124 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:46.125 DEBUG:boto:Port: 8000 2022-09-09T22:59:46.125 DEBUG:boto:Params: {} 2022-09-09T22:59:46.126 DEBUG:boto:Token: None 2022-09-09T22:59:46.127 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:46 GMT /jkcgbj-45/ 2022-09-09T22:59:46.129 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nLrLlkTLg4QQnQbLVkU7n3uWEWg= 2022-09-09T22:59:46.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:46 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nLrLlkTLg4QQnQbLVkU7n3uWEWg=', 'Content-Length': '0'} 2022-09-09T22:59:46.137 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d98a4390a29f9aa-00631bc562-13fd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:46.138 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:46.138 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:46.139 DEBUG:boto:Method: HEAD 2022-09-09T22:59:46.140 DEBUG:boto:Path: /jkcgbj-45/ 2022-09-09T22:59:46.140 DEBUG:boto:Data: 2022-09-09T22:59:46.140 DEBUG:boto:Headers: {} 2022-09-09T22:59:46.141 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:46.142 DEBUG:boto:Port: 8001 2022-09-09T22:59:46.142 DEBUG:boto:Params: {} 2022-09-09T22:59:46.143 DEBUG:boto:Token: None 2022-09-09T22:59:46.144 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:46 GMT /jkcgbj-45/ 2022-09-09T22:59:46.144 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nLrLlkTLg4QQnQbLVkU7n3uWEWg= 2022-09-09T22:59:46.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:46 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nLrLlkTLg4QQnQbLVkU7n3uWEWg=', 'Content-Length': '0'} 2022-09-09T22:59:46.150 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d9a361d1846340c-00631bc562-1403-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:46.150 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:46.151 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:46.152 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:46.152 DEBUG:boto:path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.153 DEBUG:boto:auth_path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.153 DEBUG:boto:Method: GET 2022-09-09T22:59:46.154 DEBUG:boto:Path: /jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.154 DEBUG:boto:Data: 2022-09-09T22:59:46.155 DEBUG:boto:Headers: {} 2022-09-09T22:59:46.155 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:46.156 DEBUG:boto:Port: 8000 2022-09-09T22:59:46.156 DEBUG:boto:Params: {} 2022-09-09T22:59:46.157 DEBUG:boto:Token: None 2022-09-09T22:59:46.158 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:46 GMT /jkcgbj-45/?versions 2022-09-09T22:59:46.158 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:3ppJO10Pq7FeciaEMBDNKE02AfY= 2022-09-09T22:59:46.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:46 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:3ppJO10Pq7FeciaEMBDNKE02AfY=', 'Content-Length': '0'} 2022-09-09T22:59:46.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1a26b67b5ec69ef-00631bc562-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:46.166 DEBUG:boto:b'jkcgbj-45999false' 2022-09-09T22:59:46.167 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:46.168 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:46.168 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:46.169 DEBUG:boto:path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.169 DEBUG:boto:auth_path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.170 DEBUG:boto:Method: GET 2022-09-09T22:59:46.170 DEBUG:boto:Path: /jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.171 DEBUG:boto:Data: 2022-09-09T22:59:46.172 DEBUG:boto:Headers: {} 2022-09-09T22:59:46.172 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:46.172 DEBUG:boto:Port: 8001 2022-09-09T22:59:46.173 DEBUG:boto:Params: {} 2022-09-09T22:59:46.174 DEBUG:boto:Token: None 2022-09-09T22:59:46.174 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:46 GMT /jkcgbj-45/?versions 2022-09-09T22:59:46.175 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:3ppJO10Pq7FeciaEMBDNKE02AfY= 2022-09-09T22:59:46.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:46 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:3ppJO10Pq7FeciaEMBDNKE02AfY=', 'Content-Length': '0'} 2022-09-09T22:59:46.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052b62b2720bec9b5-00631bc562-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:46.222 DEBUG:boto:b'jkcgbj-45999false' 2022-09-09T22:59:46.223 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:46.224 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:46.224 DEBUG:boto:path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.225 DEBUG:boto:auth_path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.225 DEBUG:boto:Method: GET 2022-09-09T22:59:46.226 DEBUG:boto:Path: /jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.226 DEBUG:boto:Data: 2022-09-09T22:59:46.227 DEBUG:boto:Headers: {} 2022-09-09T22:59:46.227 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:46.228 DEBUG:boto:Port: 8000 2022-09-09T22:59:46.228 DEBUG:boto:Params: {} 2022-09-09T22:59:46.229 DEBUG:boto:Token: None 2022-09-09T22:59:46.230 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:46 GMT /jkcgbj-45/?versions 2022-09-09T22:59:46.231 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:3ppJO10Pq7FeciaEMBDNKE02AfY= 2022-09-09T22:59:46.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:46 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:3ppJO10Pq7FeciaEMBDNKE02AfY=', 'Content-Length': '0'} 2022-09-09T22:59:46.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000672939746a52eb3c-00631bc562-13fd-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:46.240 DEBUG:boto:b'jkcgbj-45999false' 2022-09-09T22:59:46.241 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:46.242 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:46.243 DEBUG:boto:path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.243 DEBUG:boto:auth_path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.244 DEBUG:boto:Method: GET 2022-09-09T22:59:46.244 DEBUG:boto:Path: /jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.245 DEBUG:boto:Data: 2022-09-09T22:59:46.245 DEBUG:boto:Headers: {} 2022-09-09T22:59:46.246 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:46.246 DEBUG:boto:Port: 8001 2022-09-09T22:59:46.247 DEBUG:boto:Params: {} 2022-09-09T22:59:46.247 DEBUG:boto:Token: None 2022-09-09T22:59:46.248 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:46 GMT /jkcgbj-45/?versions 2022-09-09T22:59:46.249 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:3ppJO10Pq7FeciaEMBDNKE02AfY= 2022-09-09T22:59:46.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:46 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:3ppJO10Pq7FeciaEMBDNKE02AfY=', 'Content-Length': '0'} 2022-09-09T22:59:46.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a92fc1f3581023f8-00631bc562-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:46.257 DEBUG:boto:b'jkcgbj-45999false' 2022-09-09T22:59:46.258 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-45 zones={test-zone1, test-zone3} 2022-09-09T22:59:46.258 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jkcgbj-45 zones={test-zone2, test-zone3} 2022-09-09T22:59:46.259 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:46.259 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:46.260 DEBUG:boto:Method: HEAD 2022-09-09T22:59:46.260 DEBUG:boto:Path: /jkcgbj-45/ 2022-09-09T22:59:46.261 DEBUG:boto:Data: 2022-09-09T22:59:46.261 DEBUG:boto:Headers: {} 2022-09-09T22:59:46.262 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:46.262 DEBUG:boto:Port: 8000 2022-09-09T22:59:46.263 DEBUG:boto:Params: {} 2022-09-09T22:59:46.264 DEBUG:boto:Token: None 2022-09-09T22:59:46.265 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:46 GMT /jkcgbj-45/ 2022-09-09T22:59:46.265 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nLrLlkTLg4QQnQbLVkU7n3uWEWg= 2022-09-09T22:59:46.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:46 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nLrLlkTLg4QQnQbLVkU7n3uWEWg=', 'Content-Length': '0'} 2022-09-09T22:59:46.271 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a03836431428003-00631bc562-1549-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:46.272 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:46.272 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:46.273 DEBUG:boto:Method: HEAD 2022-09-09T22:59:46.274 DEBUG:boto:Path: /jkcgbj-45/ 2022-09-09T22:59:46.274 DEBUG:boto:Data: 2022-09-09T22:59:46.275 DEBUG:boto:Headers: {} 2022-09-09T22:59:46.275 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:46.276 DEBUG:boto:Port: 8001 2022-09-09T22:59:46.276 DEBUG:boto:Params: {} 2022-09-09T22:59:46.277 DEBUG:boto:Token: None 2022-09-09T22:59:46.278 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:46 GMT /jkcgbj-45/ 2022-09-09T22:59:46.279 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nLrLlkTLg4QQnQbLVkU7n3uWEWg= 2022-09-09T22:59:46.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:46 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nLrLlkTLg4QQnQbLVkU7n3uWEWg=', 'Content-Length': '0'} 2022-09-09T22:59:46.284 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ccf4ea5a92bb7061-00631bc562-1403-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:46.285 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-09T22:59:46.286 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:46.286 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:46.287 DEBUG:boto:path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.287 DEBUG:boto:auth_path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.288 DEBUG:boto:Method: GET 2022-09-09T22:59:46.288 DEBUG:boto:Path: /jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.289 DEBUG:boto:Data: 2022-09-09T22:59:46.289 DEBUG:boto:Headers: {} 2022-09-09T22:59:46.290 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:46.290 DEBUG:boto:Port: 8000 2022-09-09T22:59:46.291 DEBUG:boto:Params: {} 2022-09-09T22:59:46.291 DEBUG:boto:Token: None 2022-09-09T22:59:46.292 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:46 GMT /jkcgbj-45/?versions 2022-09-09T22:59:46.292 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:3ppJO10Pq7FeciaEMBDNKE02AfY= 2022-09-09T22:59:46.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:46 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:3ppJO10Pq7FeciaEMBDNKE02AfY=', 'Content-Length': '0'} 2022-09-09T22:59:46.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d80b9024de1e07ca-00631bc562-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:46.300 DEBUG:boto:b'jkcgbj-45999false' 2022-09-09T22:59:46.301 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-09T22:59:46.302 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:46.303 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:46.303 DEBUG:boto:path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.304 DEBUG:boto:auth_path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.305 DEBUG:boto:Method: GET 2022-09-09T22:59:46.306 DEBUG:boto:Path: /jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.307 DEBUG:boto:Data: 2022-09-09T22:59:46.308 DEBUG:boto:Headers: {} 2022-09-09T22:59:46.309 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:46.310 DEBUG:boto:Port: 8001 2022-09-09T22:59:46.310 DEBUG:boto:Params: {} 2022-09-09T22:59:46.311 DEBUG:boto:Token: None 2022-09-09T22:59:46.312 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:46 GMT /jkcgbj-45/?versions 2022-09-09T22:59:46.313 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:3ppJO10Pq7FeciaEMBDNKE02AfY= 2022-09-09T22:59:46.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:46 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:3ppJO10Pq7FeciaEMBDNKE02AfY=', 'Content-Length': '0'} 2022-09-09T22:59:46.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000888d4e964811d0f7-00631bc562-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:46.362 DEBUG:boto:b'jkcgbj-45999false' 2022-09-09T22:59:46.364 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:46.364 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:46.365 DEBUG:boto:path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.366 DEBUG:boto:auth_path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.367 DEBUG:boto:Method: GET 2022-09-09T22:59:46.368 DEBUG:boto:Path: /jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.370 DEBUG:boto:Data: 2022-09-09T22:59:46.371 DEBUG:boto:Headers: {} 2022-09-09T22:59:46.372 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:46.373 DEBUG:boto:Port: 8000 2022-09-09T22:59:46.373 DEBUG:boto:Params: {} 2022-09-09T22:59:46.374 DEBUG:boto:Token: None 2022-09-09T22:59:46.375 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:46 GMT /jkcgbj-45/?versions 2022-09-09T22:59:46.376 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:3ppJO10Pq7FeciaEMBDNKE02AfY= 2022-09-09T22:59:46.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:46 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:3ppJO10Pq7FeciaEMBDNKE02AfY=', 'Content-Length': '0'} 2022-09-09T22:59:46.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f409b7390394edee-00631bc562-1549-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:46.384 DEBUG:boto:b'jkcgbj-45999false' 2022-09-09T22:59:46.385 DEBUG:boto:path=/jkcgbj-45/ 2022-09-09T22:59:46.386 DEBUG:boto:auth_path=/jkcgbj-45/ 2022-09-09T22:59:46.386 DEBUG:boto:path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.387 DEBUG:boto:auth_path=/jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.387 DEBUG:boto:Method: GET 2022-09-09T22:59:46.388 DEBUG:boto:Path: /jkcgbj-45/?versions&max-keys=999 2022-09-09T22:59:46.388 DEBUG:boto:Data: 2022-09-09T22:59:46.389 DEBUG:boto:Headers: {} 2022-09-09T22:59:46.389 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:46.391 DEBUG:boto:Port: 8001 2022-09-09T22:59:46.392 DEBUG:boto:Params: {} 2022-09-09T22:59:46.393 DEBUG:boto:Token: None 2022-09-09T22:59:46.394 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 22:59:46 GMT /jkcgbj-45/?versions 2022-09-09T22:59:46.395 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:3ppJO10Pq7FeciaEMBDNKE02AfY= 2022-09-09T22:59:46.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:46 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:3ppJO10Pq7FeciaEMBDNKE02AfY=', 'Content-Length': '0'} 2022-09-09T22:59:46.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008408f85041ab0a19-00631bc562-1403-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 22:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:46.404 DEBUG:boto:b'jkcgbj-45999false' 2022-09-09T22:59:46.406 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jkcgbj-45 zones={test-zone2, test-zone3} 2022-09-09T22:59:46.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '503b1d61-4513-48c8-a308-17b76219610c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:46.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '503b1d61-4513-48c8-a308-17b76219610c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:46.408 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 503b1d61-4513-48c8-a308-17b76219610c --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:59:46.454 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:46.454 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:46.514 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T22:59:46.515 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.515 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762486.140348_44.1", 2022-09-09T22:59:46.515 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "user", 2022-09-09T22:59:46.516 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "rgw-multisite-test-user", 2022-09-09T22:59:46.516 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:28:06.140348282Z", 2022-09-09T22:59:46.516 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.517 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.517 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.517 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.517 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.518 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.518 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_hX52Pw6Vsy-toyD35b1bFsR", 2022-09-09T22:59:46.518 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.519 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.519 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.519 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.519 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.520 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.520 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.520 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.521 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762486.146147_45.1", 2022-09-09T22:59:46.521 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "user", 2022-09-09T22:59:46.521 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "rgw-multisite-test-user", 2022-09-09T22:59:46.522 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:28:06.146147984Z", 2022-09-09T22:59:46.522 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.522 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.523 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.523 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.523 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.523 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.523 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_hX52Pw6Vsy-toyD35b1bFsR", 2022-09-09T22:59:46.524 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.524 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.524 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.524 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.525 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.525 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.525 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.526 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.526 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762609.632258_38.1", 2022-09-09T22:59:46.527 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.527 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-1:4b05608c-7197-4e9b-bd29-7989607fa277.4354.1", 2022-09-09T22:59:46.528 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:09.632258573Z", 2022-09-09T22:59:46.528 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.528 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.529 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.529 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.530 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.530 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.531 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_S0f1bdC1tDuhcRUnkrIExa6", 2022-09-09T22:59:46.531 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.532 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.532 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.532 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.533 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.533 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.534 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.534 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.534 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762609.636081_39.1", 2022-09-09T22:59:46.534 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.535 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-1:4b05608c-7197-4e9b-bd29-7989607fa277.4354.1", 2022-09-09T22:59:46.535 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:09.636081844Z", 2022-09-09T22:59:46.535 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.535 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.535 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.536 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.536 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.536 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.536 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_S0f1bdC1tDuhcRUnkrIExa6", 2022-09-09T22:59:46.537 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.537 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.537 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.537 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.538 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.538 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.538 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.538 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.539 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762609.637417_40.1", 2022-09-09T22:59:46.539 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.539 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-1", 2022-09-09T22:59:46.539 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:09.637417779Z", 2022-09-09T22:59:46.540 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.540 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.540 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.540 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.541 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.541 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.541 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_ykXwHWWOzdeAKyn1PiulIFb", 2022-09-09T22:59:46.542 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.542 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.542 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.542 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.543 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.543 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.543 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.543 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.543 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762609.640815_41.1", 2022-09-09T22:59:46.544 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.544 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-1", 2022-09-09T22:59:46.544 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:09.640815201Z", 2022-09-09T22:59:46.544 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.545 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.545 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.545 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.545 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.546 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.546 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_ykXwHWWOzdeAKyn1PiulIFb", 2022-09-09T22:59:46.546 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.546 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.547 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.547 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.547 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.547 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.548 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.548 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.548 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762609.685710_42.1", 2022-09-09T22:59:46.548 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.548 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-2:4b05608c-7197-4e9b-bd29-7989607fa277.4354.2", 2022-09-09T22:59:46.549 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:09.685710351Z", 2022-09-09T22:59:46.549 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.549 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.550 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.550 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.550 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.551 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.551 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_p7pA6aFQb4GfhoLrAPoFo3L", 2022-09-09T22:59:46.551 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.551 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.551 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.552 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.552 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.552 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.552 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.553 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.553 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762609.691000_43.1", 2022-09-09T22:59:46.553 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.553 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-2:4b05608c-7197-4e9b-bd29-7989607fa277.4354.2", 2022-09-09T22:59:46.554 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:09.691000087Z", 2022-09-09T22:59:46.554 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.554 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.554 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.555 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.555 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.555 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.555 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_p7pA6aFQb4GfhoLrAPoFo3L", 2022-09-09T22:59:46.556 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.556 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.556 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.560 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.560 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.560 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.560 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.561 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.561 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762609.692503_44.1", 2022-09-09T22:59:46.561 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.562 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-2", 2022-09-09T22:59:46.562 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:09.692503564Z", 2022-09-09T22:59:46.562 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.562 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.563 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.563 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.563 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.563 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.564 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_ut5cP71qSrGfyAy9sYnUiGl", 2022-09-09T22:59:46.564 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.564 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.564 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.565 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.565 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.565 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.565 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.566 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.566 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762609.696482_45.1", 2022-09-09T22:59:46.566 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.566 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-2", 2022-09-09T22:59:46.567 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:09.696482596Z", 2022-09-09T22:59:46.567 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.567 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.568 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.568 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.568 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.569 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.569 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_ut5cP71qSrGfyAy9sYnUiGl", 2022-09-09T22:59:46.569 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.569 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.570 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.570 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.570 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.570 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.571 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.571 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.571 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762612.758261_46.1", 2022-09-09T22:59:46.571 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.572 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-3:4b05608c-7197-4e9b-bd29-7989607fa277.4354.3", 2022-09-09T22:59:46.572 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:12.758261347Z", 2022-09-09T22:59:46.572 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.572 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.573 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.573 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.573 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.574 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.574 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_DTaKu07VPRWGanYia-lsgfe", 2022-09-09T22:59:46.574 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.574 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.575 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.575 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.575 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.575 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.576 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.576 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.576 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762612.764541_47.1", 2022-09-09T22:59:46.576 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.577 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-3:4b05608c-7197-4e9b-bd29-7989607fa277.4354.3", 2022-09-09T22:59:46.577 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:12.764541107Z", 2022-09-09T22:59:46.577 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.577 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.578 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.578 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.578 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.578 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.579 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_DTaKu07VPRWGanYia-lsgfe", 2022-09-09T22:59:46.579 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.579 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.579 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.580 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.580 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.580 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.580 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.581 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.581 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762612.766069_48.1", 2022-09-09T22:59:46.581 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.581 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-3", 2022-09-09T22:59:46.582 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:12.766069866Z", 2022-09-09T22:59:46.582 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.582 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.583 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.583 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.583 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.583 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.584 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_LVgUDeF4cxLIns2No37bLQx", 2022-09-09T22:59:46.584 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.584 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.584 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.585 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.585 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.585 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.585 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.586 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.586 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762612.770414_49.1", 2022-09-09T22:59:46.586 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.586 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-3", 2022-09-09T22:59:46.587 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:12.770414961Z", 2022-09-09T22:59:46.587 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.587 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.587 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.587 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.588 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.588 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.588 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_LVgUDeF4cxLIns2No37bLQx", 2022-09-09T22:59:46.588 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.589 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.589 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.589 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.589 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.590 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.590 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.590 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.591 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762618.742117_50.1", 2022-09-09T22:59:46.591 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.591 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-4:4b05608c-7197-4e9b-bd29-7989607fa277.4354.4", 2022-09-09T22:59:46.591 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:18.742117846Z", 2022-09-09T22:59:46.592 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.592 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.592 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.592 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.592 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.593 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.593 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_QrfGp4QHJmNMDcTAMoEbJrk", 2022-09-09T22:59:46.593 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.593 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.594 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.594 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.594 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.594 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.595 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.595 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.595 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762618.747553_51.1", 2022-09-09T22:59:46.595 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.596 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-4:4b05608c-7197-4e9b-bd29-7989607fa277.4354.4", 2022-09-09T22:59:46.596 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:18.747553163Z", 2022-09-09T22:59:46.596 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.596 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.597 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.597 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.597 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.597 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.598 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_QrfGp4QHJmNMDcTAMoEbJrk", 2022-09-09T22:59:46.598 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.598 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.599 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.599 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.599 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.599 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.600 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.600 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.600 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762618.749036_52.1", 2022-09-09T22:59:46.600 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.601 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-4", 2022-09-09T22:59:46.601 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:18.749036760Z", 2022-09-09T22:59:46.601 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.601 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.602 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.602 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.602 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.603 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_pNYiU6yoVaLSavU2aw7eQra", 2022-09-09T22:59:46.603 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.603 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.603 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.604 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.604 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.604 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.604 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.604 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.605 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762618.752816_53.1", 2022-09-09T22:59:46.605 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.605 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-4", 2022-09-09T22:59:46.605 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:18.752816607Z", 2022-09-09T22:59:46.606 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.606 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.606 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.607 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.607 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.607 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_pNYiU6yoVaLSavU2aw7eQra", 2022-09-09T22:59:46.607 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.608 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.608 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.608 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.608 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.609 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.609 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.609 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.609 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762618.783525_54.1", 2022-09-09T22:59:46.610 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.610 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-5:4b05608c-7197-4e9b-bd29-7989607fa277.4354.5", 2022-09-09T22:59:46.610 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:18.783525474Z", 2022-09-09T22:59:46.610 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.611 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.611 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.611 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.611 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.612 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.612 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_PPSGwb27eWuU4vjS5103sBF", 2022-09-09T22:59:46.612 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.612 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.612 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.613 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.613 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.613 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.614 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.614 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.614 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762618.789318_55.1", 2022-09-09T22:59:46.614 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.615 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-5:4b05608c-7197-4e9b-bd29-7989607fa277.4354.5", 2022-09-09T22:59:46.615 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:18.789318940Z", 2022-09-09T22:59:46.615 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.615 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.616 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.616 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.616 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.616 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.617 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_PPSGwb27eWuU4vjS5103sBF", 2022-09-09T22:59:46.617 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.617 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.617 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.618 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.618 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.618 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.618 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.619 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.619 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762618.791554_56.1", 2022-09-09T22:59:46.619 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.619 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-5", 2022-09-09T22:59:46.620 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:18.791554821Z", 2022-09-09T22:59:46.620 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.620 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.620 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.620 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.621 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.621 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.621 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_rrNh6U2_REBalApJMN0ghUy", 2022-09-09T22:59:46.621 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.622 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.622 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.622 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.622 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.623 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.623 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.623 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.623 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762618.797453_57.1", 2022-09-09T22:59:46.624 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.624 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-5", 2022-09-09T22:59:46.624 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:18.797453446Z", 2022-09-09T22:59:46.624 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.625 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.625 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.625 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.625 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.626 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.626 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_rrNh6U2_REBalApJMN0ghUy", 2022-09-09T22:59:46.626 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.626 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.627 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.627 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.627 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.627 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.628 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.628 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.628 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762618.850246_58.1", 2022-09-09T22:59:46.628 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.629 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-6:4b05608c-7197-4e9b-bd29-7989607fa277.4354.6", 2022-09-09T22:59:46.629 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:18.850246676Z", 2022-09-09T22:59:46.629 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.629 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.629 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.630 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.630 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.630 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.630 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_cRk-wHT_rpr3K1T0yRJVf7Q", 2022-09-09T22:59:46.631 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.631 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.631 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.631 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.632 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.632 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.632 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.632 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.633 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762618.855531_59.1", 2022-09-09T22:59:46.633 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.633 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-6:4b05608c-7197-4e9b-bd29-7989607fa277.4354.6", 2022-09-09T22:59:46.634 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:18.855531993Z", 2022-09-09T22:59:46.634 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.634 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.635 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.635 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.635 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.635 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.636 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_cRk-wHT_rpr3K1T0yRJVf7Q", 2022-09-09T22:59:46.636 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.636 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.636 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.637 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.637 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.637 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.637 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.638 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.638 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762618.857124_60.1", 2022-09-09T22:59:46.638 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.638 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-6", 2022-09-09T22:59:46.638 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:18.857124257Z", 2022-09-09T22:59:46.639 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.639 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.639 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.639 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.640 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.640 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.640 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_uJKaJMt6G8HexUvWErzrMXK", 2022-09-09T22:59:46.640 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.641 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.641 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.641 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.641 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.642 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.642 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.642 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.642 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762618.860801_61.1", 2022-09-09T22:59:46.643 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.643 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-6", 2022-09-09T22:59:46.643 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:18.860801405Z", 2022-09-09T22:59:46.643 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.644 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.644 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.644 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.644 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.644 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.645 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_uJKaJMt6G8HexUvWErzrMXK", 2022-09-09T22:59:46.645 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.645 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.645 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.646 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.646 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.646 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.646 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.647 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.647 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762621.000610_62.1", 2022-09-09T22:59:46.647 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.647 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-7:4b05608c-7197-4e9b-bd29-7989607fa277.4354.7", 2022-09-09T22:59:46.648 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:21.000610746Z", 2022-09-09T22:59:46.648 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.648 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.648 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.648 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.649 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.649 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.649 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_RZXAZlTqoK-kSBG7i9Qb1iT", 2022-09-09T22:59:46.649 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.650 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.650 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.650 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.650 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.651 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.651 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.651 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.651 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762621.004495_63.1", 2022-09-09T22:59:46.652 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.652 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-7:4b05608c-7197-4e9b-bd29-7989607fa277.4354.7", 2022-09-09T22:59:46.652 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:21.004495603Z", 2022-09-09T22:59:46.652 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.652 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.653 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.653 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.653 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.653 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.654 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_RZXAZlTqoK-kSBG7i9Qb1iT", 2022-09-09T22:59:46.654 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.654 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.654 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.655 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.655 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.655 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.655 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.656 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.656 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762621.005900_64.1", 2022-09-09T22:59:46.656 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.656 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-7", 2022-09-09T22:59:46.657 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:21.005900841Z", 2022-09-09T22:59:46.657 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.657 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.657 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.657 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.658 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.658 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.658 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_kxHFo1nRBQfhvLRoWDL9llN", 2022-09-09T22:59:46.658 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.659 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.659 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.659 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.659 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.660 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.660 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.660 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.660 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762621.009874_65.1", 2022-09-09T22:59:46.661 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.661 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-7", 2022-09-09T22:59:46.661 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:21.009874532Z", 2022-09-09T22:59:46.661 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.661 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.662 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.662 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.662 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.662 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.663 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_kxHFo1nRBQfhvLRoWDL9llN", 2022-09-09T22:59:46.663 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.663 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.663 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.664 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.664 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.664 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.664 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.665 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.665 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762621.036747_66.1", 2022-09-09T22:59:46.665 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.665 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-8:4b05608c-7197-4e9b-bd29-7989607fa277.4354.8", 2022-09-09T22:59:46.665 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:21.036747351Z", 2022-09-09T22:59:46.666 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.666 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.666 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.666 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.667 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.667 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.667 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_CE4D4FZvn3xK_Q51293KOFn", 2022-09-09T22:59:46.667 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.668 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.668 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.668 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.668 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.669 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.669 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.669 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.669 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762621.042100_67.1", 2022-09-09T22:59:46.670 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.670 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-8:4b05608c-7197-4e9b-bd29-7989607fa277.4354.8", 2022-09-09T22:59:46.670 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:21.042100215Z", 2022-09-09T22:59:46.670 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.671 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.671 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.671 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.671 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.672 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.672 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_CE4D4FZvn3xK_Q51293KOFn", 2022-09-09T22:59:46.672 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.672 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.672 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.673 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.673 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.673 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.673 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.674 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.674 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762621.043792_68.1", 2022-09-09T22:59:46.674 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.674 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-8", 2022-09-09T22:59:46.675 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:21.043792412Z", 2022-09-09T22:59:46.675 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.675 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.675 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.676 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.676 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.676 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.676 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_LemC9Nq4ijGLYcUn_qbYGY1", 2022-09-09T22:59:46.677 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.677 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.677 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.677 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.677 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.678 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.678 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.678 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.678 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762621.047923_69.1", 2022-09-09T22:59:46.679 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.679 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-8", 2022-09-09T22:59:46.679 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:21.047923611Z", 2022-09-09T22:59:46.679 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.680 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.680 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.680 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.680 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.681 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.681 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_LemC9Nq4ijGLYcUn_qbYGY1", 2022-09-09T22:59:46.681 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.681 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.681 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.682 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.682 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.682 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.682 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.683 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.683 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762621.100882_70.1", 2022-09-09T22:59:46.683 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.683 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-9:4b05608c-7197-4e9b-bd29-7989607fa277.4354.9", 2022-09-09T22:59:46.684 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:21.100882035Z", 2022-09-09T22:59:46.684 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.684 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.684 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.685 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.685 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.685 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.685 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_mXmeXhs1mUho7dSTp_n1nkI", 2022-09-09T22:59:46.686 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.686 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.686 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.686 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.687 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.687 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.687 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.687 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.687 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762621.105043_71.1", 2022-09-09T22:59:46.688 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.688 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-9:4b05608c-7197-4e9b-bd29-7989607fa277.4354.9", 2022-09-09T22:59:46.688 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:21.105043152Z", 2022-09-09T22:59:46.688 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.689 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.689 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.689 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.689 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.690 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.690 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_mXmeXhs1mUho7dSTp_n1nkI", 2022-09-09T22:59:46.690 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.690 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.691 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.691 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.691 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.691 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.692 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.692 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.692 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762621.106530_72.1", 2022-09-09T22:59:46.692 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.693 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-9", 2022-09-09T22:59:46.693 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:21.106530704Z", 2022-09-09T22:59:46.693 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.693 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.693 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.694 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.694 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.694 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.694 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_4XQiLk5HQiFc0Zuqos3dBaU", 2022-09-09T22:59:46.695 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.695 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.695 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.695 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.696 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.696 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.696 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.696 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.697 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762621.110791_73.1", 2022-09-09T22:59:46.697 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.697 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-9", 2022-09-09T22:59:46.697 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:21.110791110Z", 2022-09-09T22:59:46.698 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.698 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.698 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.698 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.698 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.699 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.699 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_4XQiLk5HQiFc0Zuqos3dBaU", 2022-09-09T22:59:46.699 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.699 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.700 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.700 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.700 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.700 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.701 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.701 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.701 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762637.476803_74.1", 2022-09-09T22:59:46.701 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.702 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-7:4b05608c-7197-4e9b-bd29-7989607fa277.4354.7", 2022-09-09T22:59:46.702 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.476803832Z", 2022-09-09T22:59:46.702 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.702 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.703 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_RZXAZlTqoK-kSBG7i9Qb1iT", 2022-09-09T22:59:46.703 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.703 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.703 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.703 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_RZXAZlTqoK-kSBG7i9Qb1iT", 2022-09-09T22:59:46.704 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:46.704 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.704 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.704 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.705 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.705 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.705 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.705 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.706 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762637.483128_75.1", 2022-09-09T22:59:46.706 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.706 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-7:4b05608c-7197-4e9b-bd29-7989607fa277.4354.7", 2022-09-09T22:59:46.706 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.483128307Z", 2022-09-09T22:59:46.707 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.707 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.707 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_RZXAZlTqoK-kSBG7i9Qb1iT", 2022-09-09T22:59:46.707 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.707 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.708 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.708 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_RZXAZlTqoK-kSBG7i9Qb1iT", 2022-09-09T22:59:46.708 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:46.708 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.709 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.709 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.709 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.709 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.710 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.710 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.710 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762637.506467_76.1", 2022-09-09T22:59:46.710 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.711 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-7", 2022-09-09T22:59:46.711 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.506467744Z", 2022-09-09T22:59:46.711 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.711 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.712 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_kxHFo1nRBQfhvLRoWDL9llN", 2022-09-09T22:59:46.712 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.712 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.712 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.713 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_kxHFo1nRBQfhvLRoWDL9llN", 2022-09-09T22:59:46.713 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:46.713 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.713 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.713 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "remove" 2022-09-09T22:59:46.714 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.714 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.714 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.714 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.715 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762637.511752_77.1", 2022-09-09T22:59:46.715 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.715 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-7", 2022-09-09T22:59:46.715 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.511752143Z", 2022-09-09T22:59:46.716 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.716 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.716 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_kxHFo1nRBQfhvLRoWDL9llN", 2022-09-09T22:59:46.716 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.717 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.717 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.717 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_kxHFo1nRBQfhvLRoWDL9llN", 2022-09-09T22:59:46.717 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:46.718 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.718 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.718 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.718 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.718 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.719 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.719 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.719 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762637.579226_78.1", 2022-09-09T22:59:46.719 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.720 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-8:4b05608c-7197-4e9b-bd29-7989607fa277.4354.8", 2022-09-09T22:59:46.720 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.579226395Z", 2022-09-09T22:59:46.720 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.720 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.721 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_CE4D4FZvn3xK_Q51293KOFn", 2022-09-09T22:59:46.721 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.721 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.721 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.722 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_CE4D4FZvn3xK_Q51293KOFn", 2022-09-09T22:59:46.722 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:46.722 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.722 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.723 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.723 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.723 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.723 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.724 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.724 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762637.584225_79.1", 2022-09-09T22:59:46.724 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.724 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-8:4b05608c-7197-4e9b-bd29-7989607fa277.4354.8", 2022-09-09T22:59:46.725 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.584225809Z", 2022-09-09T22:59:46.725 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.725 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.725 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_CE4D4FZvn3xK_Q51293KOFn", 2022-09-09T22:59:46.726 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.726 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.726 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.726 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_CE4D4FZvn3xK_Q51293KOFn", 2022-09-09T22:59:46.726 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:46.727 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.727 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.727 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.727 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.728 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.728 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.728 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.728 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762637.601579_80.1", 2022-09-09T22:59:46.729 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.729 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-8", 2022-09-09T22:59:46.729 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.601579852Z", 2022-09-09T22:59:46.729 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.730 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.730 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_LemC9Nq4ijGLYcUn_qbYGY1", 2022-09-09T22:59:46.730 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.730 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.731 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.731 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_LemC9Nq4ijGLYcUn_qbYGY1", 2022-09-09T22:59:46.731 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:46.731 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.731 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.732 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "remove" 2022-09-09T22:59:46.732 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.732 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.732 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.733 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.733 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762637.608474_81.1", 2022-09-09T22:59:46.733 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.733 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-8", 2022-09-09T22:59:46.734 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.608474828Z", 2022-09-09T22:59:46.734 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.734 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.734 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_LemC9Nq4ijGLYcUn_qbYGY1", 2022-09-09T22:59:46.735 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.735 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.735 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.735 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_LemC9Nq4ijGLYcUn_qbYGY1", 2022-09-09T22:59:46.736 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:46.736 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.736 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.736 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.736 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.737 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.737 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.737 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.737 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762637.726627_82.1", 2022-09-09T22:59:46.738 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.738 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-9:4b05608c-7197-4e9b-bd29-7989607fa277.4354.9", 2022-09-09T22:59:46.738 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.726627171Z", 2022-09-09T22:59:46.738 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.739 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.739 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_mXmeXhs1mUho7dSTp_n1nkI", 2022-09-09T22:59:46.739 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.739 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.740 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.740 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_mXmeXhs1mUho7dSTp_n1nkI", 2022-09-09T22:59:46.740 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:46.740 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.741 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.741 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.741 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.741 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.742 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.742 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.742 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762637.731530_83.1", 2022-09-09T22:59:46.742 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.743 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-9:4b05608c-7197-4e9b-bd29-7989607fa277.4354.9", 2022-09-09T22:59:46.743 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.731530094Z", 2022-09-09T22:59:46.743 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.743 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.743 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_mXmeXhs1mUho7dSTp_n1nkI", 2022-09-09T22:59:46.744 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.744 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.744 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.744 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_mXmeXhs1mUho7dSTp_n1nkI", 2022-09-09T22:59:46.745 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:46.745 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.745 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.745 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.746 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.746 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.746 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.746 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.747 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762637.751637_84.1", 2022-09-09T22:59:46.747 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.747 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-9", 2022-09-09T22:59:46.747 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.751637260Z", 2022-09-09T22:59:46.747 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.748 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.748 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_4XQiLk5HQiFc0Zuqos3dBaU", 2022-09-09T22:59:46.748 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.748 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.749 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.749 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_4XQiLk5HQiFc0Zuqos3dBaU", 2022-09-09T22:59:46.749 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:46.749 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.750 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.750 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "remove" 2022-09-09T22:59:46.750 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.750 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.751 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.751 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.751 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762637.757014_85.1", 2022-09-09T22:59:46.751 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.751 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-9", 2022-09-09T22:59:46.752 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.757014147Z", 2022-09-09T22:59:46.752 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.752 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.752 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_4XQiLk5HQiFc0Zuqos3dBaU", 2022-09-09T22:59:46.753 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.753 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.753 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.753 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_4XQiLk5HQiFc0Zuqos3dBaU", 2022-09-09T22:59:46.754 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:46.754 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.754 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.754 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.755 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.755 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.755 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.755 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.755 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762643.827940_36.1", 2022-09-09T22:59:46.756 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.756 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10", 2022-09-09T22:59:46.756 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:43.827940737Z", 2022-09-09T22:59:46.756 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.757 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.757 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.757 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.757 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.758 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.758 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_H19u2ytwno45CS85TVUwNCX", 2022-09-09T22:59:46.758 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.758 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.759 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.759 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.759 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.759 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.760 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.760 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.760 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762643.831705_37.1", 2022-09-09T22:59:46.760 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.761 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10", 2022-09-09T22:59:46.761 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:43.831705348Z", 2022-09-09T22:59:46.761 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.761 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.762 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.762 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.762 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.762 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.763 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_H19u2ytwno45CS85TVUwNCX", 2022-09-09T22:59:46.763 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.763 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.763 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.764 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.764 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.764 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.764 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.765 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.765 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762643.833115_38.1", 2022-09-09T22:59:46.765 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.765 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-10", 2022-09-09T22:59:46.765 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:43.833115806Z", 2022-09-09T22:59:46.766 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.766 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.766 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.766 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.767 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.767 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.767 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_fQCrFkZsSXo9ojIGdsOR1AR", 2022-09-09T22:59:46.767 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.768 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.768 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.768 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.768 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.769 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.769 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.769 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.769 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762643.836655_39.1", 2022-09-09T22:59:46.770 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.770 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-10", 2022-09-09T22:59:46.770 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:43.836655955Z", 2022-09-09T22:59:46.770 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.770 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.771 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.771 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.771 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.771 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.772 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_fQCrFkZsSXo9ojIGdsOR1AR", 2022-09-09T22:59:46.772 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.772 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.772 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.773 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.773 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.773 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.773 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.774 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.774 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762643.862368_40.1", 2022-09-09T22:59:46.774 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.774 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11", 2022-09-09T22:59:46.774 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:43.862368801Z", 2022-09-09T22:59:46.775 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.775 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.775 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.775 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.776 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.776 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.776 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_zzZ2TMBNyJ82w6D_gHE19mR", 2022-09-09T22:59:46.776 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.777 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.777 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.777 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.777 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.778 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.778 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.778 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.778 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762643.867631_41.1", 2022-09-09T22:59:46.779 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.779 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11", 2022-09-09T22:59:46.779 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:43.867631329Z", 2022-09-09T22:59:46.779 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.780 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.780 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.780 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.780 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.780 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.781 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_zzZ2TMBNyJ82w6D_gHE19mR", 2022-09-09T22:59:46.781 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.781 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.781 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.782 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.782 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.782 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.782 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.783 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.783 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762643.869504_42.1", 2022-09-09T22:59:46.783 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.783 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-11", 2022-09-09T22:59:46.784 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:43.869504153Z", 2022-09-09T22:59:46.784 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.784 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.784 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.784 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.785 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.785 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.785 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_TbMj1OrULohgBriaPgsn-L0", 2022-09-09T22:59:46.785 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.786 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.786 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.786 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.786 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.787 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.787 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.787 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.787 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762643.874819_43.1", 2022-09-09T22:59:46.788 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.788 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-11", 2022-09-09T22:59:46.788 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:43.874819824Z", 2022-09-09T22:59:46.788 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.789 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.789 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.789 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.789 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.789 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.790 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_TbMj1OrULohgBriaPgsn-L0", 2022-09-09T22:59:46.790 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.790 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.790 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.791 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.791 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.791 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.791 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.792 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.792 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762643.923552_44.1", 2022-09-09T22:59:46.792 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.792 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12", 2022-09-09T22:59:46.793 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:43.923552476Z", 2022-09-09T22:59:46.793 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.793 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.793 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.794 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.794 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.794 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.794 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_BNcMrJxZygGN9lZLTtBZnDT", 2022-09-09T22:59:46.795 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.795 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.795 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.795 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.796 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.796 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.796 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.796 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.797 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762643.928237_45.1", 2022-09-09T22:59:46.797 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.797 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12", 2022-09-09T22:59:46.797 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:43.928237283Z", 2022-09-09T22:59:46.797 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.798 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.798 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.798 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.798 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.799 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.799 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_BNcMrJxZygGN9lZLTtBZnDT", 2022-09-09T22:59:46.799 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.799 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.800 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.800 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.800 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.800 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.801 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.801 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.801 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762643.929723_46.1", 2022-09-09T22:59:46.801 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.802 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-12", 2022-09-09T22:59:46.802 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:43.929723240Z", 2022-09-09T22:59:46.802 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.802 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.803 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.803 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.803 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.803 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.803 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_yd7O3Le1YA9DE_Q8UDH4F_4", 2022-09-09T22:59:46.804 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.804 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.804 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.804 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.805 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.805 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.805 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.805 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.806 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762643.933454_47.1", 2022-09-09T22:59:46.806 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.806 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-12", 2022-09-09T22:59:46.806 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:43.933454382Z", 2022-09-09T22:59:46.807 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.807 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.807 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.807 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.808 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.808 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.808 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_yd7O3Le1YA9DE_Q8UDH4F_4", 2022-09-09T22:59:46.808 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.809 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.809 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.809 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.809 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.809 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.810 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.810 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.810 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762786.612586_48.1", 2022-09-09T22:59:46.810 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.811 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13", 2022-09-09T22:59:46.811 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:33:06.612586069Z", 2022-09-09T22:59:46.811 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.811 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.812 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.812 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.812 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.812 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.813 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_dqgbU-URapyJOJCglaP6dNs", 2022-09-09T22:59:46.813 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.813 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.813 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.813 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.814 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.814 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.814 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.814 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.815 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762786.616793_49.1", 2022-09-09T22:59:46.815 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.815 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13", 2022-09-09T22:59:46.815 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:33:06.616793174Z", 2022-09-09T22:59:46.816 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.816 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.816 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.816 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.817 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.817 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.817 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_dqgbU-URapyJOJCglaP6dNs", 2022-09-09T22:59:46.817 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.818 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.818 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.818 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.818 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.819 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.819 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.819 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.819 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762786.618187_50.1", 2022-09-09T22:59:46.819 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.820 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-13", 2022-09-09T22:59:46.820 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:33:06.618187872Z", 2022-09-09T22:59:46.820 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.820 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.821 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.821 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.821 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.821 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.822 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_JNgA-Mcbf5d_uTKjSiKsu05", 2022-09-09T22:59:46.822 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.822 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.822 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.823 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.823 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.823 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.823 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.824 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.824 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762786.621862_51.1", 2022-09-09T22:59:46.824 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.824 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-13", 2022-09-09T22:59:46.824 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:33:06.621862110Z", 2022-09-09T22:59:46.825 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.825 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.825 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.825 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.826 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.826 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.826 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_JNgA-Mcbf5d_uTKjSiKsu05", 2022-09-09T22:59:46.826 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.827 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.827 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.827 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.827 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.828 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.828 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.828 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.828 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762786.651673_52.1", 2022-09-09T22:59:46.829 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.829 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14", 2022-09-09T22:59:46.829 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:33:06.651673719Z", 2022-09-09T22:59:46.829 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.830 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.830 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.830 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.830 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.831 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.831 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_f0wDKO7d5qM7TS5fzfH6prz", 2022-09-09T22:59:46.831 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.831 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.831 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.832 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.832 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.832 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.832 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.833 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.833 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762786.656784_53.1", 2022-09-09T22:59:46.833 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.833 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14", 2022-09-09T22:59:46.834 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:33:06.656784091Z", 2022-09-09T22:59:46.834 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.834 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.834 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.835 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.835 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.835 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.835 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_f0wDKO7d5qM7TS5fzfH6prz", 2022-09-09T22:59:46.836 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.836 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.836 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.836 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.837 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.837 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.837 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.837 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.837 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762786.658400_54.1", 2022-09-09T22:59:46.838 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.838 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-14", 2022-09-09T22:59:46.838 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:33:06.658400465Z", 2022-09-09T22:59:46.838 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.839 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.839 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.839 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.839 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.840 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.840 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_tDp27FdXJgMOU8rSYSjQb_Y", 2022-09-09T22:59:46.840 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.840 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.841 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.841 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.841 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.841 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.841 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.842 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.842 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762786.662396_55.1", 2022-09-09T22:59:46.842 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.842 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-14", 2022-09-09T22:59:46.843 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:33:06.662396356Z", 2022-09-09T22:59:46.843 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.843 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.843 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.844 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.844 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.844 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.844 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_tDp27FdXJgMOU8rSYSjQb_Y", 2022-09-09T22:59:46.845 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.845 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.845 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.845 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.845 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.846 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.846 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.846 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.846 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762786.710171_56.1", 2022-09-09T22:59:46.847 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.847 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15", 2022-09-09T22:59:46.847 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:33:06.710171968Z", 2022-09-09T22:59:46.847 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.848 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.848 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.848 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.848 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.849 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.849 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_ogbkKIIiKYU8oWXJ_RMpS5O", 2022-09-09T22:59:46.849 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.849 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.849 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.850 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.850 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.850 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.850 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.851 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.851 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762786.713926_57.1", 2022-09-09T22:59:46.851 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.851 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15", 2022-09-09T22:59:46.852 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:33:06.713926458Z", 2022-09-09T22:59:46.852 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.852 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.852 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.853 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.853 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.853 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.853 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_ogbkKIIiKYU8oWXJ_RMpS5O", 2022-09-09T22:59:46.854 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.854 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.854 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.854 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.855 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.855 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.855 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.855 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.855 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762786.715462_58.1", 2022-09-09T22:59:46.856 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.856 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-15", 2022-09-09T22:59:46.856 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:33:06.715462548Z", 2022-09-09T22:59:46.856 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.857 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.857 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.857 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.857 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.858 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.858 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_Hk_J6cauBfq0403R0YNkLtB", 2022-09-09T22:59:46.858 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.858 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.859 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.859 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.859 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.859 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.859 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.860 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.860 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762786.718887_59.1", 2022-09-09T22:59:46.860 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.860 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-15", 2022-09-09T22:59:46.861 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:33:06.718887015Z", 2022-09-09T22:59:46.861 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.861 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.861 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.862 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.862 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.862 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.862 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_Hk_J6cauBfq0403R0YNkLtB", 2022-09-09T22:59:46.863 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.863 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.863 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.863 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.864 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.864 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.864 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.864 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.864 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.055821_61.1", 2022-09-09T22:59:46.865 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.865 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16", 2022-09-09T22:59:46.865 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.055821864Z", 2022-09-09T22:59:46.865 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.866 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.866 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.866 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.866 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.867 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.867 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_jgmlG80S-C76e7qmPCPXWwY", 2022-09-09T22:59:46.867 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.867 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.868 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.868 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.868 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.868 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.868 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.869 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.869 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.060244_62.1", 2022-09-09T22:59:46.869 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.869 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16", 2022-09-09T22:59:46.870 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.060244432Z", 2022-09-09T22:59:46.870 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.870 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.870 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.871 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.871 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.871 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.871 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_jgmlG80S-C76e7qmPCPXWwY", 2022-09-09T22:59:46.872 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.872 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.872 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.872 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.873 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.873 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.873 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.873 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.874 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.061724_63.1", 2022-09-09T22:59:46.874 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.874 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-16", 2022-09-09T22:59:46.874 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.061724454Z", 2022-09-09T22:59:46.875 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.875 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.875 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.875 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.875 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.876 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.876 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_QzVpUqaeVNRQiAA9WAn6beJ", 2022-09-09T22:59:46.876 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.876 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.877 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.877 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.877 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.877 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.878 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.878 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.878 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.065560_64.1", 2022-09-09T22:59:46.878 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.879 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-16", 2022-09-09T22:59:46.879 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.065560207Z", 2022-09-09T22:59:46.879 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.879 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.880 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.880 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.880 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.880 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.880 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_QzVpUqaeVNRQiAA9WAn6beJ", 2022-09-09T22:59:46.881 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.881 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.881 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.881 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.882 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.882 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.882 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.882 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.883 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.094558_65.1", 2022-09-09T22:59:46.883 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.883 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17", 2022-09-09T22:59:46.883 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.094558838Z", 2022-09-09T22:59:46.884 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.884 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.884 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.884 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.885 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.885 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.885 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_MtuyPaZmCzfHOcgXjri3F-V", 2022-09-09T22:59:46.885 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.885 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.886 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.886 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.886 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.886 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.887 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.887 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.887 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.099477_66.1", 2022-09-09T22:59:46.887 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.888 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17", 2022-09-09T22:59:46.888 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.099477016Z", 2022-09-09T22:59:46.888 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.888 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.889 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.889 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.889 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.889 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.890 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_MtuyPaZmCzfHOcgXjri3F-V", 2022-09-09T22:59:46.890 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.890 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.890 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.891 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.891 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.891 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.891 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.891 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.892 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.101086_67.1", 2022-09-09T22:59:46.892 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.892 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-17", 2022-09-09T22:59:46.892 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.101086117Z", 2022-09-09T22:59:46.893 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.893 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.893 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.893 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.894 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.894 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.894 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_eneLpQRdXpsBeKeIZdm9Fz7", 2022-09-09T22:59:46.894 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.895 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.895 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.895 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.895 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.895 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.896 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.896 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.896 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.105046_68.1", 2022-09-09T22:59:46.896 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.897 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-17", 2022-09-09T22:59:46.897 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.105046294Z", 2022-09-09T22:59:46.897 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.897 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.898 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.898 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.898 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.898 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.899 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_eneLpQRdXpsBeKeIZdm9Fz7", 2022-09-09T22:59:46.899 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.899 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.899 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.900 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.900 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.900 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.900 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.900 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.901 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.149665_69.1", 2022-09-09T22:59:46.901 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.901 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18", 2022-09-09T22:59:46.901 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.149665741Z", 2022-09-09T22:59:46.902 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.902 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.902 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.902 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.903 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.903 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.903 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_bCS5W1GGiGEz-gWOOFrVrp4", 2022-09-09T22:59:46.903 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.904 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.904 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.904 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.904 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.905 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.905 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.905 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.905 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.156765_70.1", 2022-09-09T22:59:46.905 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.906 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18", 2022-09-09T22:59:46.906 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.156765172Z", 2022-09-09T22:59:46.906 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.906 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.907 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.907 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.907 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.907 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.908 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_bCS5W1GGiGEz-gWOOFrVrp4", 2022-09-09T22:59:46.908 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.908 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.908 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.909 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.909 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.909 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.909 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.910 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.910 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.161880_71.1", 2022-09-09T22:59:46.910 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.910 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-18", 2022-09-09T22:59:46.910 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.161880990Z", 2022-09-09T22:59:46.911 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.911 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.911 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.911 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.912 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.912 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.912 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_vVrGNRKUHx_Nc7AgE45d0q7", 2022-09-09T22:59:46.912 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.913 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.913 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.913 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.913 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.914 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.914 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.914 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.914 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.166872_72.1", 2022-09-09T22:59:46.915 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.915 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-18", 2022-09-09T22:59:46.915 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.166872706Z", 2022-09-09T22:59:46.915 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.916 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.916 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.916 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.916 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.917 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.917 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_vVrGNRKUHx_Nc7AgE45d0q7", 2022-09-09T22:59:46.917 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.917 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.917 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.918 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.918 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.918 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.918 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.919 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.919 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.191982_73.1", 2022-09-09T22:59:46.919 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.919 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16", 2022-09-09T22:59:46.920 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.191982077Z", 2022-09-09T22:59:46.920 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.920 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.920 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_jgmlG80S-C76e7qmPCPXWwY", 2022-09-09T22:59:46.921 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.921 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.921 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.921 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_jgmlG80S-C76e7qmPCPXWwY", 2022-09-09T22:59:46.922 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:46.922 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.922 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.922 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.923 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.923 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.923 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.923 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.924 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.196936_74.1", 2022-09-09T22:59:46.924 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.924 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16", 2022-09-09T22:59:46.924 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.196936218Z", 2022-09-09T22:59:46.925 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.925 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.925 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_jgmlG80S-C76e7qmPCPXWwY", 2022-09-09T22:59:46.925 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.925 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.926 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.926 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_jgmlG80S-C76e7qmPCPXWwY", 2022-09-09T22:59:46.926 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:46.926 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.927 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.927 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.927 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.927 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.928 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.928 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.928 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.211489_75.1", 2022-09-09T22:59:46.928 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.929 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17", 2022-09-09T22:59:46.929 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.211489751Z", 2022-09-09T22:59:46.929 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.929 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.929 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_MtuyPaZmCzfHOcgXjri3F-V", 2022-09-09T22:59:46.930 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.930 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.930 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.930 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_MtuyPaZmCzfHOcgXjri3F-V", 2022-09-09T22:59:46.931 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:46.931 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.931 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.931 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.932 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.932 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.932 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.932 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.933 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.217670_76.1", 2022-09-09T22:59:46.933 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.933 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17", 2022-09-09T22:59:46.933 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.217670475Z", 2022-09-09T22:59:46.934 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.934 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.934 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_MtuyPaZmCzfHOcgXjri3F-V", 2022-09-09T22:59:46.934 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.934 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.935 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.935 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_MtuyPaZmCzfHOcgXjri3F-V", 2022-09-09T22:59:46.935 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:46.935 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.936 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.936 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.936 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.936 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.937 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.937 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.937 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.236343_77.1", 2022-09-09T22:59:46.937 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.938 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18", 2022-09-09T22:59:46.938 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.236343391Z", 2022-09-09T22:59:46.938 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.938 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.939 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_bCS5W1GGiGEz-gWOOFrVrp4", 2022-09-09T22:59:46.939 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.939 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.939 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.939 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_bCS5W1GGiGEz-gWOOFrVrp4", 2022-09-09T22:59:46.940 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:46.940 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.940 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.940 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.941 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.941 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.941 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.941 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.942 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.244083_78.1", 2022-09-09T22:59:46.942 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.942 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18", 2022-09-09T22:59:46.942 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.244083785Z", 2022-09-09T22:59:46.943 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.943 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.943 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_bCS5W1GGiGEz-gWOOFrVrp4", 2022-09-09T22:59:46.943 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.944 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.944 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.944 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_bCS5W1GGiGEz-gWOOFrVrp4", 2022-09-09T22:59:46.944 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:46.944 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.945 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.945 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.945 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.945 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.946 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.946 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.946 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763177.690963_87.1", 2022-09-09T22:59:46.946 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.947 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19", 2022-09-09T22:59:46.947 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:39:37.690963866Z", 2022-09-09T22:59:46.947 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.947 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.948 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.948 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.948 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.948 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.948 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_ePT9JnPeiTDUMUQthdKtkWA", 2022-09-09T22:59:46.949 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.949 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.949 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.949 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.950 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.950 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.950 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.950 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.951 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763177.695434_88.1", 2022-09-09T22:59:46.951 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.951 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19", 2022-09-09T22:59:46.951 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:39:37.695434017Z", 2022-09-09T22:59:46.952 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.952 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.952 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.952 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.953 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.953 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.953 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_ePT9JnPeiTDUMUQthdKtkWA", 2022-09-09T22:59:46.953 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.954 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.954 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.954 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.954 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.955 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.955 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.955 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.955 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763177.697042_89.1", 2022-09-09T22:59:46.955 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.956 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-19", 2022-09-09T22:59:46.956 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:39:37.697042999Z", 2022-09-09T22:59:46.956 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.956 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.957 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.957 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.957 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.957 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.958 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_tYLgE0kYjoh4utIhhPKouFZ", 2022-09-09T22:59:46.958 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.958 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.958 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.959 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.959 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.959 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.959 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.960 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.960 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763177.701435_90.1", 2022-09-09T22:59:46.960 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.960 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-19", 2022-09-09T22:59:46.960 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:39:37.701435888Z", 2022-09-09T22:59:46.961 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.961 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.961 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.961 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.962 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.962 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.962 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_tYLgE0kYjoh4utIhhPKouFZ", 2022-09-09T22:59:46.962 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.963 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.963 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.963 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.963 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.964 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.964 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.964 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.964 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763177.715694_91.1", 2022-09-09T22:59:46.964 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.965 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19", 2022-09-09T22:59:46.965 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:39:37.715694533Z", 2022-09-09T22:59:46.965 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.965 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.966 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_ePT9JnPeiTDUMUQthdKtkWA", 2022-09-09T22:59:46.966 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.966 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.966 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.967 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_ePT9JnPeiTDUMUQthdKtkWA", 2022-09-09T22:59:46.967 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:46.967 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.967 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.968 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.968 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.968 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.968 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.969 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.969 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763177.721058_92.1", 2022-09-09T22:59:46.969 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.969 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19", 2022-09-09T22:59:46.970 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:39:37.721058937Z", 2022-09-09T22:59:46.970 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.970 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.970 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_ePT9JnPeiTDUMUQthdKtkWA", 2022-09-09T22:59:46.970 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.971 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.971 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.971 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_ePT9JnPeiTDUMUQthdKtkWA", 2022-09-09T22:59:46.971 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:46.972 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.972 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.972 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.972 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.973 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.973 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.973 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.973 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763618.848484_94.1", 2022-09-09T22:59:46.974 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.974 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20", 2022-09-09T22:59:46.974 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:46:58.848484735Z", 2022-09-09T22:59:46.974 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.975 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.975 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.975 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.975 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.975 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.976 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:46.976 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.976 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.976 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.977 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.977 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.977 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.977 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.978 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.978 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763618.852755_95.1", 2022-09-09T22:59:46.978 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.978 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20", 2022-09-09T22:59:46.979 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:46:58.852755664Z", 2022-09-09T22:59:46.979 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.979 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.979 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.979 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.980 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.980 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.980 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:46.980 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.981 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.981 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.981 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.981 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.982 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.982 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.982 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.982 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763618.854410_96.1", 2022-09-09T22:59:46.983 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.983 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-20", 2022-09-09T22:59:46.983 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:46:58.854410052Z", 2022-09-09T22:59:46.983 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.983 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.984 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.984 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.984 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.984 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.985 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_4dW2sFlblKAnYCXSGCT7lId", 2022-09-09T22:59:46.985 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.985 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.985 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.986 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.986 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.986 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.986 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.987 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.987 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763618.858372_97.1", 2022-09-09T22:59:46.987 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:46.987 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-20", 2022-09-09T22:59:46.988 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:46:58.858372321Z", 2022-09-09T22:59:46.988 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.988 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.988 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:46.988 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:46.989 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.989 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.989 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_4dW2sFlblKAnYCXSGCT7lId", 2022-09-09T22:59:46.989 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.990 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.990 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.990 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.990 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.991 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.991 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.991 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.991 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763692.030172_110.1", 2022-09-09T22:59:46.992 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.992 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20", 2022-09-09T22:59:46.992 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:48:12.030172339Z", 2022-09-09T22:59:46.992 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.993 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.993 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:46.993 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.993 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.994 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.994 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:46.994 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:46.994 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.994 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.995 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:46.995 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.995 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:46.995 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.996 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:46.996 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763692.038589_111.1", 2022-09-09T22:59:46.996 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:46.996 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20", 2022-09-09T22:59:46.997 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:48:12.038589581Z", 2022-09-09T22:59:46.997 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:46.997 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:46.997 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:46.998 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:46.998 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.998 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:46.998 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:46.999 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:46.999 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:46.999 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:46.999 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:46.999 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.000 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.000 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.000 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.000 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763726.102012_112.1", 2022-09-09T22:59:47.001 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.001 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20", 2022-09-09T22:59:47.001 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:48:46.102012638Z", 2022-09-09T22:59:47.001 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.002 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.002 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:47.002 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.002 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.003 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.003 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:47.003 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 3 2022-09-09T22:59:47.003 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.003 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.004 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.004 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.004 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.004 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.005 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.005 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763726.109956_113.1", 2022-09-09T22:59:47.005 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.005 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20", 2022-09-09T22:59:47.006 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:48:46.109956678Z", 2022-09-09T22:59:47.006 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.006 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.006 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:47.007 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.007 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.007 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.007 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:47.008 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 3 2022-09-09T22:59:47.008 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.008 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.008 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.009 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.009 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.009 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.009 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.009 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.234301_114.1", 2022-09-09T22:59:47.010 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.010 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21", 2022-09-09T22:59:47.010 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.234301916Z", 2022-09-09T22:59:47.010 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.011 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.011 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.011 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.011 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.012 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.012 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_vO_bfm5gHdzZApgtPNMJdSt", 2022-09-09T22:59:47.012 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.012 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.013 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.013 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.013 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.013 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.014 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.014 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.014 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.239008_115.1", 2022-09-09T22:59:47.014 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.015 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21", 2022-09-09T22:59:47.015 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.239008120Z", 2022-09-09T22:59:47.015 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.015 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.015 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.016 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.016 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.016 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.016 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_vO_bfm5gHdzZApgtPNMJdSt", 2022-09-09T22:59:47.017 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.017 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.017 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.017 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.018 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.018 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.018 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.018 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.019 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.240497_116.1", 2022-09-09T22:59:47.019 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.019 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-21", 2022-09-09T22:59:47.019 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.240497138Z", 2022-09-09T22:59:47.020 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.020 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.020 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.020 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.021 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.021 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.021 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_K_Aww59oBrAI7mXNiMdGk-A", 2022-09-09T22:59:47.021 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.022 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.022 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.022 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.022 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.023 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.023 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.023 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.023 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.244134_117.1", 2022-09-09T22:59:47.024 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.024 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-21", 2022-09-09T22:59:47.024 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.244134829Z", 2022-09-09T22:59:47.024 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.024 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.025 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.025 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.025 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.025 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.026 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_K_Aww59oBrAI7mXNiMdGk-A", 2022-09-09T22:59:47.026 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.026 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.026 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.027 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.027 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.027 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.027 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.028 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.028 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.271364_118.1", 2022-09-09T22:59:47.028 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.028 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22", 2022-09-09T22:59:47.029 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.271364170Z", 2022-09-09T22:59:47.029 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.029 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.029 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.029 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.030 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.030 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.030 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_oq35luOSMiKR6aRP4lKrHCx", 2022-09-09T22:59:47.030 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.031 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.031 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.031 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.031 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.032 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.032 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.032 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.032 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.276910_119.1", 2022-09-09T22:59:47.033 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.033 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22", 2022-09-09T22:59:47.033 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.276910204Z", 2022-09-09T22:59:47.033 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.034 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.034 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.034 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.034 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.035 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.035 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_oq35luOSMiKR6aRP4lKrHCx", 2022-09-09T22:59:47.035 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.035 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.035 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.036 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.036 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.036 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.036 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.037 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.037 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.278731_120.1", 2022-09-09T22:59:47.037 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.037 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-22", 2022-09-09T22:59:47.038 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.278731955Z", 2022-09-09T22:59:47.038 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.038 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.038 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.039 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.039 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.039 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.039 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_oBwxyUPEtECOF6uEAIe2K67", 2022-09-09T22:59:47.040 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.040 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.040 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.040 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.040 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.041 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.041 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.041 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.041 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.284005_121.1", 2022-09-09T22:59:47.042 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.042 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-22", 2022-09-09T22:59:47.042 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.284005465Z", 2022-09-09T22:59:47.042 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.043 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.043 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.043 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.043 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.044 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.044 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_oBwxyUPEtECOF6uEAIe2K67", 2022-09-09T22:59:47.044 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.044 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.044 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.045 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.045 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.045 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.045 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.046 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.046 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.330455_122.1", 2022-09-09T22:59:47.046 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.046 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23", 2022-09-09T22:59:47.047 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.330455662Z", 2022-09-09T22:59:47.047 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.047 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.047 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.048 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.048 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.048 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.048 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_TBNrJnEgH9G6t0MeX057y1f", 2022-09-09T22:59:47.048 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.049 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.049 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.049 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.049 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.050 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.050 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.050 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.050 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.335107_123.1", 2022-09-09T22:59:47.051 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.051 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23", 2022-09-09T22:59:47.051 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.335107502Z", 2022-09-09T22:59:47.051 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.052 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.052 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.052 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.052 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.053 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.053 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_TBNrJnEgH9G6t0MeX057y1f", 2022-09-09T22:59:47.053 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.053 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.053 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.054 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.054 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.054 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.054 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.055 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.055 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.336855_124.1", 2022-09-09T22:59:47.055 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.055 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-23", 2022-09-09T22:59:47.056 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.336855486Z", 2022-09-09T22:59:47.056 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.056 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.056 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.057 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.057 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.057 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.057 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_DGKEfxrBasYDPbGTJt7zCFo", 2022-09-09T22:59:47.058 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.058 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.058 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.058 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.059 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.059 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.059 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.059 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.059 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.340827_125.1", 2022-09-09T22:59:47.060 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.060 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-23", 2022-09-09T22:59:47.060 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.340827648Z", 2022-09-09T22:59:47.060 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.061 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.061 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.061 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.061 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.062 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.062 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_DGKEfxrBasYDPbGTJt7zCFo", 2022-09-09T22:59:47.062 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.062 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.063 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.063 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.063 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.063 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.064 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.064 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.064 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.368888_126.1", 2022-09-09T22:59:47.064 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.065 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21", 2022-09-09T22:59:47.065 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.368888518Z", 2022-09-09T22:59:47.065 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.065 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.066 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_vO_bfm5gHdzZApgtPNMJdSt", 2022-09-09T22:59:47.066 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.066 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.066 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.067 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_vO_bfm5gHdzZApgtPNMJdSt", 2022-09-09T22:59:47.067 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.067 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.067 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.067 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.068 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.068 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.068 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.068 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.069 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.374441_127.1", 2022-09-09T22:59:47.069 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.069 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21", 2022-09-09T22:59:47.069 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.374441110Z", 2022-09-09T22:59:47.070 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.070 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.070 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_vO_bfm5gHdzZApgtPNMJdSt", 2022-09-09T22:59:47.070 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.071 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.071 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.071 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_vO_bfm5gHdzZApgtPNMJdSt", 2022-09-09T22:59:47.071 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.072 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.072 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.072 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.073 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.073 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.073 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.073 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.074 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.389455_128.1", 2022-09-09T22:59:47.074 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.074 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22", 2022-09-09T22:59:47.074 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.389455212Z", 2022-09-09T22:59:47.075 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.075 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.075 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_oq35luOSMiKR6aRP4lKrHCx", 2022-09-09T22:59:47.075 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.076 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.076 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.076 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_oq35luOSMiKR6aRP4lKrHCx", 2022-09-09T22:59:47.076 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.076 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.077 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.077 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.077 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.077 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.078 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.078 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.078 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.396062_129.1", 2022-09-09T22:59:47.078 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.079 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22", 2022-09-09T22:59:47.079 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.396062782Z", 2022-09-09T22:59:47.079 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.079 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.080 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_oq35luOSMiKR6aRP4lKrHCx", 2022-09-09T22:59:47.080 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.080 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.080 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.081 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_oq35luOSMiKR6aRP4lKrHCx", 2022-09-09T22:59:47.081 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.081 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.081 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.082 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.082 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.082 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.082 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.083 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.083 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.417475_130.1", 2022-09-09T22:59:47.083 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.083 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23", 2022-09-09T22:59:47.083 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.417475123Z", 2022-09-09T22:59:47.084 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.084 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.084 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_TBNrJnEgH9G6t0MeX057y1f", 2022-09-09T22:59:47.084 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.085 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.085 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.085 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_TBNrJnEgH9G6t0MeX057y1f", 2022-09-09T22:59:47.085 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.086 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.086 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.086 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.086 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.087 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.087 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.087 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.087 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.426933_131.1", 2022-09-09T22:59:47.088 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.088 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23", 2022-09-09T22:59:47.088 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.426933442Z", 2022-09-09T22:59:47.088 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.089 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.089 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_TBNrJnEgH9G6t0MeX057y1f", 2022-09-09T22:59:47.089 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.089 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.089 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.090 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_TBNrJnEgH9G6t0MeX057y1f", 2022-09-09T22:59:47.090 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.090 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.090 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.091 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.091 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.091 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.091 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.092 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.092 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.243136_133.1", 2022-09-09T22:59:47.092 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.092 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24", 2022-09-09T22:59:47.093 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.243136650Z", 2022-09-09T22:59:47.093 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.093 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.093 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.094 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.094 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.094 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.094 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:47.095 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.095 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.095 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.095 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.096 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.096 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.096 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.096 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.096 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.247428_134.1", 2022-09-09T22:59:47.097 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.097 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24", 2022-09-09T22:59:47.097 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.247428396Z", 2022-09-09T22:59:47.097 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.098 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.098 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.098 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.098 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.099 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.099 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:47.099 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.099 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.100 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.100 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.100 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.100 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.101 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.101 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.101 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.249044_135.1", 2022-09-09T22:59:47.101 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.101 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-24", 2022-09-09T22:59:47.102 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.249044082Z", 2022-09-09T22:59:47.102 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.102 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.102 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.103 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.103 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.103 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.103 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_GPbPaiqkbVjC5D9wQQQP2jZ", 2022-09-09T22:59:47.104 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.104 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.104 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.104 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.105 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.105 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.105 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.105 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.106 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.253611_136.1", 2022-09-09T22:59:47.106 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.106 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-24", 2022-09-09T22:59:47.106 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.253611226Z", 2022-09-09T22:59:47.106 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.107 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.107 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.107 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.107 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.108 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.108 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_GPbPaiqkbVjC5D9wQQQP2jZ", 2022-09-09T22:59:47.108 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.109 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.109 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.109 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.109 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.110 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.110 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.110 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.110 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.282622_137.1", 2022-09-09T22:59:47.111 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.111 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25", 2022-09-09T22:59:47.111 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.282622692Z", 2022-09-09T22:59:47.111 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.112 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.112 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.112 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.112 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.113 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.113 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:47.113 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.113 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.114 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.114 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.114 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.114 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.115 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.115 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.115 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.287087_138.1", 2022-09-09T22:59:47.115 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.115 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25", 2022-09-09T22:59:47.116 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.287087929Z", 2022-09-09T22:59:47.116 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.116 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.116 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.117 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.117 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.117 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.117 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:47.118 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.118 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.118 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.118 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.119 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.119 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.119 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.119 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.119 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.288605_139.1", 2022-09-09T22:59:47.120 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.120 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-25", 2022-09-09T22:59:47.120 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.288605630Z", 2022-09-09T22:59:47.120 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.121 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.121 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.121 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.121 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.122 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.122 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_ZPxDZ2lg7jpxqasfBs-kwky", 2022-09-09T22:59:47.122 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.122 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.123 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.123 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.123 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.123 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.124 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.124 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.124 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.292651_140.1", 2022-09-09T22:59:47.124 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.125 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-25", 2022-09-09T22:59:47.125 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.292651882Z", 2022-09-09T22:59:47.125 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.125 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.125 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.126 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.126 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.126 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.126 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_ZPxDZ2lg7jpxqasfBs-kwky", 2022-09-09T22:59:47.127 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.127 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.127 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.127 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.128 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.128 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.128 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.128 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.129 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.347788_141.1", 2022-09-09T22:59:47.129 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.129 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26", 2022-09-09T22:59:47.129 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.347788063Z", 2022-09-09T22:59:47.129 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.130 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.130 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.130 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.130 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.131 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.131 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:47.131 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.131 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.132 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.132 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.132 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.132 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.133 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.133 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.133 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.352949_142.1", 2022-09-09T22:59:47.133 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.133 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26", 2022-09-09T22:59:47.134 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.352949747Z", 2022-09-09T22:59:47.134 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.134 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.134 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.135 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.135 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.135 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.135 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:47.136 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.136 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.136 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.136 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.137 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.137 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.137 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.137 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.138 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.354746_143.1", 2022-09-09T22:59:47.138 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.138 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-26", 2022-09-09T22:59:47.138 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.354746994Z", 2022-09-09T22:59:47.139 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.139 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.139 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.139 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.139 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.140 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.140 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_sgAtkuvW-X_3Moy5BHrhaBp", 2022-09-09T22:59:47.140 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.140 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.141 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.141 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.141 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.141 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.142 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.142 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.142 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.358613_144.1", 2022-09-09T22:59:47.142 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.143 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-26", 2022-09-09T22:59:47.143 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.358613174Z", 2022-09-09T22:59:47.143 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.143 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.144 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.144 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.144 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.144 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.144 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_sgAtkuvW-X_3Moy5BHrhaBp", 2022-09-09T22:59:47.145 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.145 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.145 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.145 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.146 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.146 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.146 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.146 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.147 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.386362_145.1", 2022-09-09T22:59:47.147 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.147 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24", 2022-09-09T22:59:47.147 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.386362667Z", 2022-09-09T22:59:47.148 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.148 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.148 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:47.148 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.149 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.149 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.149 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:47.149 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.150 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.150 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.150 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.150 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.151 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.151 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.151 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.151 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.392952_146.1", 2022-09-09T22:59:47.151 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.152 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24", 2022-09-09T22:59:47.152 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.392952544Z", 2022-09-09T22:59:47.152 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.152 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.153 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:47.153 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.153 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.153 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.154 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:47.154 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.154 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.154 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.155 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.155 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.155 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.155 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.156 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.156 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.405357_147.1", 2022-09-09T22:59:47.156 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.156 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24", 2022-09-09T22:59:47.156 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.405357667Z", 2022-09-09T22:59:47.157 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.157 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.157 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:47.157 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.158 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.158 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.158 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:47.158 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 3 2022-09-09T22:59:47.159 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.159 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.159 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.159 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.160 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.160 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.160 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.160 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.413164_148.1", 2022-09-09T22:59:47.160 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.161 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24", 2022-09-09T22:59:47.161 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.413164542Z", 2022-09-09T22:59:47.161 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.161 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.162 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:47.162 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.162 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.162 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.163 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:47.163 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 3 2022-09-09T22:59:47.163 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.163 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.164 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.164 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.164 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.164 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.164 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.165 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.428806_149.1", 2022-09-09T22:59:47.165 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.165 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25", 2022-09-09T22:59:47.165 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.428806301Z", 2022-09-09T22:59:47.166 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.166 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.166 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:47.166 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.167 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.167 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.167 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:47.167 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.168 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.168 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.168 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.168 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.168 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.169 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.169 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.169 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.436953_150.1", 2022-09-09T22:59:47.169 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.170 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25", 2022-09-09T22:59:47.170 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.436953749Z", 2022-09-09T22:59:47.170 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.170 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.171 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:47.171 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.171 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.171 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.172 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:47.172 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.172 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.172 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.172 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.173 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.173 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.173 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.173 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.174 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.460100_151.1", 2022-09-09T22:59:47.174 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.174 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25", 2022-09-09T22:59:47.174 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.460100101Z", 2022-09-09T22:59:47.175 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.175 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.175 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:47.175 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.176 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.176 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.176 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:47.176 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 3 2022-09-09T22:59:47.177 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.177 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.177 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.177 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.177 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.178 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.178 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.178 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.471650_152.1", 2022-09-09T22:59:47.178 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.179 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25", 2022-09-09T22:59:47.179 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.471650692Z", 2022-09-09T22:59:47.179 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.179 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.180 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:47.180 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.180 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.180 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.181 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:47.181 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 3 2022-09-09T22:59:47.181 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.181 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.181 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.182 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.182 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.182 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.182 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.183 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.513547_153.1", 2022-09-09T22:59:47.183 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.183 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26", 2022-09-09T22:59:47.183 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.513547080Z", 2022-09-09T22:59:47.184 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.184 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.184 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:47.184 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.185 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.185 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.185 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:47.185 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.186 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.186 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.186 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.186 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.187 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.187 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.187 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.187 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.520151_154.1", 2022-09-09T22:59:47.187 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.188 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26", 2022-09-09T22:59:47.188 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.520151282Z", 2022-09-09T22:59:47.188 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.188 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.189 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:47.189 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.189 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.189 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.190 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:47.190 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.190 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.190 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.191 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.191 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.191 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.191 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.192 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.192 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.543475_155.1", 2022-09-09T22:59:47.192 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.192 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26", 2022-09-09T22:59:47.192 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.543475276Z", 2022-09-09T22:59:47.193 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.193 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.193 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:47.193 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.194 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.194 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.194 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:47.194 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 3 2022-09-09T22:59:47.195 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.195 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.195 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.195 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.196 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.196 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.196 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.196 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.550710_156.1", 2022-09-09T22:59:47.196 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.197 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26", 2022-09-09T22:59:47.197 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.550710511Z", 2022-09-09T22:59:47.197 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.197 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.198 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:47.198 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.198 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.198 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.199 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:47.199 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 3 2022-09-09T22:59:47.199 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.199 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.200 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.200 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.200 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.200 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.201 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.201 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.587061_165.1", 2022-09-09T22:59:47.201 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.201 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-27:4b05608c-7197-4e9b-bd29-7989607fa277.4354.27", 2022-09-09T22:59:47.201 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.587061072Z", 2022-09-09T22:59:47.202 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.202 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.202 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.202 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.203 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.203 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.203 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_idGRpIZD_cBqHmSr0C8anAE", 2022-09-09T22:59:47.203 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.204 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.204 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.204 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.204 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.205 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.205 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.205 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.205 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.591517_166.1", 2022-09-09T22:59:47.206 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.206 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-27:4b05608c-7197-4e9b-bd29-7989607fa277.4354.27", 2022-09-09T22:59:47.206 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.591517503Z", 2022-09-09T22:59:47.206 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.207 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.207 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.207 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.207 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.207 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.208 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_idGRpIZD_cBqHmSr0C8anAE", 2022-09-09T22:59:47.208 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.208 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.208 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.209 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.209 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.209 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.209 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.210 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.210 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.593291_167.1", 2022-09-09T22:59:47.210 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.210 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-27", 2022-09-09T22:59:47.211 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.593291391Z", 2022-09-09T22:59:47.211 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.211 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.211 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.212 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.212 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.212 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.212 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_fil3WVqAmCZtFEsYnU9kPha", 2022-09-09T22:59:47.213 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.213 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.213 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.213 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.213 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.214 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.214 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.214 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.214 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.597347_168.1", 2022-09-09T22:59:47.215 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.215 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-27", 2022-09-09T22:59:47.215 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.597347657Z", 2022-09-09T22:59:47.215 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.216 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.216 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.216 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.216 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.217 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.217 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_fil3WVqAmCZtFEsYnU9kPha", 2022-09-09T22:59:47.217 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.217 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.217 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.218 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.218 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.218 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.218 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.219 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.219 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.629370_169.1", 2022-09-09T22:59:47.219 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.219 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-28:4b05608c-7197-4e9b-bd29-7989607fa277.4354.28", 2022-09-09T22:59:47.220 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.629370653Z", 2022-09-09T22:59:47.220 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.220 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.220 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.221 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.221 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.221 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.221 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_hWVXDIs2ffmJdaY5i2EUA49", 2022-09-09T22:59:47.222 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.222 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.222 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.222 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.223 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.223 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.223 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.223 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.224 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.634323_170.1", 2022-09-09T22:59:47.224 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.224 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-28:4b05608c-7197-4e9b-bd29-7989607fa277.4354.28", 2022-09-09T22:59:47.224 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.634323436Z", 2022-09-09T22:59:47.225 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.225 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.225 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.225 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.226 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.226 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.226 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_hWVXDIs2ffmJdaY5i2EUA49", 2022-09-09T22:59:47.226 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.226 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.227 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.227 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.227 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.227 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.228 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.228 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.228 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.636142_171.1", 2022-09-09T22:59:47.228 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.229 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-28", 2022-09-09T22:59:47.229 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.636142661Z", 2022-09-09T22:59:47.229 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.229 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.230 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.230 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.230 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.230 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.231 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_Rr2OoVgGeS8W2Ih7H5QimUB", 2022-09-09T22:59:47.231 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.231 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.231 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.231 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.232 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.232 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.232 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.232 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.233 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.640366_172.1", 2022-09-09T22:59:47.233 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.233 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-28", 2022-09-09T22:59:47.233 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.640366460Z", 2022-09-09T22:59:47.234 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.234 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.234 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.234 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.235 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.235 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.235 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_Rr2OoVgGeS8W2Ih7H5QimUB", 2022-09-09T22:59:47.235 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.236 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.236 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.236 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.236 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.236 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.237 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.237 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.237 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.691642_173.1", 2022-09-09T22:59:47.237 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.238 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-29:4b05608c-7197-4e9b-bd29-7989607fa277.4354.29", 2022-09-09T22:59:47.238 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.691642561Z", 2022-09-09T22:59:47.238 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.238 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.239 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.239 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.239 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.239 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.240 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_c7Bu5fk9awm3ApVCIJ8jMq6", 2022-09-09T22:59:47.240 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.240 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.240 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.240 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.241 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.241 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.241 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.241 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.242 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.695580_174.1", 2022-09-09T22:59:47.242 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.242 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-29:4b05608c-7197-4e9b-bd29-7989607fa277.4354.29", 2022-09-09T22:59:47.242 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.695580156Z", 2022-09-09T22:59:47.243 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.243 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.243 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.243 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.244 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.244 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.244 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_c7Bu5fk9awm3ApVCIJ8jMq6", 2022-09-09T22:59:47.244 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.245 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.245 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.245 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.245 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.245 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.246 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.246 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.246 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.696997_175.1", 2022-09-09T22:59:47.246 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.247 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-29", 2022-09-09T22:59:47.247 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.696997195Z", 2022-09-09T22:59:47.247 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.247 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.248 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.248 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.248 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.248 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.249 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_oMWT857Kx25Jy8rmeTqaRTI", 2022-09-09T22:59:47.249 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.249 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.249 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.249 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.250 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.250 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.250 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.250 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.251 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.701289_176.1", 2022-09-09T22:59:47.251 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.251 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-29", 2022-09-09T22:59:47.251 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.701289662Z", 2022-09-09T22:59:47.252 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.252 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.252 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.252 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.253 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.253 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.253 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_oMWT857Kx25Jy8rmeTqaRTI", 2022-09-09T22:59:47.253 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.254 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.254 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.254 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.254 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.255 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.255 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.255 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.255 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.733662_177.1", 2022-09-09T22:59:47.256 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.256 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-27:4b05608c-7197-4e9b-bd29-7989607fa277.4354.27", 2022-09-09T22:59:47.256 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.733662223Z", 2022-09-09T22:59:47.256 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.256 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.257 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_idGRpIZD_cBqHmSr0C8anAE", 2022-09-09T22:59:47.257 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.257 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.257 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.258 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_idGRpIZD_cBqHmSr0C8anAE", 2022-09-09T22:59:47.258 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.258 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.258 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.259 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.259 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.259 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.259 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.260 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.260 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.739796_178.1", 2022-09-09T22:59:47.260 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.260 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-27:4b05608c-7197-4e9b-bd29-7989607fa277.4354.27", 2022-09-09T22:59:47.260 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.739796617Z", 2022-09-09T22:59:47.261 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.261 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.261 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_idGRpIZD_cBqHmSr0C8anAE", 2022-09-09T22:59:47.261 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.262 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.262 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.262 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_idGRpIZD_cBqHmSr0C8anAE", 2022-09-09T22:59:47.263 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.263 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.263 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.263 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.264 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.264 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.264 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.264 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.265 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.815014_179.1", 2022-09-09T22:59:47.265 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.265 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-28:4b05608c-7197-4e9b-bd29-7989607fa277.4354.28", 2022-09-09T22:59:47.265 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.815014271Z", 2022-09-09T22:59:47.266 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.266 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.266 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_hWVXDIs2ffmJdaY5i2EUA49", 2022-09-09T22:59:47.266 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.267 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.267 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.267 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_hWVXDIs2ffmJdaY5i2EUA49", 2022-09-09T22:59:47.267 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.267 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.268 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.268 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.268 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.268 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.269 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.269 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.269 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.823504_180.1", 2022-09-09T22:59:47.269 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.270 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-28:4b05608c-7197-4e9b-bd29-7989607fa277.4354.28", 2022-09-09T22:59:47.270 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.823504725Z", 2022-09-09T22:59:47.270 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.270 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.271 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_hWVXDIs2ffmJdaY5i2EUA49", 2022-09-09T22:59:47.271 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.271 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.271 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.272 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_hWVXDIs2ffmJdaY5i2EUA49", 2022-09-09T22:59:47.272 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.272 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.272 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.273 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.273 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.273 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.273 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.273 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.274 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.861162_181.1", 2022-09-09T22:59:47.274 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.274 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-29:4b05608c-7197-4e9b-bd29-7989607fa277.4354.29", 2022-09-09T22:59:47.274 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.861162453Z", 2022-09-09T22:59:47.275 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.275 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.275 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_c7Bu5fk9awm3ApVCIJ8jMq6", 2022-09-09T22:59:47.275 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.276 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.276 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.276 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_c7Bu5fk9awm3ApVCIJ8jMq6", 2022-09-09T22:59:47.276 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.277 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.277 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.277 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.277 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.278 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.278 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.278 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.278 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.868518_182.1", 2022-09-09T22:59:47.279 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.279 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-29:4b05608c-7197-4e9b-bd29-7989607fa277.4354.29", 2022-09-09T22:59:47.279 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.868518409Z", 2022-09-09T22:59:47.279 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.279 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.280 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_c7Bu5fk9awm3ApVCIJ8jMq6", 2022-09-09T22:59:47.280 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.280 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.280 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.281 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_c7Bu5fk9awm3ApVCIJ8jMq6", 2022-09-09T22:59:47.281 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.281 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.281 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.282 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.282 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.282 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.282 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.283 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.283 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.905416_183.1", 2022-09-09T22:59:47.283 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.283 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-30:4b05608c-7197-4e9b-bd29-7989607fa277.4354.30", 2022-09-09T22:59:47.284 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.905416190Z", 2022-09-09T22:59:47.284 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.284 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.284 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.285 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.285 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.285 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.285 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xuD0Sb75U_5LCkKCpVp22OC", 2022-09-09T22:59:47.285 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.286 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.286 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.286 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.286 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.287 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.287 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.287 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.287 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.909698_184.1", 2022-09-09T22:59:47.288 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.288 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-30:4b05608c-7197-4e9b-bd29-7989607fa277.4354.30", 2022-09-09T22:59:47.288 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.909698021Z", 2022-09-09T22:59:47.288 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.289 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.289 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.289 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.289 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.289 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.290 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xuD0Sb75U_5LCkKCpVp22OC", 2022-09-09T22:59:47.290 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.290 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.290 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.291 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.291 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.291 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.291 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.292 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.292 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.911381_185.1", 2022-09-09T22:59:47.292 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.292 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-30", 2022-09-09T22:59:47.293 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.911381624Z", 2022-09-09T22:59:47.293 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.293 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.293 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.294 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.294 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.294 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.294 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_rADCtMkJa27KS0fuztGkOJ4", 2022-09-09T22:59:47.295 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.295 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.295 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.295 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.295 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.296 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.296 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.296 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.296 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.915543_186.1", 2022-09-09T22:59:47.297 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.297 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-30", 2022-09-09T22:59:47.297 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.915543203Z", 2022-09-09T22:59:47.297 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.298 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.298 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.298 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.298 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.299 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.299 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_rADCtMkJa27KS0fuztGkOJ4", 2022-09-09T22:59:47.299 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.299 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.299 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.300 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.300 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.300 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.300 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.301 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.301 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.957457_187.1", 2022-09-09T22:59:47.301 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.301 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-31:4b05608c-7197-4e9b-bd29-7989607fa277.4354.31", 2022-09-09T22:59:47.302 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.957457179Z", 2022-09-09T22:59:47.302 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.302 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.302 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.303 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.303 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.303 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.303 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_RG9oYzLzZKilowVQDrZqfZp", 2022-09-09T22:59:47.304 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.304 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.304 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.304 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.304 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.305 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.305 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.305 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.305 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.961791_188.1", 2022-09-09T22:59:47.306 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.306 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-31:4b05608c-7197-4e9b-bd29-7989607fa277.4354.31", 2022-09-09T22:59:47.306 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.961791758Z", 2022-09-09T22:59:47.306 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.307 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.307 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.307 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.307 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.308 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.308 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_RG9oYzLzZKilowVQDrZqfZp", 2022-09-09T22:59:47.308 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.308 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.309 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.309 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.309 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.309 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.309 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.310 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.310 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.963310_189.1", 2022-09-09T22:59:47.310 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.310 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-31", 2022-09-09T22:59:47.311 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.963310744Z", 2022-09-09T22:59:47.311 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.311 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.311 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.312 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.312 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.312 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.312 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_u7CGf9AlYIHVBDPwmE_8lDH", 2022-09-09T22:59:47.313 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.313 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.313 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.313 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.313 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.314 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.314 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.314 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.314 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.966754_190.1", 2022-09-09T22:59:47.315 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.315 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-31", 2022-09-09T22:59:47.315 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.966754250Z", 2022-09-09T22:59:47.315 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.316 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.316 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.316 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.316 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.317 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.317 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_u7CGf9AlYIHVBDPwmE_8lDH", 2022-09-09T22:59:47.317 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.317 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.318 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.318 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.318 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.318 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.318 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.319 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.319 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.029534_191.1", 2022-09-09T22:59:47.319 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.319 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-32:4b05608c-7197-4e9b-bd29-7989607fa277.4354.32", 2022-09-09T22:59:47.320 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.029534998Z", 2022-09-09T22:59:47.320 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.320 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.320 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.321 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.321 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.321 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.321 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_Uke9qkxor72jlh3_r-i6E2w", 2022-09-09T22:59:47.322 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.322 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.322 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.322 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.323 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.323 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.323 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.323 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.324 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.036126_192.1", 2022-09-09T22:59:47.324 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.324 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-32:4b05608c-7197-4e9b-bd29-7989607fa277.4354.32", 2022-09-09T22:59:47.324 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.036126389Z", 2022-09-09T22:59:47.325 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.325 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.325 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.325 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.326 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.326 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.326 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_Uke9qkxor72jlh3_r-i6E2w", 2022-09-09T22:59:47.327 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.327 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.327 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.327 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.327 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.328 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.328 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.328 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.329 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.040740_193.1", 2022-09-09T22:59:47.329 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.329 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-32", 2022-09-09T22:59:47.330 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.040740810Z", 2022-09-09T22:59:47.330 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.330 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.330 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.331 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.331 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.331 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.331 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_SYdiOqvbUDZ5DpIXlyitzm5", 2022-09-09T22:59:47.332 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.332 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.332 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.332 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.333 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.333 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.333 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.333 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.334 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.045233_194.1", 2022-09-09T22:59:47.334 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.334 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-32", 2022-09-09T22:59:47.334 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.045233474Z", 2022-09-09T22:59:47.334 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.335 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.335 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.335 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.335 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.336 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.336 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_SYdiOqvbUDZ5DpIXlyitzm5", 2022-09-09T22:59:47.336 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.337 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.337 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.337 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.338 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.338 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.338 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.338 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.339 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.086181_195.1", 2022-09-09T22:59:47.339 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.339 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-30:4b05608c-7197-4e9b-bd29-7989607fa277.4354.30", 2022-09-09T22:59:47.340 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.086181840Z", 2022-09-09T22:59:47.340 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.340 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.341 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xuD0Sb75U_5LCkKCpVp22OC", 2022-09-09T22:59:47.341 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.342 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.342 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.342 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xuD0Sb75U_5LCkKCpVp22OC", 2022-09-09T22:59:47.343 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.343 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.343 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.344 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.344 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.344 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.345 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.345 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.345 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.092819_196.1", 2022-09-09T22:59:47.345 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.346 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-30:4b05608c-7197-4e9b-bd29-7989607fa277.4354.30", 2022-09-09T22:59:47.346 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.092819829Z", 2022-09-09T22:59:47.346 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.346 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.346 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xuD0Sb75U_5LCkKCpVp22OC", 2022-09-09T22:59:47.347 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.347 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.347 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.347 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xuD0Sb75U_5LCkKCpVp22OC", 2022-09-09T22:59:47.348 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.348 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.348 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.348 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.349 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.349 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.349 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.349 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.350 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.131992_197.1", 2022-09-09T22:59:47.350 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.350 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-31:4b05608c-7197-4e9b-bd29-7989607fa277.4354.31", 2022-09-09T22:59:47.350 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.131992122Z", 2022-09-09T22:59:47.351 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.351 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.351 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_RG9oYzLzZKilowVQDrZqfZp", 2022-09-09T22:59:47.351 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.352 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.352 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.352 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_RG9oYzLzZKilowVQDrZqfZp", 2022-09-09T22:59:47.352 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.353 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.353 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.353 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.353 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.354 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.354 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.354 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.354 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.140406_198.1", 2022-09-09T22:59:47.355 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.355 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-31:4b05608c-7197-4e9b-bd29-7989607fa277.4354.31", 2022-09-09T22:59:47.355 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.140406171Z", 2022-09-09T22:59:47.355 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.356 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.356 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_RG9oYzLzZKilowVQDrZqfZp", 2022-09-09T22:59:47.356 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.356 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.356 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.357 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_RG9oYzLzZKilowVQDrZqfZp", 2022-09-09T22:59:47.357 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.357 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.357 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.358 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.358 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.358 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.358 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.359 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.359 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.234470_199.1", 2022-09-09T22:59:47.359 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.359 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-32:4b05608c-7197-4e9b-bd29-7989607fa277.4354.32", 2022-09-09T22:59:47.360 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.234470787Z", 2022-09-09T22:59:47.360 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.360 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.360 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_Uke9qkxor72jlh3_r-i6E2w", 2022-09-09T22:59:47.361 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.361 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.361 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.361 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_Uke9qkxor72jlh3_r-i6E2w", 2022-09-09T22:59:47.362 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.362 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.362 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.363 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.363 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.363 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.363 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.364 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.364 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.242680_200.1", 2022-09-09T22:59:47.364 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.364 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-32:4b05608c-7197-4e9b-bd29-7989607fa277.4354.32", 2022-09-09T22:59:47.365 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.242680646Z", 2022-09-09T22:59:47.365 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.365 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.365 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_Uke9qkxor72jlh3_r-i6E2w", 2022-09-09T22:59:47.366 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.366 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.366 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.366 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_Uke9qkxor72jlh3_r-i6E2w", 2022-09-09T22:59:47.367 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.367 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.367 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.367 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.368 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.368 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.368 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.368 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.369 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.342621_201.1", 2022-09-09T22:59:47.369 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.369 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-33:4b05608c-7197-4e9b-bd29-7989607fa277.4354.33", 2022-09-09T22:59:47.369 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.342621802Z", 2022-09-09T22:59:47.370 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.370 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.370 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.370 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.371 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.371 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.371 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xQnGVMAoouaLAz4qR87BBiu", 2022-09-09T22:59:47.371 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.372 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.372 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.372 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.372 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.373 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.373 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.373 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.373 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.348018_202.1", 2022-09-09T22:59:47.374 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.374 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-33:4b05608c-7197-4e9b-bd29-7989607fa277.4354.33", 2022-09-09T22:59:47.374 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.348018686Z", 2022-09-09T22:59:47.374 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.375 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.375 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.375 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.375 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.376 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.376 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xQnGVMAoouaLAz4qR87BBiu", 2022-09-09T22:59:47.376 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.376 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.377 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.377 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.377 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.377 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.378 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.378 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.378 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.352316_203.1", 2022-09-09T22:59:47.378 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.379 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-33", 2022-09-09T22:59:47.379 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.352316326Z", 2022-09-09T22:59:47.379 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.379 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.380 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.380 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.380 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.380 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.381 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_py9xB0upsDT3rp1yJKL0fW7", 2022-09-09T22:59:47.381 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.381 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.381 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.382 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.382 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.382 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.382 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.383 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.383 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.367301_204.1", 2022-09-09T22:59:47.383 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.383 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-33", 2022-09-09T22:59:47.384 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.367301625Z", 2022-09-09T22:59:47.384 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.384 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.384 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.384 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.385 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.385 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.385 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_py9xB0upsDT3rp1yJKL0fW7", 2022-09-09T22:59:47.385 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.386 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.386 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.386 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.386 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.387 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.387 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.387 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.387 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.433156_205.1", 2022-09-09T22:59:47.388 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.388 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-34:4b05608c-7197-4e9b-bd29-7989607fa277.4354.34", 2022-09-09T22:59:47.388 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.433156697Z", 2022-09-09T22:59:47.388 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.389 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.389 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.389 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.389 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.390 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.390 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_4_jQEFJZa-xniCNap3y5x4k", 2022-09-09T22:59:47.390 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.390 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.391 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.391 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.391 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.391 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.391 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.392 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.392 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.438238_206.1", 2022-09-09T22:59:47.392 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.392 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-34:4b05608c-7197-4e9b-bd29-7989607fa277.4354.34", 2022-09-09T22:59:47.393 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.438238112Z", 2022-09-09T22:59:47.393 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.393 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.393 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.394 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.394 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.394 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.394 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_4_jQEFJZa-xniCNap3y5x4k", 2022-09-09T22:59:47.395 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.395 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.395 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.395 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.395 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.396 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.396 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.396 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.396 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.439911_207.1", 2022-09-09T22:59:47.397 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.397 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-34", 2022-09-09T22:59:47.397 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.439911797Z", 2022-09-09T22:59:47.397 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.398 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.398 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.398 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.398 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.399 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.399 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_i4pt2ScTLCpaSy-sTWzcDsg", 2022-09-09T22:59:47.399 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.399 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.400 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.400 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.400 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.400 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.400 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.401 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.401 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.444301_208.1", 2022-09-09T22:59:47.401 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.401 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-34", 2022-09-09T22:59:47.402 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.444301272Z", 2022-09-09T22:59:47.402 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.402 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.402 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.403 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.403 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.403 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.403 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_i4pt2ScTLCpaSy-sTWzcDsg", 2022-09-09T22:59:47.404 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.404 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.404 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.404 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.405 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.405 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.405 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.405 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.406 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.490350_209.1", 2022-09-09T22:59:47.406 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.406 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-35:4b05608c-7197-4e9b-bd29-7989607fa277.4354.35", 2022-09-09T22:59:47.406 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.490350184Z", 2022-09-09T22:59:47.407 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.407 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.407 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.407 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.407 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.408 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.408 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xarcxlcdFlngGuAvxUBGTlW", 2022-09-09T22:59:47.408 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.408 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.409 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.409 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.409 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.409 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.410 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.410 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.410 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.495064_210.1", 2022-09-09T22:59:47.410 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.411 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-35:4b05608c-7197-4e9b-bd29-7989607fa277.4354.35", 2022-09-09T22:59:47.411 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.495064159Z", 2022-09-09T22:59:47.411 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.411 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.412 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.412 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.412 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.412 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.413 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xarcxlcdFlngGuAvxUBGTlW", 2022-09-09T22:59:47.413 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.413 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.413 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.413 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.414 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.414 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.414 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.414 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.415 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.496525_211.1", 2022-09-09T22:59:47.415 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.415 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-35", 2022-09-09T22:59:47.415 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.496525235Z", 2022-09-09T22:59:47.416 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.416 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.416 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.416 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.417 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.417 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.417 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_b22QH41XyKk67qZyODLiaOx", 2022-09-09T22:59:47.417 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.418 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.418 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.418 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.418 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.419 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.419 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.419 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.419 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.500218_212.1", 2022-09-09T22:59:47.419 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.420 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-35", 2022-09-09T22:59:47.420 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.500218886Z", 2022-09-09T22:59:47.420 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.420 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.421 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.421 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.421 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.421 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.422 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_b22QH41XyKk67qZyODLiaOx", 2022-09-09T22:59:47.422 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.422 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.422 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.423 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.423 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.423 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.423 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.424 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.424 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.527104_213.1", 2022-09-09T22:59:47.424 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.424 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-33:4b05608c-7197-4e9b-bd29-7989607fa277.4354.33", 2022-09-09T22:59:47.425 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.527104670Z", 2022-09-09T22:59:47.425 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.425 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.425 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xQnGVMAoouaLAz4qR87BBiu", 2022-09-09T22:59:47.425 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.426 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.426 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.426 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xQnGVMAoouaLAz4qR87BBiu", 2022-09-09T22:59:47.426 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.427 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.427 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.427 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.427 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.428 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.428 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.428 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.428 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.533014_214.1", 2022-09-09T22:59:47.429 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.429 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-33:4b05608c-7197-4e9b-bd29-7989607fa277.4354.33", 2022-09-09T22:59:47.429 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.533014771Z", 2022-09-09T22:59:47.429 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.430 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.430 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xQnGVMAoouaLAz4qR87BBiu", 2022-09-09T22:59:47.430 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.430 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.431 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.431 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xQnGVMAoouaLAz4qR87BBiu", 2022-09-09T22:59:47.431 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.431 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.432 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.432 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.432 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.432 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.432 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.433 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.433 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763989.558518_215.1", 2022-09-09T22:59:47.433 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.433 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-34:4b05608c-7197-4e9b-bd29-7989607fa277.4354.34", 2022-09-09T22:59:47.434 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:09.558518140Z", 2022-09-09T22:59:47.434 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.434 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.434 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_4_jQEFJZa-xniCNap3y5x4k", 2022-09-09T22:59:47.435 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.435 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.435 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.435 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_4_jQEFJZa-xniCNap3y5x4k", 2022-09-09T22:59:47.436 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.436 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.436 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.436 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.437 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.437 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.437 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.437 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.438 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763989.567553_216.1", 2022-09-09T22:59:47.438 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.438 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-34:4b05608c-7197-4e9b-bd29-7989607fa277.4354.34", 2022-09-09T22:59:47.438 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:09.567553622Z", 2022-09-09T22:59:47.439 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.439 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.439 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_4_jQEFJZa-xniCNap3y5x4k", 2022-09-09T22:59:47.439 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.439 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.440 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.440 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_4_jQEFJZa-xniCNap3y5x4k", 2022-09-09T22:59:47.440 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.440 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.441 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.441 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.441 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.441 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.442 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.442 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.442 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763990.604998_217.1", 2022-09-09T22:59:47.442 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.443 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-35:4b05608c-7197-4e9b-bd29-7989607fa277.4354.35", 2022-09-09T22:59:47.443 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:10.604998041Z", 2022-09-09T22:59:47.443 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.443 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.444 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xarcxlcdFlngGuAvxUBGTlW", 2022-09-09T22:59:47.444 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.444 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.444 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.444 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xarcxlcdFlngGuAvxUBGTlW", 2022-09-09T22:59:47.445 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.445 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.445 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.445 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.446 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.446 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.446 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.446 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.447 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763990.613292_218.1", 2022-09-09T22:59:47.447 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.447 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-35:4b05608c-7197-4e9b-bd29-7989607fa277.4354.35", 2022-09-09T22:59:47.447 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:10.613292272Z", 2022-09-09T22:59:47.448 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.448 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.448 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xarcxlcdFlngGuAvxUBGTlW", 2022-09-09T22:59:47.448 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.448 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.449 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.449 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xarcxlcdFlngGuAvxUBGTlW", 2022-09-09T22:59:47.449 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:47.449 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.450 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.450 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.450 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.450 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.451 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.451 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.451 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763990.663150_219.1", 2022-09-09T22:59:47.451 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.452 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36", 2022-09-09T22:59:47.452 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:10.663150053Z", 2022-09-09T22:59:47.452 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.452 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.453 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.453 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.453 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.453 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.453 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_jPN1kAi0u5MPj33UDl-gVII", 2022-09-09T22:59:47.454 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.454 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.454 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.454 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.455 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.455 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.455 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.455 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.456 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763990.667756_220.1", 2022-09-09T22:59:47.456 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.456 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36", 2022-09-09T22:59:47.456 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:10.667756273Z", 2022-09-09T22:59:47.457 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.457 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.457 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.457 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.458 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.458 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.458 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_jPN1kAi0u5MPj33UDl-gVII", 2022-09-09T22:59:47.458 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.459 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.459 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.459 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.459 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.459 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.460 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.460 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.460 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763990.669560_221.1", 2022-09-09T22:59:47.460 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.461 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-36", 2022-09-09T22:59:47.461 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:10.669560632Z", 2022-09-09T22:59:47.461 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.461 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.462 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.462 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.462 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.463 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.463 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_gu46bZkwwhtar51KzoH8j_M", 2022-09-09T22:59:47.463 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.463 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.464 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.464 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.464 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.464 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.465 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.465 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.465 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763990.673725_222.1", 2022-09-09T22:59:47.465 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.466 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-36", 2022-09-09T22:59:47.466 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:10.673725202Z", 2022-09-09T22:59:47.466 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.466 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.467 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.467 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.467 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.467 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.468 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_gu46bZkwwhtar51KzoH8j_M", 2022-09-09T22:59:47.468 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.468 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.468 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.468 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.469 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.469 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.469 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.469 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.470 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763990.710819_223.1", 2022-09-09T22:59:47.470 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.470 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37", 2022-09-09T22:59:47.470 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:10.710819128Z", 2022-09-09T22:59:47.471 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.471 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.471 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.471 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.472 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.472 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.472 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_FX49dxEb4uM_d2ZifiXBHNo", 2022-09-09T22:59:47.472 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.473 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.473 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.473 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.473 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.474 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.474 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.474 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.474 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763990.715764_224.1", 2022-09-09T22:59:47.475 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.475 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37", 2022-09-09T22:59:47.475 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:10.715764014Z", 2022-09-09T22:59:47.475 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.475 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.476 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.476 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.476 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.476 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.477 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_FX49dxEb4uM_d2ZifiXBHNo", 2022-09-09T22:59:47.477 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.477 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.477 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.478 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.478 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.478 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.478 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.479 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.479 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763990.717500_225.1", 2022-09-09T22:59:47.479 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.479 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-37", 2022-09-09T22:59:47.480 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:10.717500107Z", 2022-09-09T22:59:47.480 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.480 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.480 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.480 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.481 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.481 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.481 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_hJzznYY2Hded3epYJ_abHVf", 2022-09-09T22:59:47.481 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.482 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.482 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.487 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.488 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.488 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.488 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.488 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.489 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763990.722195_226.1", 2022-09-09T22:59:47.489 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.489 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-37", 2022-09-09T22:59:47.489 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:10.722195904Z", 2022-09-09T22:59:47.490 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.490 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.490 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.490 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.491 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.491 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.491 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_hJzznYY2Hded3epYJ_abHVf", 2022-09-09T22:59:47.491 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.492 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.492 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.492 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.492 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.493 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.493 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.493 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.493 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763990.784611_227.1", 2022-09-09T22:59:47.494 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.494 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38", 2022-09-09T22:59:47.494 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:10.784611285Z", 2022-09-09T22:59:47.494 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.495 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.495 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.495 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.495 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.495 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.496 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_qDxaPPFFwg6ZX6ss6qndeGO", 2022-09-09T22:59:47.496 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.496 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.496 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.497 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.497 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.497 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.498 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.498 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.498 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763990.788678_228.1", 2022-09-09T22:59:47.498 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.499 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38", 2022-09-09T22:59:47.499 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:10.788678416Z", 2022-09-09T22:59:47.499 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.499 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.499 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.500 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.500 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.500 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.500 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_qDxaPPFFwg6ZX6ss6qndeGO", 2022-09-09T22:59:47.501 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.501 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.501 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.501 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.502 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.502 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.502 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.502 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.503 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763990.792062_229.1", 2022-09-09T22:59:47.503 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.503 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-38", 2022-09-09T22:59:47.503 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:10.792062116Z", 2022-09-09T22:59:47.504 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.504 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.504 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.504 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.504 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.505 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.505 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_gCNfZZlq_uw7Nsy58KndY3K", 2022-09-09T22:59:47.505 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.505 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.506 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.506 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.506 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.506 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.507 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.507 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.507 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763990.796002_230.1", 2022-09-09T22:59:47.507 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.507 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-38", 2022-09-09T22:59:47.508 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:10.796002253Z", 2022-09-09T22:59:47.508 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.508 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.508 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.509 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.509 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.509 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.509 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_gCNfZZlq_uw7Nsy58KndY3K", 2022-09-09T22:59:47.510 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.510 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.510 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.510 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.511 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.511 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.511 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.511 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.512 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764240.029920_231.1", 2022-09-09T22:59:47.512 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.512 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-39:4b05608c-7197-4e9b-bd29-7989607fa277.4354.39", 2022-09-09T22:59:47.512 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:57:20.029920453Z", 2022-09-09T22:59:47.513 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.513 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.513 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.513 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.514 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.514 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.514 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_yHfFFXCLbn_I9-CIE0fRcSE", 2022-09-09T22:59:47.514 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.515 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.515 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.515 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.515 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.516 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.516 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.516 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.516 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764240.034539_232.1", 2022-09-09T22:59:47.516 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.517 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-39:4b05608c-7197-4e9b-bd29-7989607fa277.4354.39", 2022-09-09T22:59:47.517 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:57:20.034539448Z", 2022-09-09T22:59:47.517 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.517 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.518 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.518 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.518 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.518 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.519 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_yHfFFXCLbn_I9-CIE0fRcSE", 2022-09-09T22:59:47.519 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.519 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.519 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.520 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.520 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.520 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.520 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.521 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.521 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764240.036306_233.1", 2022-09-09T22:59:47.521 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.521 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-39", 2022-09-09T22:59:47.522 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:57:20.036306500Z", 2022-09-09T22:59:47.522 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.522 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.523 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.523 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.523 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.523 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.524 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_kuQlL_FG34kY8wCt0puQ02d", 2022-09-09T22:59:47.524 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.524 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.524 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.525 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.525 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.525 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.525 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.525 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.526 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764240.040726_234.1", 2022-09-09T22:59:47.526 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.526 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-39", 2022-09-09T22:59:47.526 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:57:20.040726112Z", 2022-09-09T22:59:47.527 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.527 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.527 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.527 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.528 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.528 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.528 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_kuQlL_FG34kY8wCt0puQ02d", 2022-09-09T22:59:47.528 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.529 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.529 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.529 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.529 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.530 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.530 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.530 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.530 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764240.078400_235.1", 2022-09-09T22:59:47.531 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.531 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-40:4b05608c-7197-4e9b-bd29-7989607fa277.4354.40", 2022-09-09T22:59:47.531 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:57:20.078400843Z", 2022-09-09T22:59:47.531 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.531 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.532 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.532 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.532 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.532 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.533 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_vfgdN60g8MjxPCCEgs2R_vv", 2022-09-09T22:59:47.533 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.533 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.533 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.534 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.534 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.534 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.534 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.535 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.535 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764240.084545_236.1", 2022-09-09T22:59:47.535 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.535 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-40:4b05608c-7197-4e9b-bd29-7989607fa277.4354.40", 2022-09-09T22:59:47.536 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:57:20.084545560Z", 2022-09-09T22:59:47.536 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.536 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.536 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.537 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.537 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.537 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.537 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_vfgdN60g8MjxPCCEgs2R_vv", 2022-09-09T22:59:47.537 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.538 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.538 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.538 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.538 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.539 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.539 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.539 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.539 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764240.086165_237.1", 2022-09-09T22:59:47.540 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.540 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-40", 2022-09-09T22:59:47.540 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:57:20.086165638Z", 2022-09-09T22:59:47.540 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.541 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.541 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.541 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.541 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.542 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.542 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_TJv9-9zJFLq6wQcqNFr3ADa", 2022-09-09T22:59:47.542 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.542 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.543 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.543 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.543 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.543 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.544 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.544 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.544 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764240.090984_238.1", 2022-09-09T22:59:47.544 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.544 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-40", 2022-09-09T22:59:47.545 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:57:20.090984173Z", 2022-09-09T22:59:47.545 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.545 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.545 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.546 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.546 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.546 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.546 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_TJv9-9zJFLq6wQcqNFr3ADa", 2022-09-09T22:59:47.547 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.547 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.547 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.547 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.548 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.548 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.548 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.548 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.549 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764240.151909_239.1", 2022-09-09T22:59:47.549 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.549 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-41:4b05608c-7197-4e9b-bd29-7989607fa277.4354.41", 2022-09-09T22:59:47.549 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:57:20.151909503Z", 2022-09-09T22:59:47.550 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.550 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.550 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.550 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.550 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.551 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.551 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_lh1D7Ui8P7X3ZyWeXN_Rbr6", 2022-09-09T22:59:47.551 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.551 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.552 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.552 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.552 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.552 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.553 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.553 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.553 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764240.157917_240.1", 2022-09-09T22:59:47.553 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.554 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-41:4b05608c-7197-4e9b-bd29-7989607fa277.4354.41", 2022-09-09T22:59:47.554 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:57:20.157917182Z", 2022-09-09T22:59:47.554 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.554 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.554 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.555 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.555 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.555 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.555 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_lh1D7Ui8P7X3ZyWeXN_Rbr6", 2022-09-09T22:59:47.556 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.556 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.556 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.556 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.557 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.557 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.557 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.557 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.558 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764240.159693_241.1", 2022-09-09T22:59:47.558 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.558 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-41", 2022-09-09T22:59:47.558 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:57:20.159693443Z", 2022-09-09T22:59:47.559 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.559 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.559 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.559 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.559 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.560 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.560 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xxvbuFhSEtP4oKwQrN2eVjC", 2022-09-09T22:59:47.560 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.560 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.561 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.561 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:47.561 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.561 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.562 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.562 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:47.562 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764240.168869_243.1", 2022-09-09T22:59:47.562 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:47.563 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-41", 2022-09-09T22:59:47.563 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:57:20.168869275Z", 2022-09-09T22:59:47.563 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:47.563 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:47.564 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:47.564 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:47.564 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.564 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:47.564 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xxvbuFhSEtP4oKwQrN2eVjC", 2022-09-09T22:59:47.565 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:47.565 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:47.565 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:47.565 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:47.566 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.566 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.566 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:47.566 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T22:59:47.570 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '503b1d61-4513-48c8-a308-17b76219610c', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:47.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '503b1d61-4513-48c8-a308-17b76219610c', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:47.571 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 503b1d61-4513-48c8-a308-17b76219610c --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:59:47.616 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:47.616 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:47.687 INFO:teuthology.orchestra.run.smithi080.stdout:[ 2022-09-09T22:59:47.687 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.687 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762486.140348_44.1", 2022-09-09T22:59:47.688 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "user", 2022-09-09T22:59:47.688 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "rgw-multisite-test-user", 2022-09-09T22:59:47.688 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.140348282Z", 2022-09-09T22:59:47.688 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.689 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.689 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.689 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.689 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.690 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.690 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_hX52Pw6Vsy-toyD35b1bFsR", 2022-09-09T22:59:47.690 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.690 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.691 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.691 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:47.691 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.692 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.692 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.692 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.693 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762486.146147_45.1", 2022-09-09T22:59:47.693 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "user", 2022-09-09T22:59:47.693 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "rgw-multisite-test-user", 2022-09-09T22:59:47.693 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147984Z", 2022-09-09T22:59:47.694 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.694 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.694 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.695 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.695 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.695 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.695 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_hX52Pw6Vsy-toyD35b1bFsR", 2022-09-09T22:59:47.695 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.696 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.696 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.696 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.697 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.697 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.697 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.697 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.698 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762609.632258_38.1", 2022-09-09T22:59:47.698 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.698 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-1:4b05608c-7197-4e9b-bd29-7989607fa277.4354.1", 2022-09-09T22:59:47.699 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:09.632258573Z", 2022-09-09T22:59:47.699 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.699 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.699 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.699 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.700 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.700 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.700 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_S0f1bdC1tDuhcRUnkrIExa6", 2022-09-09T22:59:47.700 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.701 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.701 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.702 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:47.702 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.702 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.703 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.703 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.704 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762609.636081_39.1", 2022-09-09T22:59:47.704 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.705 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-1:4b05608c-7197-4e9b-bd29-7989607fa277.4354.1", 2022-09-09T22:59:47.705 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:09.636081844Z", 2022-09-09T22:59:47.706 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.706 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.706 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.707 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.707 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.708 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.708 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_S0f1bdC1tDuhcRUnkrIExa6", 2022-09-09T22:59:47.709 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.709 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.710 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.710 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.711 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.711 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.711 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.712 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.712 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762609.637417_40.1", 2022-09-09T22:59:47.713 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.713 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-1", 2022-09-09T22:59:47.713 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:09.637417779Z", 2022-09-09T22:59:47.714 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.714 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.715 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.715 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.715 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.715 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.716 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_ykXwHWWOzdeAKyn1PiulIFb", 2022-09-09T22:59:47.716 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.716 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.716 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.716 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:47.717 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.717 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.717 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.717 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.718 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762609.640815_41.1", 2022-09-09T22:59:47.718 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.718 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-1", 2022-09-09T22:59:47.718 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:09.640815201Z", 2022-09-09T22:59:47.719 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.719 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.719 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.719 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.720 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.720 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.720 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_ykXwHWWOzdeAKyn1PiulIFb", 2022-09-09T22:59:47.720 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.721 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.721 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.721 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.721 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.722 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.722 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.722 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.722 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762609.685710_42.1", 2022-09-09T22:59:47.723 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.723 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-2:4b05608c-7197-4e9b-bd29-7989607fa277.4354.2", 2022-09-09T22:59:47.723 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:09.685710351Z", 2022-09-09T22:59:47.723 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.724 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.724 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.724 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.724 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.725 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.725 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_p7pA6aFQb4GfhoLrAPoFo3L", 2022-09-09T22:59:47.725 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.725 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.726 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.726 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:47.726 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.726 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.727 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.727 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.727 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762609.691000_43.1", 2022-09-09T22:59:47.727 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.727 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-2:4b05608c-7197-4e9b-bd29-7989607fa277.4354.2", 2022-09-09T22:59:47.728 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:09.691000087Z", 2022-09-09T22:59:47.728 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.728 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.728 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.729 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.729 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.729 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.729 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_p7pA6aFQb4GfhoLrAPoFo3L", 2022-09-09T22:59:47.730 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.730 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.730 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.733 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.733 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.734 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.734 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.734 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.734 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762609.692503_44.1", 2022-09-09T22:59:47.735 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.735 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-2", 2022-09-09T22:59:47.735 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:09.692503564Z", 2022-09-09T22:59:47.735 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.736 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.736 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.736 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.736 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.737 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.737 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_ut5cP71qSrGfyAy9sYnUiGl", 2022-09-09T22:59:47.737 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.737 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.738 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.738 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:47.738 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.738 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.739 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.739 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.739 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762609.696482_45.1", 2022-09-09T22:59:47.739 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.739 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-2", 2022-09-09T22:59:47.740 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:09.696482596Z", 2022-09-09T22:59:47.740 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.740 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.740 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.741 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.741 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.741 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.741 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_ut5cP71qSrGfyAy9sYnUiGl", 2022-09-09T22:59:47.742 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.742 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.742 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.742 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.743 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.743 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.743 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.743 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.744 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762612.758261_46.1", 2022-09-09T22:59:47.744 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.744 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-3:4b05608c-7197-4e9b-bd29-7989607fa277.4354.3", 2022-09-09T22:59:47.744 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:12.758261347Z", 2022-09-09T22:59:47.745 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.745 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.745 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.745 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.746 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.746 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.746 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_DTaKu07VPRWGanYia-lsgfe", 2022-09-09T22:59:47.746 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.747 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.747 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.747 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:47.747 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.748 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.748 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.748 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.748 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762612.764541_47.1", 2022-09-09T22:59:47.748 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.749 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-3:4b05608c-7197-4e9b-bd29-7989607fa277.4354.3", 2022-09-09T22:59:47.749 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:12.764541107Z", 2022-09-09T22:59:47.749 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.749 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.750 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.750 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.750 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.750 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.751 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_DTaKu07VPRWGanYia-lsgfe", 2022-09-09T22:59:47.751 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.751 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.751 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.752 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.752 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.752 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.752 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.753 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.753 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762612.766069_48.1", 2022-09-09T22:59:47.753 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.753 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-3", 2022-09-09T22:59:47.753 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:12.766069866Z", 2022-09-09T22:59:47.754 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.754 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.754 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.754 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.755 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.755 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.755 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_LVgUDeF4cxLIns2No37bLQx", 2022-09-09T22:59:47.755 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.756 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.756 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.756 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:47.756 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.757 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.757 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.757 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.757 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762612.770414_49.1", 2022-09-09T22:59:47.758 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.758 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-3", 2022-09-09T22:59:47.758 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:12.770414961Z", 2022-09-09T22:59:47.758 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.759 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.759 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.759 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.759 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.760 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.760 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_LVgUDeF4cxLIns2No37bLQx", 2022-09-09T22:59:47.760 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.760 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.761 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.761 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.761 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.761 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.761 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.762 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.762 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762618.742117_50.1", 2022-09-09T22:59:47.762 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.762 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-4:4b05608c-7197-4e9b-bd29-7989607fa277.4354.4", 2022-09-09T22:59:47.763 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:18.742117846Z", 2022-09-09T22:59:47.763 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.763 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.763 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.764 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.764 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.764 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.765 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_QrfGp4QHJmNMDcTAMoEbJrk", 2022-09-09T22:59:47.765 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.765 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.765 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.765 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:47.766 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.766 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.766 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.766 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.767 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762618.747553_51.1", 2022-09-09T22:59:47.767 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.767 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-4:4b05608c-7197-4e9b-bd29-7989607fa277.4354.4", 2022-09-09T22:59:47.767 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:18.747553163Z", 2022-09-09T22:59:47.768 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.768 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.768 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.768 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.769 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.769 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.769 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_QrfGp4QHJmNMDcTAMoEbJrk", 2022-09-09T22:59:47.769 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.770 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.770 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.770 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.770 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.770 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.771 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.771 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.771 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762618.749036_52.1", 2022-09-09T22:59:47.771 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.772 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-4", 2022-09-09T22:59:47.772 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:18.749036760Z", 2022-09-09T22:59:47.772 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.772 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.773 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.773 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.773 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.773 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.774 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_pNYiU6yoVaLSavU2aw7eQra", 2022-09-09T22:59:47.774 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.774 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.774 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.775 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:47.775 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.775 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.775 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.776 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.776 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762618.752816_53.1", 2022-09-09T22:59:47.776 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.776 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-4", 2022-09-09T22:59:47.777 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:18.752816607Z", 2022-09-09T22:59:47.777 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.777 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.777 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.778 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.778 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.778 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.778 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_pNYiU6yoVaLSavU2aw7eQra", 2022-09-09T22:59:47.779 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.779 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.779 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.779 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.780 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.780 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.780 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.780 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.781 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762618.783525_54.1", 2022-09-09T22:59:47.781 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.781 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-5:4b05608c-7197-4e9b-bd29-7989607fa277.4354.5", 2022-09-09T22:59:47.781 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:18.783525474Z", 2022-09-09T22:59:47.781 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.782 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.782 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.782 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.782 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.783 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.783 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_PPSGwb27eWuU4vjS5103sBF", 2022-09-09T22:59:47.783 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.783 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.784 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.784 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:47.784 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.784 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.785 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.785 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.785 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762618.789318_55.1", 2022-09-09T22:59:47.785 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.786 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-5:4b05608c-7197-4e9b-bd29-7989607fa277.4354.5", 2022-09-09T22:59:47.786 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:18.789318940Z", 2022-09-09T22:59:47.786 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.786 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.787 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.787 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.787 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.787 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.787 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_PPSGwb27eWuU4vjS5103sBF", 2022-09-09T22:59:47.788 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.788 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.788 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.788 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.789 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.789 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.789 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.789 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.790 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762618.791554_56.1", 2022-09-09T22:59:47.790 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.790 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-5", 2022-09-09T22:59:47.790 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:18.791554821Z", 2022-09-09T22:59:47.791 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.791 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.791 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.791 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.791 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.792 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.792 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_rrNh6U2_REBalApJMN0ghUy", 2022-09-09T22:59:47.792 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.792 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.793 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.793 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:47.793 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.793 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.794 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.794 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.794 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762618.797453_57.1", 2022-09-09T22:59:47.794 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.795 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-5", 2022-09-09T22:59:47.795 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:18.797453446Z", 2022-09-09T22:59:47.795 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.795 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.795 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.796 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.796 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.796 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.796 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_rrNh6U2_REBalApJMN0ghUy", 2022-09-09T22:59:47.797 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.797 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.797 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.797 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.798 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.798 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.798 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.798 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.799 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762618.850246_58.1", 2022-09-09T22:59:47.799 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.799 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-6:4b05608c-7197-4e9b-bd29-7989607fa277.4354.6", 2022-09-09T22:59:47.799 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:18.850246676Z", 2022-09-09T22:59:47.800 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.800 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.800 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.800 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.800 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.801 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.801 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_cRk-wHT_rpr3K1T0yRJVf7Q", 2022-09-09T22:59:47.801 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.801 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.802 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.802 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:47.802 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.802 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.803 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.803 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.803 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762618.855531_59.1", 2022-09-09T22:59:47.803 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.804 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-6:4b05608c-7197-4e9b-bd29-7989607fa277.4354.6", 2022-09-09T22:59:47.804 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:18.855531993Z", 2022-09-09T22:59:47.804 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.804 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.804 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.805 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.805 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.805 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.805 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_cRk-wHT_rpr3K1T0yRJVf7Q", 2022-09-09T22:59:47.806 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.806 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.806 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.806 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.807 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.807 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.807 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.807 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.808 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762618.857124_60.1", 2022-09-09T22:59:47.808 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.808 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-6", 2022-09-09T22:59:47.808 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:18.857124257Z", 2022-09-09T22:59:47.809 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.809 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.809 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.809 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.809 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.810 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.810 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_uJKaJMt6G8HexUvWErzrMXK", 2022-09-09T22:59:47.810 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.810 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.811 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.811 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:47.811 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.811 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.812 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.812 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.812 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762618.860801_61.1", 2022-09-09T22:59:47.812 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.813 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-6", 2022-09-09T22:59:47.813 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:18.860801405Z", 2022-09-09T22:59:47.813 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.813 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.814 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.814 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.814 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.814 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.815 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_uJKaJMt6G8HexUvWErzrMXK", 2022-09-09T22:59:47.815 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.815 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.815 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.816 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.816 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.816 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.816 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.817 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.817 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762621.000610_62.1", 2022-09-09T22:59:47.817 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.817 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-7:4b05608c-7197-4e9b-bd29-7989607fa277.4354.7", 2022-09-09T22:59:47.818 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:21.000610746Z", 2022-09-09T22:59:47.818 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.818 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.818 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.819 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.819 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.819 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.819 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_RZXAZlTqoK-kSBG7i9Qb1iT", 2022-09-09T22:59:47.820 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.820 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.820 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.820 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:47.820 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.821 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.821 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.821 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.821 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762621.004495_63.1", 2022-09-09T22:59:47.822 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.822 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-7:4b05608c-7197-4e9b-bd29-7989607fa277.4354.7", 2022-09-09T22:59:47.822 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:21.004495603Z", 2022-09-09T22:59:47.822 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.823 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.823 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.823 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.823 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.824 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.824 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_RZXAZlTqoK-kSBG7i9Qb1iT", 2022-09-09T22:59:47.824 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.824 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.825 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.825 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.825 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.825 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.826 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.826 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.826 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762621.005900_64.1", 2022-09-09T22:59:47.827 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.827 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-7", 2022-09-09T22:59:47.827 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:21.005900841Z", 2022-09-09T22:59:47.827 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.827 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.828 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.828 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.828 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.828 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.829 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_kxHFo1nRBQfhvLRoWDL9llN", 2022-09-09T22:59:47.829 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.829 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.829 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.830 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:47.830 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.830 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.830 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.831 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.831 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762621.009874_65.1", 2022-09-09T22:59:47.831 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.831 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-7", 2022-09-09T22:59:47.832 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:21.009874532Z", 2022-09-09T22:59:47.832 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.832 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.832 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.833 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.833 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.833 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.833 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_kxHFo1nRBQfhvLRoWDL9llN", 2022-09-09T22:59:47.834 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.834 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.834 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.834 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.835 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.835 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.835 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.835 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.835 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762621.036747_66.1", 2022-09-09T22:59:47.836 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.836 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-8:4b05608c-7197-4e9b-bd29-7989607fa277.4354.8", 2022-09-09T22:59:47.836 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:21.036747351Z", 2022-09-09T22:59:47.836 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.837 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.837 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.837 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.837 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.838 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.838 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_CE4D4FZvn3xK_Q51293KOFn", 2022-09-09T22:59:47.838 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.838 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.839 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.839 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:47.839 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.839 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.840 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.840 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.840 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762621.042100_67.1", 2022-09-09T22:59:47.840 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.840 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-8:4b05608c-7197-4e9b-bd29-7989607fa277.4354.8", 2022-09-09T22:59:47.841 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:21.042100215Z", 2022-09-09T22:59:47.841 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.841 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.841 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.842 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.842 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.842 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.842 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_CE4D4FZvn3xK_Q51293KOFn", 2022-09-09T22:59:47.843 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.843 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.843 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.843 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.844 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.844 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.844 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.844 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.845 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762621.043792_68.1", 2022-09-09T22:59:47.845 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.845 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-8", 2022-09-09T22:59:47.845 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:21.043792412Z", 2022-09-09T22:59:47.845 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.846 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.846 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.846 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.846 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.847 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.847 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_LemC9Nq4ijGLYcUn_qbYGY1", 2022-09-09T22:59:47.847 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.847 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.848 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.848 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:47.848 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.848 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.849 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.849 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.849 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762621.047923_69.1", 2022-09-09T22:59:47.849 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.849 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-8", 2022-09-09T22:59:47.850 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:21.047923611Z", 2022-09-09T22:59:47.850 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.850 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.850 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.851 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.851 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.851 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.851 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_LemC9Nq4ijGLYcUn_qbYGY1", 2022-09-09T22:59:47.852 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.852 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.852 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.852 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.853 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.853 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.853 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.853 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.854 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762621.100882_70.1", 2022-09-09T22:59:47.854 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.854 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-9:4b05608c-7197-4e9b-bd29-7989607fa277.4354.9", 2022-09-09T22:59:47.854 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:21.100882035Z", 2022-09-09T22:59:47.854 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.855 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.855 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.855 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.855 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.856 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.856 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_mXmeXhs1mUho7dSTp_n1nkI", 2022-09-09T22:59:47.856 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.856 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.857 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.857 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:47.857 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.857 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.858 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.858 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.858 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762621.105043_71.1", 2022-09-09T22:59:47.859 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.859 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-9:4b05608c-7197-4e9b-bd29-7989607fa277.4354.9", 2022-09-09T22:59:47.859 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:21.105043152Z", 2022-09-09T22:59:47.859 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.860 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.860 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.860 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.861 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.861 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.861 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_mXmeXhs1mUho7dSTp_n1nkI", 2022-09-09T22:59:47.861 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.862 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.862 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.862 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.862 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.863 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.863 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.863 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.863 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762621.106530_72.1", 2022-09-09T22:59:47.864 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.864 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-9", 2022-09-09T22:59:47.864 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:21.106530704Z", 2022-09-09T22:59:47.864 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.865 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.865 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.865 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.865 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.866 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.866 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_4XQiLk5HQiFc0Zuqos3dBaU", 2022-09-09T22:59:47.866 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.866 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.867 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.867 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:47.867 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.867 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.867 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.868 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.868 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762621.110791_73.1", 2022-09-09T22:59:47.868 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.868 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-9", 2022-09-09T22:59:47.869 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:21.110791110Z", 2022-09-09T22:59:47.869 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.869 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.869 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.870 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.870 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.870 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.870 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_4XQiLk5HQiFc0Zuqos3dBaU", 2022-09-09T22:59:47.871 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.871 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.871 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.871 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.872 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.872 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.872 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.872 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.872 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762637.476803_74.1", 2022-09-09T22:59:47.873 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.873 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-7:4b05608c-7197-4e9b-bd29-7989607fa277.4354.7", 2022-09-09T22:59:47.873 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.476803832Z", 2022-09-09T22:59:47.873 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.874 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.874 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_RZXAZlTqoK-kSBG7i9Qb1iT", 2022-09-09T22:59:47.874 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.874 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.875 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.875 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_RZXAZlTqoK-kSBG7i9Qb1iT", 2022-09-09T22:59:47.875 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:47.875 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.876 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.876 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:47.876 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.876 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.877 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.877 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.877 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762637.483128_75.1", 2022-09-09T22:59:47.877 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.877 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-7:4b05608c-7197-4e9b-bd29-7989607fa277.4354.7", 2022-09-09T22:59:47.878 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.483128307Z", 2022-09-09T22:59:47.878 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.878 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.878 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_RZXAZlTqoK-kSBG7i9Qb1iT", 2022-09-09T22:59:47.879 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.879 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.879 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.879 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_RZXAZlTqoK-kSBG7i9Qb1iT", 2022-09-09T22:59:47.880 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:47.880 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.880 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.880 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.881 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.881 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.881 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.881 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.882 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762637.506467_76.1", 2022-09-09T22:59:47.882 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.882 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-7", 2022-09-09T22:59:47.882 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.506467744Z", 2022-09-09T22:59:47.883 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.883 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.883 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_kxHFo1nRBQfhvLRoWDL9llN", 2022-09-09T22:59:47.883 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.884 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.884 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.884 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_kxHFo1nRBQfhvLRoWDL9llN", 2022-09-09T22:59:47.884 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:47.884 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.885 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.885 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "remove" 2022-09-09T22:59:47.885 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.885 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.886 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.886 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.886 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762637.511752_77.1", 2022-09-09T22:59:47.886 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.887 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-7", 2022-09-09T22:59:47.887 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.511752143Z", 2022-09-09T22:59:47.887 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.887 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.888 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_kxHFo1nRBQfhvLRoWDL9llN", 2022-09-09T22:59:47.888 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.888 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.888 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.889 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_kxHFo1nRBQfhvLRoWDL9llN", 2022-09-09T22:59:47.889 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:47.889 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.889 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.890 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.890 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.890 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.890 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.891 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.891 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762637.579226_78.1", 2022-09-09T22:59:47.891 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.891 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-8:4b05608c-7197-4e9b-bd29-7989607fa277.4354.8", 2022-09-09T22:59:47.891 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.579226395Z", 2022-09-09T22:59:47.892 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.892 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.892 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_CE4D4FZvn3xK_Q51293KOFn", 2022-09-09T22:59:47.892 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.893 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.893 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.893 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_CE4D4FZvn3xK_Q51293KOFn", 2022-09-09T22:59:47.893 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:47.894 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.894 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.894 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:47.894 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.895 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.895 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.895 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.895 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762637.584225_79.1", 2022-09-09T22:59:47.896 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.896 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-8:4b05608c-7197-4e9b-bd29-7989607fa277.4354.8", 2022-09-09T22:59:47.896 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.584225809Z", 2022-09-09T22:59:47.896 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.896 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.897 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_CE4D4FZvn3xK_Q51293KOFn", 2022-09-09T22:59:47.897 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.897 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.897 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.898 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_CE4D4FZvn3xK_Q51293KOFn", 2022-09-09T22:59:47.898 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:47.898 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.898 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.899 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.899 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.899 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.899 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.900 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.900 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762637.601579_80.1", 2022-09-09T22:59:47.900 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.900 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-8", 2022-09-09T22:59:47.901 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.601579852Z", 2022-09-09T22:59:47.901 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.901 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.901 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_LemC9Nq4ijGLYcUn_qbYGY1", 2022-09-09T22:59:47.902 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.902 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.902 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.902 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_LemC9Nq4ijGLYcUn_qbYGY1", 2022-09-09T22:59:47.902 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:47.903 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.903 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.903 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "remove" 2022-09-09T22:59:47.903 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.904 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.904 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.904 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.904 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762637.608474_81.1", 2022-09-09T22:59:47.905 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.905 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-8", 2022-09-09T22:59:47.905 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.608474828Z", 2022-09-09T22:59:47.905 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.906 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.906 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_LemC9Nq4ijGLYcUn_qbYGY1", 2022-09-09T22:59:47.906 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.906 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.907 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.907 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_LemC9Nq4ijGLYcUn_qbYGY1", 2022-09-09T22:59:47.907 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:47.907 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.908 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.908 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.908 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.908 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.908 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.909 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.909 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762637.726627_82.1", 2022-09-09T22:59:47.909 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.909 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-9:4b05608c-7197-4e9b-bd29-7989607fa277.4354.9", 2022-09-09T22:59:47.910 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.726627171Z", 2022-09-09T22:59:47.910 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.910 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.910 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_mXmeXhs1mUho7dSTp_n1nkI", 2022-09-09T22:59:47.911 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.911 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.911 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.911 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_mXmeXhs1mUho7dSTp_n1nkI", 2022-09-09T22:59:47.912 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:47.912 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.912 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.912 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:47.913 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.913 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.913 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.913 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.914 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762637.731530_83.1", 2022-09-09T22:59:47.914 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.914 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-9:4b05608c-7197-4e9b-bd29-7989607fa277.4354.9", 2022-09-09T22:59:47.914 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.731530094Z", 2022-09-09T22:59:47.915 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.915 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.915 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_mXmeXhs1mUho7dSTp_n1nkI", 2022-09-09T22:59:47.915 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.915 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.916 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.916 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_mXmeXhs1mUho7dSTp_n1nkI", 2022-09-09T22:59:47.916 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:47.916 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.917 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.917 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.917 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.917 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.918 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.918 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.918 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762637.751637_84.1", 2022-09-09T22:59:47.918 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.919 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-9", 2022-09-09T22:59:47.919 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.751637260Z", 2022-09-09T22:59:47.919 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.919 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.920 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_4XQiLk5HQiFc0Zuqos3dBaU", 2022-09-09T22:59:47.920 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.920 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.920 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.920 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_4XQiLk5HQiFc0Zuqos3dBaU", 2022-09-09T22:59:47.921 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:47.921 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.921 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.921 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "remove" 2022-09-09T22:59:47.922 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.922 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.922 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.922 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.923 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762637.757014_85.1", 2022-09-09T22:59:47.923 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.923 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-9", 2022-09-09T22:59:47.923 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014147Z", 2022-09-09T22:59:47.924 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.924 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.924 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_4XQiLk5HQiFc0Zuqos3dBaU", 2022-09-09T22:59:47.924 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.925 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.925 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.925 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_4XQiLk5HQiFc0Zuqos3dBaU", 2022-09-09T22:59:47.925 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:47.925 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.926 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.926 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.926 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.926 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.927 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.927 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.927 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762643.827940_36.1", 2022-09-09T22:59:47.927 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.928 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10", 2022-09-09T22:59:47.928 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:43.827940737Z", 2022-09-09T22:59:47.928 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.928 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.929 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.929 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.929 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.929 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.930 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_H19u2ytwno45CS85TVUwNCX", 2022-09-09T22:59:47.930 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.930 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.930 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.931 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:47.931 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.931 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.931 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.931 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.932 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762643.831705_37.1", 2022-09-09T22:59:47.932 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.932 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10", 2022-09-09T22:59:47.932 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:43.831705348Z", 2022-09-09T22:59:47.933 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.933 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.933 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.933 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.934 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.934 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.934 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_H19u2ytwno45CS85TVUwNCX", 2022-09-09T22:59:47.934 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.935 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.935 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.935 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.935 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.936 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.936 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.936 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.936 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762643.833115_38.1", 2022-09-09T22:59:47.937 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.937 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-10", 2022-09-09T22:59:47.937 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:43.833115806Z", 2022-09-09T22:59:47.937 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.938 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.938 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.938 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.938 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.938 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.939 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_fQCrFkZsSXo9ojIGdsOR1AR", 2022-09-09T22:59:47.939 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.939 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.939 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.940 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:47.940 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.940 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.940 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.941 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.941 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762643.836655_39.1", 2022-09-09T22:59:47.941 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.941 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-10", 2022-09-09T22:59:47.942 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:43.836655955Z", 2022-09-09T22:59:47.942 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.942 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.942 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.943 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.943 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.943 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.943 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_fQCrFkZsSXo9ojIGdsOR1AR", 2022-09-09T22:59:47.944 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.944 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.944 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.944 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.945 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.945 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.945 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.945 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.945 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762643.862368_40.1", 2022-09-09T22:59:47.946 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.946 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11", 2022-09-09T22:59:47.946 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:43.862368801Z", 2022-09-09T22:59:47.946 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.947 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.947 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.947 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.947 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.948 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.948 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_zzZ2TMBNyJ82w6D_gHE19mR", 2022-09-09T22:59:47.948 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.948 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.949 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.949 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:47.949 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.949 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.950 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.950 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.950 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762643.867631_41.1", 2022-09-09T22:59:47.950 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.951 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11", 2022-09-09T22:59:47.951 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:43.867631329Z", 2022-09-09T22:59:47.951 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.951 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.952 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.952 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.952 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.952 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.952 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_zzZ2TMBNyJ82w6D_gHE19mR", 2022-09-09T22:59:47.953 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.953 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.953 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.954 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.954 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.954 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.954 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.955 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.955 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762643.869504_42.1", 2022-09-09T22:59:47.955 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.955 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-11", 2022-09-09T22:59:47.956 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:43.869504153Z", 2022-09-09T22:59:47.956 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.956 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.956 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.957 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.957 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.957 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.957 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_TbMj1OrULohgBriaPgsn-L0", 2022-09-09T22:59:47.958 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.958 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.958 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.958 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:47.958 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.959 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.959 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.959 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.959 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762643.874819_43.1", 2022-09-09T22:59:47.960 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.960 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-11", 2022-09-09T22:59:47.960 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:43.874819824Z", 2022-09-09T22:59:47.960 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.961 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.961 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.961 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.961 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.962 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.962 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_TbMj1OrULohgBriaPgsn-L0", 2022-09-09T22:59:47.962 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.962 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.963 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.963 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.963 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.963 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.964 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.964 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.964 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762643.923552_44.1", 2022-09-09T22:59:47.964 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.965 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12", 2022-09-09T22:59:47.965 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:43.923552476Z", 2022-09-09T22:59:47.965 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.965 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.966 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.966 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.966 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.966 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.967 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_BNcMrJxZygGN9lZLTtBZnDT", 2022-09-09T22:59:47.967 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.967 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.967 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.967 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:47.968 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.968 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.968 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.968 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.969 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762643.928237_45.1", 2022-09-09T22:59:47.969 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.969 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12", 2022-09-09T22:59:47.969 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:43.928237283Z", 2022-09-09T22:59:47.970 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.970 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.970 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.970 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.971 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.971 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.971 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_BNcMrJxZygGN9lZLTtBZnDT", 2022-09-09T22:59:47.971 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.971 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.972 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.972 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.972 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.972 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.973 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.973 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.973 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762643.929723_46.1", 2022-09-09T22:59:47.973 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.974 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-12", 2022-09-09T22:59:47.974 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:43.929723240Z", 2022-09-09T22:59:47.974 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.974 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.975 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.975 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.975 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.975 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.975 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_yd7O3Le1YA9DE_Q8UDH4F_4", 2022-09-09T22:59:47.976 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.976 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.976 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.977 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:47.977 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.977 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.977 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.977 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.978 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762643.933454_47.1", 2022-09-09T22:59:47.978 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.978 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-12", 2022-09-09T22:59:47.978 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:43.933454382Z", 2022-09-09T22:59:47.979 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.979 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.979 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.979 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.980 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.980 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.980 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_yd7O3Le1YA9DE_Q8UDH4F_4", 2022-09-09T22:59:47.980 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.981 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.981 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.981 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.981 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.982 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.982 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.982 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.982 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762786.612586_48.1", 2022-09-09T22:59:47.983 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.983 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13", 2022-09-09T22:59:47.983 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:33:06.612586069Z", 2022-09-09T22:59:47.983 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.983 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.984 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.984 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.984 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.984 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.985 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_dqgbU-URapyJOJCglaP6dNs", 2022-09-09T22:59:47.985 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.985 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.985 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.986 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:47.986 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.986 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.986 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.987 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.987 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762786.616793_49.1", 2022-09-09T22:59:47.987 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:47.987 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13", 2022-09-09T22:59:47.988 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:33:06.616793174Z", 2022-09-09T22:59:47.988 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.988 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.988 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.988 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.989 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.989 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.989 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_dqgbU-URapyJOJCglaP6dNs", 2022-09-09T22:59:47.989 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.990 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.990 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.990 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.990 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.991 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.991 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.991 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.991 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762786.618187_50.1", 2022-09-09T22:59:47.992 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.992 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-13", 2022-09-09T22:59:47.992 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:33:06.618187872Z", 2022-09-09T22:59:47.992 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.992 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.993 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.993 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.993 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.993 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.994 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_JNgA-Mcbf5d_uTKjSiKsu05", 2022-09-09T22:59:47.994 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.994 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.994 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.995 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:47.995 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.995 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:47.995 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.996 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:47.996 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762786.621862_51.1", 2022-09-09T22:59:47.996 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:47.996 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-13", 2022-09-09T22:59:47.997 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:33:06.621862110Z", 2022-09-09T22:59:47.997 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:47.997 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:47.997 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:47.997 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:47.998 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.998 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:47.998 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_JNgA-Mcbf5d_uTKjSiKsu05", 2022-09-09T22:59:47.998 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:47.999 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:47.999 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:47.999 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:47.999 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.000 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.000 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.000 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.000 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762786.651673_52.1", 2022-09-09T22:59:48.001 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.001 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14", 2022-09-09T22:59:48.001 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:33:06.651673719Z", 2022-09-09T22:59:48.001 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.002 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.002 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.002 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.002 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.003 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.003 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_f0wDKO7d5qM7TS5fzfH6prz", 2022-09-09T22:59:48.003 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.003 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.003 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.004 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.004 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.004 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.004 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.005 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.005 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762786.656784_53.1", 2022-09-09T22:59:48.005 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.005 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14", 2022-09-09T22:59:48.006 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:33:06.656784091Z", 2022-09-09T22:59:48.006 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.006 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.006 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.007 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.007 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.007 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.007 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_f0wDKO7d5qM7TS5fzfH6prz", 2022-09-09T22:59:48.008 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.008 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.008 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.008 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.009 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.009 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.009 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.009 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.009 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762786.658400_54.1", 2022-09-09T22:59:48.010 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.010 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-14", 2022-09-09T22:59:48.010 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:33:06.658400465Z", 2022-09-09T22:59:48.010 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.011 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.011 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.011 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.011 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.012 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.012 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_tDp27FdXJgMOU8rSYSjQb_Y", 2022-09-09T22:59:48.012 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.012 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.013 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.013 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.013 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.013 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.014 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.014 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.014 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762786.662396_55.1", 2022-09-09T22:59:48.014 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.015 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-14", 2022-09-09T22:59:48.015 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:33:06.662396356Z", 2022-09-09T22:59:48.015 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.015 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.016 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.016 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.016 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.016 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.016 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_tDp27FdXJgMOU8rSYSjQb_Y", 2022-09-09T22:59:48.017 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.022 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.022 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.023 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.023 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.023 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.023 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.024 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.024 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762786.710171_56.1", 2022-09-09T22:59:48.024 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.024 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15", 2022-09-09T22:59:48.025 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:33:06.710171968Z", 2022-09-09T22:59:48.025 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.025 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.025 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.026 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.026 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.026 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.026 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_ogbkKIIiKYU8oWXJ_RMpS5O", 2022-09-09T22:59:48.027 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.027 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.027 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.027 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.028 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.028 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.028 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.028 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.029 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762786.713926_57.1", 2022-09-09T22:59:48.029 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.029 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15", 2022-09-09T22:59:48.030 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:33:06.713926458Z", 2022-09-09T22:59:48.030 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.030 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.030 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.031 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.031 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.031 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.031 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_ogbkKIIiKYU8oWXJ_RMpS5O", 2022-09-09T22:59:48.031 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.032 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.032 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.032 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.032 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.033 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.033 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.033 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.033 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762786.715462_58.1", 2022-09-09T22:59:48.034 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.034 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-15", 2022-09-09T22:59:48.034 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:33:06.715462548Z", 2022-09-09T22:59:48.034 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.035 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.035 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.035 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.035 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.036 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.036 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_Hk_J6cauBfq0403R0YNkLtB", 2022-09-09T22:59:48.036 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.036 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.037 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.037 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.037 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.037 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.038 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.038 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.038 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762786.718887_59.1", 2022-09-09T22:59:48.038 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.039 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-15", 2022-09-09T22:59:48.039 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:33:06.718887015Z", 2022-09-09T22:59:48.039 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.039 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.040 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.040 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.040 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.040 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.040 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_Hk_J6cauBfq0403R0YNkLtB", 2022-09-09T22:59:48.041 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.041 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.041 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.041 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.042 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.042 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.042 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.042 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.043 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.055821_61.1", 2022-09-09T22:59:48.043 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.043 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16", 2022-09-09T22:59:48.043 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.055821864Z", 2022-09-09T22:59:48.044 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.044 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.044 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.044 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.045 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.045 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.045 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_jgmlG80S-C76e7qmPCPXWwY", 2022-09-09T22:59:48.045 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.046 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.046 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.046 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.046 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.046 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.047 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.047 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.047 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.060244_62.1", 2022-09-09T22:59:48.047 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.048 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16", 2022-09-09T22:59:48.048 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.060244432Z", 2022-09-09T22:59:48.048 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.048 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.049 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.049 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.049 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.049 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.050 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_jgmlG80S-C76e7qmPCPXWwY", 2022-09-09T22:59:48.050 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.050 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.050 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.051 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.051 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.051 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.051 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.052 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.052 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.061724_63.1", 2022-09-09T22:59:48.052 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.052 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-16", 2022-09-09T22:59:48.052 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.061724454Z", 2022-09-09T22:59:48.053 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.053 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.053 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.053 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.054 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.054 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.054 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_QzVpUqaeVNRQiAA9WAn6beJ", 2022-09-09T22:59:48.054 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.055 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.055 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.055 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.055 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.056 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.056 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.056 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.056 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.065560_64.1", 2022-09-09T22:59:48.057 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.057 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-16", 2022-09-09T22:59:48.057 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.065560207Z", 2022-09-09T22:59:48.057 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.058 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.058 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.058 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.058 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.059 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.059 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_QzVpUqaeVNRQiAA9WAn6beJ", 2022-09-09T22:59:48.059 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.059 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.059 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.060 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.060 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.060 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.060 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.061 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.061 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.094558_65.1", 2022-09-09T22:59:48.061 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.061 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17", 2022-09-09T22:59:48.062 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.094558838Z", 2022-09-09T22:59:48.062 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.062 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.062 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.063 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.063 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.063 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.063 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_MtuyPaZmCzfHOcgXjri3F-V", 2022-09-09T22:59:48.064 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.064 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.064 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.064 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.065 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.065 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.065 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.065 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.066 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.099477_66.1", 2022-09-09T22:59:48.066 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.066 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17", 2022-09-09T22:59:48.066 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.099477016Z", 2022-09-09T22:59:48.067 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.067 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.067 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.067 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.068 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.068 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.068 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_MtuyPaZmCzfHOcgXjri3F-V", 2022-09-09T22:59:48.068 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.069 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.069 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.069 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.069 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.069 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.070 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.070 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.070 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.101086_67.1", 2022-09-09T22:59:48.070 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.071 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-17", 2022-09-09T22:59:48.071 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.101086117Z", 2022-09-09T22:59:48.071 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.071 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.072 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.072 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.072 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.072 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.073 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_eneLpQRdXpsBeKeIZdm9Fz7", 2022-09-09T22:59:48.073 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.073 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.073 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.074 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.074 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.074 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.074 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.075 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.075 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.105046_68.1", 2022-09-09T22:59:48.075 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.075 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-17", 2022-09-09T22:59:48.076 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.105046294Z", 2022-09-09T22:59:48.076 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.076 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.076 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.077 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.077 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.077 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.077 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_eneLpQRdXpsBeKeIZdm9Fz7", 2022-09-09T22:59:48.078 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.078 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.078 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.078 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.078 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.079 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.079 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.079 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.079 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.149665_69.1", 2022-09-09T22:59:48.080 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.080 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18", 2022-09-09T22:59:48.080 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.149665741Z", 2022-09-09T22:59:48.080 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.081 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.081 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.081 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.081 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.082 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.082 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_bCS5W1GGiGEz-gWOOFrVrp4", 2022-09-09T22:59:48.082 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.082 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.083 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.083 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.083 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.083 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.084 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.084 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.084 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.156765_70.1", 2022-09-09T22:59:48.084 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.085 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18", 2022-09-09T22:59:48.085 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.156765172Z", 2022-09-09T22:59:48.085 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.085 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.086 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.086 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.086 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.086 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.087 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_bCS5W1GGiGEz-gWOOFrVrp4", 2022-09-09T22:59:48.087 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.087 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.087 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.087 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.088 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.088 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.088 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.088 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.089 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.161880_71.1", 2022-09-09T22:59:48.089 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.089 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-18", 2022-09-09T22:59:48.089 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.161880990Z", 2022-09-09T22:59:48.090 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.090 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.090 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.090 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.091 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.091 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.091 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_vVrGNRKUHx_Nc7AgE45d0q7", 2022-09-09T22:59:48.091 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.092 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.092 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.092 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.092 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.092 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.093 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.093 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.093 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.166872_72.1", 2022-09-09T22:59:48.093 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.094 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-18", 2022-09-09T22:59:48.094 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.166872706Z", 2022-09-09T22:59:48.094 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.094 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.095 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.095 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.095 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.095 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.096 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_vVrGNRKUHx_Nc7AgE45d0q7", 2022-09-09T22:59:48.096 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.096 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.096 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.097 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.097 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.097 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.097 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.098 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.098 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.191982_73.1", 2022-09-09T22:59:48.098 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.098 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16", 2022-09-09T22:59:48.098 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.191982077Z", 2022-09-09T22:59:48.099 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.099 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.099 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_jgmlG80S-C76e7qmPCPXWwY", 2022-09-09T22:59:48.099 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.100 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.100 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.100 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_jgmlG80S-C76e7qmPCPXWwY", 2022-09-09T22:59:48.100 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.101 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.101 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.101 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.101 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.102 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.102 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.102 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.102 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.196936_74.1", 2022-09-09T22:59:48.103 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.103 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16", 2022-09-09T22:59:48.103 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.196936218Z", 2022-09-09T22:59:48.103 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.104 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.104 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_jgmlG80S-C76e7qmPCPXWwY", 2022-09-09T22:59:48.104 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.104 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.105 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.105 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_jgmlG80S-C76e7qmPCPXWwY", 2022-09-09T22:59:48.105 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.105 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.105 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.106 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.106 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.106 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.106 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.107 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.107 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.211489_75.1", 2022-09-09T22:59:48.107 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.107 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17", 2022-09-09T22:59:48.108 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.211489751Z", 2022-09-09T22:59:48.108 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.108 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.108 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_MtuyPaZmCzfHOcgXjri3F-V", 2022-09-09T22:59:48.109 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.109 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.109 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.109 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_MtuyPaZmCzfHOcgXjri3F-V", 2022-09-09T22:59:48.109 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.110 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.110 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.110 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.110 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.111 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.111 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.111 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.111 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.217670_76.1", 2022-09-09T22:59:48.112 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.112 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17", 2022-09-09T22:59:48.112 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.217670475Z", 2022-09-09T22:59:48.112 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.113 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.113 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_MtuyPaZmCzfHOcgXjri3F-V", 2022-09-09T22:59:48.113 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.113 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.114 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.114 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_MtuyPaZmCzfHOcgXjri3F-V", 2022-09-09T22:59:48.114 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.114 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.115 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.115 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.115 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.115 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.116 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.116 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.116 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.236343_77.1", 2022-09-09T22:59:48.116 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.116 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18", 2022-09-09T22:59:48.117 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.236343391Z", 2022-09-09T22:59:48.117 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.117 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.117 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_bCS5W1GGiGEz-gWOOFrVrp4", 2022-09-09T22:59:48.118 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.118 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.118 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.118 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_bCS5W1GGiGEz-gWOOFrVrp4", 2022-09-09T22:59:48.119 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.119 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.119 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.119 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.120 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.120 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.120 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.120 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.121 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.244083_78.1", 2022-09-09T22:59:48.121 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.121 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18", 2022-09-09T22:59:48.121 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.244083785Z", 2022-09-09T22:59:48.121 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.122 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.122 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_bCS5W1GGiGEz-gWOOFrVrp4", 2022-09-09T22:59:48.122 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.122 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.123 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.123 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_bCS5W1GGiGEz-gWOOFrVrp4", 2022-09-09T22:59:48.123 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.123 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.124 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.124 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.124 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.124 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.125 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.125 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.125 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763177.690963_87.1", 2022-09-09T22:59:48.125 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.126 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19", 2022-09-09T22:59:48.126 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:39:37.690963866Z", 2022-09-09T22:59:48.126 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.126 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.127 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.127 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.127 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.127 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.128 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_ePT9JnPeiTDUMUQthdKtkWA", 2022-09-09T22:59:48.128 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.128 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.128 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.128 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.129 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.129 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.129 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.129 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.130 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763177.695434_88.1", 2022-09-09T22:59:48.130 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.130 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19", 2022-09-09T22:59:48.130 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:39:37.695434017Z", 2022-09-09T22:59:48.131 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.131 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.131 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.131 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.132 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.132 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.132 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_ePT9JnPeiTDUMUQthdKtkWA", 2022-09-09T22:59:48.132 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.133 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.133 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.133 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.133 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.134 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.134 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.134 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.134 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763177.697042_89.1", 2022-09-09T22:59:48.135 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.135 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-19", 2022-09-09T22:59:48.135 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:39:37.697042999Z", 2022-09-09T22:59:48.135 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.135 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.136 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.136 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.136 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.136 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.137 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_tYLgE0kYjoh4utIhhPKouFZ", 2022-09-09T22:59:48.137 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.137 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.137 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.138 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.138 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.138 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.138 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.139 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.139 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763177.701435_90.1", 2022-09-09T22:59:48.139 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.139 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-19", 2022-09-09T22:59:48.140 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:39:37.701435888Z", 2022-09-09T22:59:48.140 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.140 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.140 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.140 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.141 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.141 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.141 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_tYLgE0kYjoh4utIhhPKouFZ", 2022-09-09T22:59:48.141 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.142 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.142 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.142 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.142 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.143 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.143 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.143 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.143 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763177.715694_91.1", 2022-09-09T22:59:48.144 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.144 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19", 2022-09-09T22:59:48.144 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:39:37.715694533Z", 2022-09-09T22:59:48.144 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.145 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.145 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_ePT9JnPeiTDUMUQthdKtkWA", 2022-09-09T22:59:48.145 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.145 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.146 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.146 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_ePT9JnPeiTDUMUQthdKtkWA", 2022-09-09T22:59:48.146 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.146 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.147 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.147 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.147 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.147 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.148 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.148 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.148 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763177.721058_92.1", 2022-09-09T22:59:48.149 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.149 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19", 2022-09-09T22:59:48.149 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:39:37.721058937Z", 2022-09-09T22:59:48.149 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.150 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.150 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_ePT9JnPeiTDUMUQthdKtkWA", 2022-09-09T22:59:48.150 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.150 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.151 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.151 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_ePT9JnPeiTDUMUQthdKtkWA", 2022-09-09T22:59:48.151 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.151 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.152 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.152 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.152 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.152 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.153 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.153 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.153 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763618.848484_94.1", 2022-09-09T22:59:48.153 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.154 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20", 2022-09-09T22:59:48.154 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:46:58.848484735Z", 2022-09-09T22:59:48.154 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.154 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.155 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.155 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.155 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.155 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.156 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:48.156 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.156 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.157 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.157 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.157 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.157 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.158 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.158 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.158 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763618.852755_95.1", 2022-09-09T22:59:48.158 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.159 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20", 2022-09-09T22:59:48.159 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:46:58.852755664Z", 2022-09-09T22:59:48.159 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.159 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.160 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.160 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.160 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.160 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.161 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:48.161 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.161 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.161 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.162 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.162 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.162 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.162 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.163 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.163 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763618.854410_96.1", 2022-09-09T22:59:48.163 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.163 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-20", 2022-09-09T22:59:48.164 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:46:58.854410052Z", 2022-09-09T22:59:48.164 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.164 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.164 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.164 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.165 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.165 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.165 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_4dW2sFlblKAnYCXSGCT7lId", 2022-09-09T22:59:48.165 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.166 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.166 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.166 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.166 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.167 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.167 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.167 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.167 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763618.858372_97.1", 2022-09-09T22:59:48.168 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.168 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-20", 2022-09-09T22:59:48.168 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:46:58.858372321Z", 2022-09-09T22:59:48.168 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.169 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.169 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.169 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.169 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.170 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.170 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_4dW2sFlblKAnYCXSGCT7lId", 2022-09-09T22:59:48.170 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.170 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.171 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.171 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.171 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.171 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.172 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.172 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.172 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763692.030172_110.1", 2022-09-09T22:59:48.172 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.173 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20", 2022-09-09T22:59:48.173 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:48:12.030172339Z", 2022-09-09T22:59:48.173 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.173 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.174 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:48.174 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.174 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.174 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.175 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:48.175 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.175 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.175 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.176 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.176 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.176 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.177 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.177 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.177 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763692.038589_111.1", 2022-09-09T22:59:48.178 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.178 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20", 2022-09-09T22:59:48.178 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:48:12.038589581Z", 2022-09-09T22:59:48.178 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.179 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.179 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:48.179 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.179 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.180 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.180 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:48.180 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.180 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.181 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.181 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.181 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.181 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.182 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.182 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.182 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763726.102012_112.1", 2022-09-09T22:59:48.182 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.183 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20", 2022-09-09T22:59:48.183 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:48:46.102012638Z", 2022-09-09T22:59:48.183 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.183 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.184 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:48.184 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.184 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.185 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.185 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:48.185 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 3 2022-09-09T22:59:48.185 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.186 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.186 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.186 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.187 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.187 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.187 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.187 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763726.109956_113.1", 2022-09-09T22:59:48.188 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.188 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20", 2022-09-09T22:59:48.188 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:48:46.109956678Z", 2022-09-09T22:59:48.189 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.189 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.189 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:48.190 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.190 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.190 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.191 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:48.191 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 3 2022-09-09T22:59:48.191 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.192 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.192 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.192 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.192 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.193 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.193 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.193 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.234301_114.1", 2022-09-09T22:59:48.193 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.194 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21", 2022-09-09T22:59:48.194 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.234301916Z", 2022-09-09T22:59:48.194 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.194 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.195 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.195 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.195 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.195 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.196 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_vO_bfm5gHdzZApgtPNMJdSt", 2022-09-09T22:59:48.196 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.196 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.196 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.197 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.197 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.197 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.197 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.198 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.198 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.239008_115.1", 2022-09-09T22:59:48.198 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.198 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21", 2022-09-09T22:59:48.199 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.239008120Z", 2022-09-09T22:59:48.199 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.199 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.199 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.200 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.200 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.200 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.200 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_vO_bfm5gHdzZApgtPNMJdSt", 2022-09-09T22:59:48.201 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.201 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.201 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.201 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.201 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.202 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.202 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.202 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.202 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.240497_116.1", 2022-09-09T22:59:48.203 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.203 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-21", 2022-09-09T22:59:48.203 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.240497138Z", 2022-09-09T22:59:48.203 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.204 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.204 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.204 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.204 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.205 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.205 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_K_Aww59oBrAI7mXNiMdGk-A", 2022-09-09T22:59:48.205 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.205 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.206 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.206 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.206 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.206 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.207 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.207 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.207 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.244134_117.1", 2022-09-09T22:59:48.207 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.208 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-21", 2022-09-09T22:59:48.208 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.244134829Z", 2022-09-09T22:59:48.208 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.208 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.209 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.209 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.209 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.209 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.210 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_K_Aww59oBrAI7mXNiMdGk-A", 2022-09-09T22:59:48.210 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.210 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.210 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.211 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.211 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.211 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.211 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.212 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.212 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.271364_118.1", 2022-09-09T22:59:48.212 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.212 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22", 2022-09-09T22:59:48.213 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.271364170Z", 2022-09-09T22:59:48.213 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.213 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.213 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.214 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.214 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.214 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.214 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_oq35luOSMiKR6aRP4lKrHCx", 2022-09-09T22:59:48.215 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.215 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.215 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.215 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.216 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.216 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.216 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.216 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.217 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.276910_119.1", 2022-09-09T22:59:48.217 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.217 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22", 2022-09-09T22:59:48.217 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.276910204Z", 2022-09-09T22:59:48.218 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.218 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.218 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.218 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.219 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.219 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.219 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_oq35luOSMiKR6aRP4lKrHCx", 2022-09-09T22:59:48.219 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.220 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.220 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.220 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.220 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.220 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.221 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.221 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.221 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.278731_120.1", 2022-09-09T22:59:48.221 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.222 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-22", 2022-09-09T22:59:48.222 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.278731955Z", 2022-09-09T22:59:48.222 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.222 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.223 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.223 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.223 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.223 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.224 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_oBwxyUPEtECOF6uEAIe2K67", 2022-09-09T22:59:48.224 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.224 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.225 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.225 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.225 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.226 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.226 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.226 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.226 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.284005_121.1", 2022-09-09T22:59:48.227 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.227 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-22", 2022-09-09T22:59:48.227 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.284005465Z", 2022-09-09T22:59:48.227 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.228 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.228 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.228 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.228 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.229 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.229 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_oBwxyUPEtECOF6uEAIe2K67", 2022-09-09T22:59:48.229 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.229 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.230 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.230 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.230 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.230 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.231 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.231 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.231 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.330455_122.1", 2022-09-09T22:59:48.231 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.232 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23", 2022-09-09T22:59:48.232 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.330455662Z", 2022-09-09T22:59:48.232 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.232 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.232 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.233 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.233 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.233 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.233 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_TBNrJnEgH9G6t0MeX057y1f", 2022-09-09T22:59:48.234 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.234 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.234 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.234 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.235 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.235 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.235 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.235 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.236 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.335107_123.1", 2022-09-09T22:59:48.236 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.236 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23", 2022-09-09T22:59:48.236 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.335107502Z", 2022-09-09T22:59:48.237 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.237 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.237 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.237 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.238 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.238 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.238 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_TBNrJnEgH9G6t0MeX057y1f", 2022-09-09T22:59:48.238 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.238 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.239 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.239 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.239 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.239 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.240 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.240 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.240 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.336855_124.1", 2022-09-09T22:59:48.240 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.241 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-23", 2022-09-09T22:59:48.241 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.336855486Z", 2022-09-09T22:59:48.241 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.241 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.242 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.242 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.242 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.242 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.243 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_DGKEfxrBasYDPbGTJt7zCFo", 2022-09-09T22:59:48.243 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.243 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.243 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.244 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.244 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.244 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.244 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.244 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.245 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.340827_125.1", 2022-09-09T22:59:48.245 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.245 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-23", 2022-09-09T22:59:48.245 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.340827648Z", 2022-09-09T22:59:48.246 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.246 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.246 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.246 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.247 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.247 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.247 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_DGKEfxrBasYDPbGTJt7zCFo", 2022-09-09T22:59:48.247 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.248 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.248 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.248 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.248 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.249 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.249 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.249 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.249 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.368888_126.1", 2022-09-09T22:59:48.250 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.250 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21", 2022-09-09T22:59:48.250 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.368888518Z", 2022-09-09T22:59:48.250 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.251 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.251 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_vO_bfm5gHdzZApgtPNMJdSt", 2022-09-09T22:59:48.251 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.251 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.251 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.252 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_vO_bfm5gHdzZApgtPNMJdSt", 2022-09-09T22:59:48.252 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.252 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.252 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.253 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.253 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.253 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.253 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.254 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.254 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.374441_127.1", 2022-09-09T22:59:48.254 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.254 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21", 2022-09-09T22:59:48.255 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.374441110Z", 2022-09-09T22:59:48.255 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.255 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.255 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_vO_bfm5gHdzZApgtPNMJdSt", 2022-09-09T22:59:48.256 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.256 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.256 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.256 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_vO_bfm5gHdzZApgtPNMJdSt", 2022-09-09T22:59:48.256 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.257 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.257 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.257 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.257 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.258 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.258 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.258 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.258 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.389455_128.1", 2022-09-09T22:59:48.259 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.259 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22", 2022-09-09T22:59:48.259 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.389455212Z", 2022-09-09T22:59:48.259 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.260 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.260 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_oq35luOSMiKR6aRP4lKrHCx", 2022-09-09T22:59:48.260 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.260 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.261 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.261 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_oq35luOSMiKR6aRP4lKrHCx", 2022-09-09T22:59:48.261 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.261 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.262 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.262 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.262 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.262 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.263 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.263 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.263 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.396062_129.1", 2022-09-09T22:59:48.263 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.264 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22", 2022-09-09T22:59:48.264 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.396062782Z", 2022-09-09T22:59:48.264 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.264 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.264 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_oq35luOSMiKR6aRP4lKrHCx", 2022-09-09T22:59:48.265 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.265 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.265 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.265 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_oq35luOSMiKR6aRP4lKrHCx", 2022-09-09T22:59:48.266 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.266 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.266 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.266 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.267 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.267 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.267 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.267 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.268 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.417475_130.1", 2022-09-09T22:59:48.268 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.268 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23", 2022-09-09T22:59:48.268 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.417475123Z", 2022-09-09T22:59:48.269 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.269 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.269 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_TBNrJnEgH9G6t0MeX057y1f", 2022-09-09T22:59:48.270 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.270 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.270 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.270 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_TBNrJnEgH9G6t0MeX057y1f", 2022-09-09T22:59:48.271 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.271 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.271 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.271 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.271 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.272 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.272 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.272 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.272 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.426933_131.1", 2022-09-09T22:59:48.273 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.273 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23", 2022-09-09T22:59:48.273 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.426933442Z", 2022-09-09T22:59:48.274 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.274 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.274 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_TBNrJnEgH9G6t0MeX057y1f", 2022-09-09T22:59:48.275 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.275 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.275 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.276 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_TBNrJnEgH9G6t0MeX057y1f", 2022-09-09T22:59:48.276 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.276 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.276 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.277 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.277 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.277 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.277 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.277 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.278 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.243136_133.1", 2022-09-09T22:59:48.278 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.278 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24", 2022-09-09T22:59:48.278 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.243136650Z", 2022-09-09T22:59:48.279 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.279 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.279 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.279 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.280 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.280 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.280 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:48.280 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.281 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.281 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.281 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.281 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.282 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.282 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.282 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.282 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.247428_134.1", 2022-09-09T22:59:48.283 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.283 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24", 2022-09-09T22:59:48.283 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.247428396Z", 2022-09-09T22:59:48.283 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.284 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.284 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.284 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.284 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.284 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.285 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:48.285 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.285 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.285 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.286 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.286 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.286 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.286 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.287 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.287 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.249044_135.1", 2022-09-09T22:59:48.287 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.287 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-24", 2022-09-09T22:59:48.288 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.249044082Z", 2022-09-09T22:59:48.288 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.288 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.288 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.289 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.289 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.289 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.289 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_GPbPaiqkbVjC5D9wQQQP2jZ", 2022-09-09T22:59:48.290 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.290 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.290 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.290 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.291 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.291 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.291 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.291 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.291 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.253611_136.1", 2022-09-09T22:59:48.292 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.292 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-24", 2022-09-09T22:59:48.292 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.253611226Z", 2022-09-09T22:59:48.292 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.293 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.293 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.293 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.293 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.294 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.294 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_GPbPaiqkbVjC5D9wQQQP2jZ", 2022-09-09T22:59:48.294 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.294 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.295 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.295 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.295 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.295 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.296 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.296 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.296 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.282622_137.1", 2022-09-09T22:59:48.296 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.297 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25", 2022-09-09T22:59:48.297 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.282622692Z", 2022-09-09T22:59:48.297 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.297 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.297 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.298 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.298 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.298 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.298 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:48.299 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.299 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.299 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.299 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.300 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.300 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.300 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.300 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.301 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.287087_138.1", 2022-09-09T22:59:48.301 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.301 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25", 2022-09-09T22:59:48.301 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.287087929Z", 2022-09-09T22:59:48.302 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.302 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.302 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.302 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.303 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.303 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.303 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:48.303 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.304 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.304 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.304 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.304 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.305 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.305 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.305 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.306 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.288605_139.1", 2022-09-09T22:59:48.306 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.306 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-25", 2022-09-09T22:59:48.306 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.288605630Z", 2022-09-09T22:59:48.307 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.307 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.307 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.307 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.308 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.308 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.308 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_ZPxDZ2lg7jpxqasfBs-kwky", 2022-09-09T22:59:48.308 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.309 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.309 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.309 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.309 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.310 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.310 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.310 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.310 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.292651_140.1", 2022-09-09T22:59:48.311 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.311 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-25", 2022-09-09T22:59:48.311 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.292651882Z", 2022-09-09T22:59:48.311 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.312 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.312 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.312 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.312 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.312 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.313 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_ZPxDZ2lg7jpxqasfBs-kwky", 2022-09-09T22:59:48.313 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.313 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.313 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.314 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.314 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.314 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.314 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.315 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.315 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.347788_141.1", 2022-09-09T22:59:48.315 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.315 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26", 2022-09-09T22:59:48.316 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.347788063Z", 2022-09-09T22:59:48.316 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.316 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.316 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.317 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.317 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.317 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.317 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:48.317 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.318 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.318 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.318 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.319 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.319 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.319 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.319 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.319 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.352949_142.1", 2022-09-09T22:59:48.320 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.320 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26", 2022-09-09T22:59:48.320 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.352949747Z", 2022-09-09T22:59:48.320 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.321 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.321 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.321 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.321 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.322 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.322 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:48.322 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.322 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.323 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.323 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.323 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.323 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.324 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.324 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.324 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.354746_143.1", 2022-09-09T22:59:48.324 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.324 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-26", 2022-09-09T22:59:48.325 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.354746994Z", 2022-09-09T22:59:48.325 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.325 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.325 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.326 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.326 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.326 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.326 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_sgAtkuvW-X_3Moy5BHrhaBp", 2022-09-09T22:59:48.327 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.327 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.327 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.327 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.328 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.328 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.328 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.328 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.329 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.358613_144.1", 2022-09-09T22:59:48.329 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.329 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-26", 2022-09-09T22:59:48.329 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.358613174Z", 2022-09-09T22:59:48.329 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.330 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.330 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.330 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.330 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.331 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.331 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_sgAtkuvW-X_3Moy5BHrhaBp", 2022-09-09T22:59:48.331 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.331 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.332 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.332 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.332 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.332 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.333 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.333 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.333 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.386362_145.1", 2022-09-09T22:59:48.333 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.334 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24", 2022-09-09T22:59:48.334 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.386362667Z", 2022-09-09T22:59:48.334 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.334 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.335 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:48.335 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.335 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.335 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.336 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:48.336 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.336 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.337 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.337 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.337 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.337 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.338 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.338 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.338 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.392952_146.1", 2022-09-09T22:59:48.339 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.339 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24", 2022-09-09T22:59:48.339 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.392952544Z", 2022-09-09T22:59:48.340 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.340 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.340 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:48.341 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.341 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.342 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.342 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:48.342 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.343 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.343 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.344 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.344 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.344 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.344 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.345 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.345 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.405357_147.1", 2022-09-09T22:59:48.345 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.345 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24", 2022-09-09T22:59:48.346 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.405357667Z", 2022-09-09T22:59:48.346 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.346 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.346 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:48.347 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.347 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.347 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.348 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:48.348 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 3 2022-09-09T22:59:48.348 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.348 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.349 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.349 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.349 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.349 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.350 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.350 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.413164_148.1", 2022-09-09T22:59:48.350 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.350 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24", 2022-09-09T22:59:48.351 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.413164542Z", 2022-09-09T22:59:48.351 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.351 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.352 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:48.352 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.352 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.352 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.353 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:48.353 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 3 2022-09-09T22:59:48.353 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.353 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.354 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.354 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.354 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.354 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.355 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.355 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.428806_149.1", 2022-09-09T22:59:48.355 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.355 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25", 2022-09-09T22:59:48.356 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.428806301Z", 2022-09-09T22:59:48.356 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.356 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.356 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:48.356 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.357 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.357 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.357 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:48.357 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.358 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.358 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.358 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.358 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.359 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.359 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.359 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.359 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.436953_150.1", 2022-09-09T22:59:48.360 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.360 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25", 2022-09-09T22:59:48.360 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.436953749Z", 2022-09-09T22:59:48.360 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.361 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.361 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:48.361 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.361 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.362 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.362 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:48.362 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.362 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.362 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.363 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.363 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.363 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.363 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.364 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.364 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.460100_151.1", 2022-09-09T22:59:48.364 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.364 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25", 2022-09-09T22:59:48.365 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.460100101Z", 2022-09-09T22:59:48.365 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.365 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.365 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:48.366 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.366 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.366 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.366 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:48.367 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 3 2022-09-09T22:59:48.367 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.367 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.367 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.367 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.368 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.368 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.368 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.368 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.471650_152.1", 2022-09-09T22:59:48.369 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.369 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25", 2022-09-09T22:59:48.369 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.471650692Z", 2022-09-09T22:59:48.369 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.370 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.370 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:48.370 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.370 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.371 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.371 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:48.371 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 3 2022-09-09T22:59:48.371 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.372 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.372 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.372 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.372 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.373 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.373 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.373 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.513547_153.1", 2022-09-09T22:59:48.373 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.373 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26", 2022-09-09T22:59:48.374 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.513547080Z", 2022-09-09T22:59:48.374 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.374 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.374 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:48.375 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.375 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.375 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.375 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:48.376 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.376 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.376 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.376 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.377 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.377 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.377 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.377 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.378 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.520151_154.1", 2022-09-09T22:59:48.378 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.378 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26", 2022-09-09T22:59:48.378 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.520151282Z", 2022-09-09T22:59:48.379 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.379 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.379 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:48.379 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.380 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.380 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.380 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:48.380 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.380 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.381 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.381 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.381 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.381 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.382 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.382 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.382 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.543475_155.1", 2022-09-09T22:59:48.383 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.383 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26", 2022-09-09T22:59:48.383 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.543475276Z", 2022-09-09T22:59:48.383 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.384 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.384 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:48.384 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.384 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.385 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.385 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:48.385 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 3 2022-09-09T22:59:48.385 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.386 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.386 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.386 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.387 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.387 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.387 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.387 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.550710_156.1", 2022-09-09T22:59:48.387 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.388 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26", 2022-09-09T22:59:48.388 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.550710511Z", 2022-09-09T22:59:48.388 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.388 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.389 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:48.389 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.389 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.389 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.390 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:48.390 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 3 2022-09-09T22:59:48.390 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.390 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.391 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.391 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.391 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.391 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.392 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.392 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.587061_165.1", 2022-09-09T22:59:48.392 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.392 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-27:4b05608c-7197-4e9b-bd29-7989607fa277.4354.27", 2022-09-09T22:59:48.393 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.587061072Z", 2022-09-09T22:59:48.393 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.393 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.393 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.394 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.394 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.394 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.394 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_idGRpIZD_cBqHmSr0C8anAE", 2022-09-09T22:59:48.395 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.395 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.395 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.395 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.395 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.396 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.396 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.396 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.396 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.591517_166.1", 2022-09-09T22:59:48.397 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.397 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-27:4b05608c-7197-4e9b-bd29-7989607fa277.4354.27", 2022-09-09T22:59:48.397 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.591517503Z", 2022-09-09T22:59:48.397 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.398 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.398 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.398 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.398 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.399 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.399 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_idGRpIZD_cBqHmSr0C8anAE", 2022-09-09T22:59:48.399 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.399 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.399 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.400 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.400 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.400 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.400 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.401 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.401 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.593291_167.1", 2022-09-09T22:59:48.401 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.401 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-27", 2022-09-09T22:59:48.402 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.593291391Z", 2022-09-09T22:59:48.402 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.402 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.402 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.403 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.403 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.403 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.403 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_fil3WVqAmCZtFEsYnU9kPha", 2022-09-09T22:59:48.404 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.404 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.404 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.404 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.404 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.405 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.405 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.405 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.405 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.597347_168.1", 2022-09-09T22:59:48.406 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.406 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-27", 2022-09-09T22:59:48.406 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.597347657Z", 2022-09-09T22:59:48.406 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.407 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.407 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.407 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.407 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.408 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.408 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_fil3WVqAmCZtFEsYnU9kPha", 2022-09-09T22:59:48.408 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.408 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.409 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.409 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.409 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.409 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.410 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.410 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.410 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.629370_169.1", 2022-09-09T22:59:48.410 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.411 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-28:4b05608c-7197-4e9b-bd29-7989607fa277.4354.28", 2022-09-09T22:59:48.411 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.629370653Z", 2022-09-09T22:59:48.411 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.411 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.412 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.412 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.412 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.412 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.413 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_hWVXDIs2ffmJdaY5i2EUA49", 2022-09-09T22:59:48.413 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.413 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.413 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.414 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.414 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.414 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.414 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.415 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.415 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.634323_170.1", 2022-09-09T22:59:48.415 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.415 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-28:4b05608c-7197-4e9b-bd29-7989607fa277.4354.28", 2022-09-09T22:59:48.416 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.634323436Z", 2022-09-09T22:59:48.416 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.416 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.416 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.417 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.417 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.417 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.417 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_hWVXDIs2ffmJdaY5i2EUA49", 2022-09-09T22:59:48.418 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.418 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.418 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.418 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.419 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.419 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.419 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.419 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.420 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.636142_171.1", 2022-09-09T22:59:48.420 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.420 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-28", 2022-09-09T22:59:48.420 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.636142661Z", 2022-09-09T22:59:48.421 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.421 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.421 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.421 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.421 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.422 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.422 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_Rr2OoVgGeS8W2Ih7H5QimUB", 2022-09-09T22:59:48.422 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.422 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.423 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.423 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.423 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.423 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.424 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.424 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.424 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.640366_172.1", 2022-09-09T22:59:48.424 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.425 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-28", 2022-09-09T22:59:48.425 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.640366460Z", 2022-09-09T22:59:48.425 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.425 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.426 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.426 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.426 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.426 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.427 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_Rr2OoVgGeS8W2Ih7H5QimUB", 2022-09-09T22:59:48.427 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.427 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.427 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.428 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.428 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.428 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.428 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.428 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.429 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.691642_173.1", 2022-09-09T22:59:48.429 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.429 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-29:4b05608c-7197-4e9b-bd29-7989607fa277.4354.29", 2022-09-09T22:59:48.429 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.691642561Z", 2022-09-09T22:59:48.430 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.430 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.431 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.431 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.432 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.432 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.433 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_c7Bu5fk9awm3ApVCIJ8jMq6", 2022-09-09T22:59:48.433 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.434 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.434 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.435 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.435 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.436 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.436 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.437 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.437 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.695580_174.1", 2022-09-09T22:59:48.438 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.438 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-29:4b05608c-7197-4e9b-bd29-7989607fa277.4354.29", 2022-09-09T22:59:48.439 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.695580156Z", 2022-09-09T22:59:48.440 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.440 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.441 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.441 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.442 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.442 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.442 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_c7Bu5fk9awm3ApVCIJ8jMq6", 2022-09-09T22:59:48.443 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.443 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.444 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.444 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.445 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.445 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.445 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.446 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.446 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.696997_175.1", 2022-09-09T22:59:48.447 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.447 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-29", 2022-09-09T22:59:48.448 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.696997195Z", 2022-09-09T22:59:48.448 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.448 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.449 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.449 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.449 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.449 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.450 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_oMWT857Kx25Jy8rmeTqaRTI", 2022-09-09T22:59:48.450 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.450 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.451 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.451 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.451 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.451 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.452 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.452 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.452 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.701289_176.1", 2022-09-09T22:59:48.452 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.453 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-29", 2022-09-09T22:59:48.453 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.701289662Z", 2022-09-09T22:59:48.453 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.453 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.454 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.454 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.454 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.454 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.455 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_oMWT857Kx25Jy8rmeTqaRTI", 2022-09-09T22:59:48.455 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.455 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.456 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.456 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.456 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.457 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.457 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.458 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.458 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.733662_177.1", 2022-09-09T22:59:48.459 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.459 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-27:4b05608c-7197-4e9b-bd29-7989607fa277.4354.27", 2022-09-09T22:59:48.460 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.733662223Z", 2022-09-09T22:59:48.460 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.461 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.461 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_idGRpIZD_cBqHmSr0C8anAE", 2022-09-09T22:59:48.462 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.462 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.463 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.463 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_idGRpIZD_cBqHmSr0C8anAE", 2022-09-09T22:59:48.463 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.463 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.464 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.464 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.464 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.464 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.465 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.465 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.465 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.739796_178.1", 2022-09-09T22:59:48.465 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.466 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-27:4b05608c-7197-4e9b-bd29-7989607fa277.4354.27", 2022-09-09T22:59:48.466 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.739796617Z", 2022-09-09T22:59:48.466 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.467 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.467 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_idGRpIZD_cBqHmSr0C8anAE", 2022-09-09T22:59:48.467 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.468 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.468 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.468 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_idGRpIZD_cBqHmSr0C8anAE", 2022-09-09T22:59:48.468 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.469 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.469 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.469 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.469 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.470 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.470 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.470 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.471 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.815014_179.1", 2022-09-09T22:59:48.471 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.471 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-28:4b05608c-7197-4e9b-bd29-7989607fa277.4354.28", 2022-09-09T22:59:48.472 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.815014271Z", 2022-09-09T22:59:48.472 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.472 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.472 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_hWVXDIs2ffmJdaY5i2EUA49", 2022-09-09T22:59:48.473 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.473 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.473 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.474 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_hWVXDIs2ffmJdaY5i2EUA49", 2022-09-09T22:59:48.474 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.475 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.475 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.476 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.476 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.476 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.477 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.477 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.478 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.823504_180.1", 2022-09-09T22:59:48.478 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.479 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-28:4b05608c-7197-4e9b-bd29-7989607fa277.4354.28", 2022-09-09T22:59:48.479 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.823504725Z", 2022-09-09T22:59:48.479 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.480 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.480 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_hWVXDIs2ffmJdaY5i2EUA49", 2022-09-09T22:59:48.481 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.481 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.481 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.481 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_hWVXDIs2ffmJdaY5i2EUA49", 2022-09-09T22:59:48.482 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.482 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.482 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.483 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.483 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.484 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.484 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.484 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.485 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.861162_181.1", 2022-09-09T22:59:48.485 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.486 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-29:4b05608c-7197-4e9b-bd29-7989607fa277.4354.29", 2022-09-09T22:59:48.486 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.861162453Z", 2022-09-09T22:59:48.486 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.487 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.487 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_c7Bu5fk9awm3ApVCIJ8jMq6", 2022-09-09T22:59:48.488 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.488 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.488 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.489 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_c7Bu5fk9awm3ApVCIJ8jMq6", 2022-09-09T22:59:48.489 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.490 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.490 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.490 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.491 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.491 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.491 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.491 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.492 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.868518_182.1", 2022-09-09T22:59:48.492 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.492 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-29:4b05608c-7197-4e9b-bd29-7989607fa277.4354.29", 2022-09-09T22:59:48.492 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.868518409Z", 2022-09-09T22:59:48.493 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.493 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.493 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_c7Bu5fk9awm3ApVCIJ8jMq6", 2022-09-09T22:59:48.493 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.494 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.494 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.494 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_c7Bu5fk9awm3ApVCIJ8jMq6", 2022-09-09T22:59:48.494 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.495 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.495 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.495 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.495 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.496 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.496 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.496 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.496 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.905416_183.1", 2022-09-09T22:59:48.497 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.497 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-30:4b05608c-7197-4e9b-bd29-7989607fa277.4354.30", 2022-09-09T22:59:48.497 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.905416190Z", 2022-09-09T22:59:48.497 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.498 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.498 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.498 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.498 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.499 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.499 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xuD0Sb75U_5LCkKCpVp22OC", 2022-09-09T22:59:48.499 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.499 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.500 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.500 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.500 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.500 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.501 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.501 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.501 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.909698_184.1", 2022-09-09T22:59:48.501 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.502 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-30:4b05608c-7197-4e9b-bd29-7989607fa277.4354.30", 2022-09-09T22:59:48.502 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.909698021Z", 2022-09-09T22:59:48.502 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.502 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.503 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.503 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.503 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.503 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.504 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xuD0Sb75U_5LCkKCpVp22OC", 2022-09-09T22:59:48.504 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.504 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.505 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.505 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.505 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.505 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.505 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.506 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.506 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.911381_185.1", 2022-09-09T22:59:48.506 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.507 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-30", 2022-09-09T22:59:48.507 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.911381624Z", 2022-09-09T22:59:48.507 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.507 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.507 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.508 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.508 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.508 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.508 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_rADCtMkJa27KS0fuztGkOJ4", 2022-09-09T22:59:48.509 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.509 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.509 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.509 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.510 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.510 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.510 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.510 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.511 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.915543_186.1", 2022-09-09T22:59:48.511 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.511 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-30", 2022-09-09T22:59:48.511 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.915543203Z", 2022-09-09T22:59:48.512 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.512 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.512 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.512 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.513 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.513 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.513 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_rADCtMkJa27KS0fuztGkOJ4", 2022-09-09T22:59:48.513 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.514 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.514 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.514 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.514 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.515 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.515 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.516 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.516 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.957457_187.1", 2022-09-09T22:59:48.516 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.516 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-31:4b05608c-7197-4e9b-bd29-7989607fa277.4354.31", 2022-09-09T22:59:48.517 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.957457179Z", 2022-09-09T22:59:48.517 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.518 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.518 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.518 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.518 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.519 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.519 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_RG9oYzLzZKilowVQDrZqfZp", 2022-09-09T22:59:48.519 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.519 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.520 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.520 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.520 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.520 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.521 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.521 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.521 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.961791_188.1", 2022-09-09T22:59:48.521 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.522 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-31:4b05608c-7197-4e9b-bd29-7989607fa277.4354.31", 2022-09-09T22:59:48.522 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.961791758Z", 2022-09-09T22:59:48.522 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.522 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.523 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.523 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.523 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.523 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.524 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_RG9oYzLzZKilowVQDrZqfZp", 2022-09-09T22:59:48.524 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.524 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.524 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.525 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.525 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.525 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.525 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.526 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.526 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.963310_189.1", 2022-09-09T22:59:48.526 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.526 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-31", 2022-09-09T22:59:48.527 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.963310744Z", 2022-09-09T22:59:48.527 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.527 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.527 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.528 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.528 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.528 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.528 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_u7CGf9AlYIHVBDPwmE_8lDH", 2022-09-09T22:59:48.529 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.529 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.529 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.529 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.530 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.530 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.530 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.530 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.531 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.966754_190.1", 2022-09-09T22:59:48.531 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.531 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-31", 2022-09-09T22:59:48.531 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.966754250Z", 2022-09-09T22:59:48.532 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.532 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.532 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.532 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.533 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.533 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.533 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_u7CGf9AlYIHVBDPwmE_8lDH", 2022-09-09T22:59:48.533 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.534 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.534 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.534 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.534 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.535 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.535 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.535 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.535 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.029534_191.1", 2022-09-09T22:59:48.536 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.536 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-32:4b05608c-7197-4e9b-bd29-7989607fa277.4354.32", 2022-09-09T22:59:48.536 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.029534998Z", 2022-09-09T22:59:48.536 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.537 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.537 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.537 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.537 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.538 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.538 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_Uke9qkxor72jlh3_r-i6E2w", 2022-09-09T22:59:48.538 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.539 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.539 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.539 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.539 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.540 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.540 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.540 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.540 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.036126_192.1", 2022-09-09T22:59:48.541 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.541 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-32:4b05608c-7197-4e9b-bd29-7989607fa277.4354.32", 2022-09-09T22:59:48.541 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.036126389Z", 2022-09-09T22:59:48.541 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.542 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.542 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.542 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.542 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.543 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.543 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_Uke9qkxor72jlh3_r-i6E2w", 2022-09-09T22:59:48.543 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.543 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.544 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.544 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.544 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.544 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.545 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.545 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.545 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.040740_193.1", 2022-09-09T22:59:48.545 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.546 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-32", 2022-09-09T22:59:48.546 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.040740810Z", 2022-09-09T22:59:48.546 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.546 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.547 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.547 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.547 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.547 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.548 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_SYdiOqvbUDZ5DpIXlyitzm5", 2022-09-09T22:59:48.548 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.548 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.548 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.549 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.549 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.549 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.549 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.550 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.550 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.045233_194.1", 2022-09-09T22:59:48.550 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.550 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-32", 2022-09-09T22:59:48.551 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.045233474Z", 2022-09-09T22:59:48.551 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.551 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.551 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.552 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.552 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.552 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.552 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_SYdiOqvbUDZ5DpIXlyitzm5", 2022-09-09T22:59:48.553 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.553 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.553 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.554 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.554 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.554 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.554 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.555 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.555 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.086181_195.1", 2022-09-09T22:59:48.555 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.555 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-30:4b05608c-7197-4e9b-bd29-7989607fa277.4354.30", 2022-09-09T22:59:48.556 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.086181840Z", 2022-09-09T22:59:48.556 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.556 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.556 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xuD0Sb75U_5LCkKCpVp22OC", 2022-09-09T22:59:48.557 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.557 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.557 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.557 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xuD0Sb75U_5LCkKCpVp22OC", 2022-09-09T22:59:48.558 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.558 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.558 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.559 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.559 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.559 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.559 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.560 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.560 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.092819_196.1", 2022-09-09T22:59:48.560 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.560 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-30:4b05608c-7197-4e9b-bd29-7989607fa277.4354.30", 2022-09-09T22:59:48.561 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.092819829Z", 2022-09-09T22:59:48.561 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.561 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.561 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xuD0Sb75U_5LCkKCpVp22OC", 2022-09-09T22:59:48.562 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.562 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.562 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.562 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xuD0Sb75U_5LCkKCpVp22OC", 2022-09-09T22:59:48.563 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.563 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.563 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.563 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.564 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.564 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.564 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.564 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.565 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.131992_197.1", 2022-09-09T22:59:48.565 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.565 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-31:4b05608c-7197-4e9b-bd29-7989607fa277.4354.31", 2022-09-09T22:59:48.565 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.131992122Z", 2022-09-09T22:59:48.566 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.566 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.566 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_RG9oYzLzZKilowVQDrZqfZp", 2022-09-09T22:59:48.567 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.567 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.567 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.567 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_RG9oYzLzZKilowVQDrZqfZp", 2022-09-09T22:59:48.567 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.568 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.568 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.568 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.568 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.569 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.569 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.569 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.569 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.140406_198.1", 2022-09-09T22:59:48.570 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.570 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-31:4b05608c-7197-4e9b-bd29-7989607fa277.4354.31", 2022-09-09T22:59:48.570 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.140406171Z", 2022-09-09T22:59:48.571 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.571 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.571 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_RG9oYzLzZKilowVQDrZqfZp", 2022-09-09T22:59:48.571 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.572 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.572 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.572 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_RG9oYzLzZKilowVQDrZqfZp", 2022-09-09T22:59:48.572 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.573 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.573 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.573 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.573 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.574 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.574 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.574 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.574 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.234470_199.1", 2022-09-09T22:59:48.575 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.575 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-32:4b05608c-7197-4e9b-bd29-7989607fa277.4354.32", 2022-09-09T22:59:48.575 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.234470787Z", 2022-09-09T22:59:48.575 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.576 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.576 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_Uke9qkxor72jlh3_r-i6E2w", 2022-09-09T22:59:48.576 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.576 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.577 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.577 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_Uke9qkxor72jlh3_r-i6E2w", 2022-09-09T22:59:48.577 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.577 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.578 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.578 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.578 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.578 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.579 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.579 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.579 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.242680_200.1", 2022-09-09T22:59:48.579 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.580 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-32:4b05608c-7197-4e9b-bd29-7989607fa277.4354.32", 2022-09-09T22:59:48.580 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.242680646Z", 2022-09-09T22:59:48.580 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.580 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.581 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_Uke9qkxor72jlh3_r-i6E2w", 2022-09-09T22:59:48.581 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.581 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.581 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.582 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_Uke9qkxor72jlh3_r-i6E2w", 2022-09-09T22:59:48.582 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.582 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.582 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.583 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.583 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.583 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.583 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.584 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.584 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.342621_201.1", 2022-09-09T22:59:48.584 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.584 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-33:4b05608c-7197-4e9b-bd29-7989607fa277.4354.33", 2022-09-09T22:59:48.585 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.342621802Z", 2022-09-09T22:59:48.585 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.585 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.585 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.586 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.586 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.586 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.586 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xQnGVMAoouaLAz4qR87BBiu", 2022-09-09T22:59:48.587 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.587 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.587 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.587 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.588 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.588 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.588 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.588 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.589 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.348018_202.1", 2022-09-09T22:59:48.589 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.589 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-33:4b05608c-7197-4e9b-bd29-7989607fa277.4354.33", 2022-09-09T22:59:48.589 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.348018686Z", 2022-09-09T22:59:48.590 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.590 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.590 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.590 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.591 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.591 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.591 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xQnGVMAoouaLAz4qR87BBiu", 2022-09-09T22:59:48.591 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.592 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.592 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.592 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.592 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.593 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.593 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.593 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.593 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.352316_203.1", 2022-09-09T22:59:48.594 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.594 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-33", 2022-09-09T22:59:48.594 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.352316326Z", 2022-09-09T22:59:48.594 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.595 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.595 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.595 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.595 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.596 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.596 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_py9xB0upsDT3rp1yJKL0fW7", 2022-09-09T22:59:48.596 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.596 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.597 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.597 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.597 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.597 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.598 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.598 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.598 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.367301_204.1", 2022-09-09T22:59:48.598 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.599 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-33", 2022-09-09T22:59:48.599 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.367301625Z", 2022-09-09T22:59:48.599 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.600 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.600 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.600 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.600 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.601 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.601 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_py9xB0upsDT3rp1yJKL0fW7", 2022-09-09T22:59:48.601 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.601 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.602 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.602 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.602 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.602 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.603 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.603 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.603 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.433156_205.1", 2022-09-09T22:59:48.603 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.604 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-34:4b05608c-7197-4e9b-bd29-7989607fa277.4354.34", 2022-09-09T22:59:48.604 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.433156697Z", 2022-09-09T22:59:48.604 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.604 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.605 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.605 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.605 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.605 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.606 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_4_jQEFJZa-xniCNap3y5x4k", 2022-09-09T22:59:48.606 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.606 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.606 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.607 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.607 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.607 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.607 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.608 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.608 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.438238_206.1", 2022-09-09T22:59:48.608 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.608 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-34:4b05608c-7197-4e9b-bd29-7989607fa277.4354.34", 2022-09-09T22:59:48.609 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.438238112Z", 2022-09-09T22:59:48.609 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.609 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.609 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.610 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.610 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.610 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.610 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_4_jQEFJZa-xniCNap3y5x4k", 2022-09-09T22:59:48.611 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.611 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.611 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.611 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.612 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.612 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.612 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.612 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.613 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.439911_207.1", 2022-09-09T22:59:48.613 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.613 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-34", 2022-09-09T22:59:48.613 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.439911797Z", 2022-09-09T22:59:48.614 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.614 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.614 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.614 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.615 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.615 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.615 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_i4pt2ScTLCpaSy-sTWzcDsg", 2022-09-09T22:59:48.615 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.616 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.616 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.616 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.616 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.617 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.617 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.617 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.618 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.444301_208.1", 2022-09-09T22:59:48.618 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.618 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-34", 2022-09-09T22:59:48.618 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.444301272Z", 2022-09-09T22:59:48.619 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.619 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.619 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.619 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.620 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.620 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.620 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_i4pt2ScTLCpaSy-sTWzcDsg", 2022-09-09T22:59:48.620 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.621 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.621 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.621 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.621 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.622 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.622 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.622 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.622 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.490350_209.1", 2022-09-09T22:59:48.623 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.623 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-35:4b05608c-7197-4e9b-bd29-7989607fa277.4354.35", 2022-09-09T22:59:48.623 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.490350184Z", 2022-09-09T22:59:48.623 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.624 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.624 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.624 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.624 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.625 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.625 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xarcxlcdFlngGuAvxUBGTlW", 2022-09-09T22:59:48.625 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.625 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.626 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.626 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.626 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.626 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.627 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.627 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.627 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.495064_210.1", 2022-09-09T22:59:48.627 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.628 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-35:4b05608c-7197-4e9b-bd29-7989607fa277.4354.35", 2022-09-09T22:59:48.628 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.495064159Z", 2022-09-09T22:59:48.628 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.628 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.629 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.629 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.629 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.630 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.630 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xarcxlcdFlngGuAvxUBGTlW", 2022-09-09T22:59:48.630 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.631 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.631 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.631 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.631 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.632 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.632 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.632 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.632 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.496525_211.1", 2022-09-09T22:59:48.633 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.633 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-35", 2022-09-09T22:59:48.633 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.496525235Z", 2022-09-09T22:59:48.633 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.634 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.634 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.634 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.635 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.635 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.635 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_b22QH41XyKk67qZyODLiaOx", 2022-09-09T22:59:48.635 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.636 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.636 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.636 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.636 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.637 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.637 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.637 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.637 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.500218_212.1", 2022-09-09T22:59:48.638 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.638 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-35", 2022-09-09T22:59:48.638 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.500218886Z", 2022-09-09T22:59:48.638 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.639 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.639 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.639 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.639 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.640 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.640 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_b22QH41XyKk67qZyODLiaOx", 2022-09-09T22:59:48.640 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.640 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.641 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.641 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.641 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.641 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.642 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.642 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.642 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.527104_213.1", 2022-09-09T22:59:48.642 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.643 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-33:4b05608c-7197-4e9b-bd29-7989607fa277.4354.33", 2022-09-09T22:59:48.643 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.527104670Z", 2022-09-09T22:59:48.643 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.643 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.643 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xQnGVMAoouaLAz4qR87BBiu", 2022-09-09T22:59:48.644 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.644 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.644 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.645 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xQnGVMAoouaLAz4qR87BBiu", 2022-09-09T22:59:48.645 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.645 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.645 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.645 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.646 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.646 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.646 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.646 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.647 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.533014_214.1", 2022-09-09T22:59:48.647 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.647 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-33:4b05608c-7197-4e9b-bd29-7989607fa277.4354.33", 2022-09-09T22:59:48.647 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.533014771Z", 2022-09-09T22:59:48.648 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.648 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.648 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xQnGVMAoouaLAz4qR87BBiu", 2022-09-09T22:59:48.648 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.649 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.649 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.649 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xQnGVMAoouaLAz4qR87BBiu", 2022-09-09T22:59:48.649 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.650 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.650 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.650 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.650 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.651 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.651 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.651 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.651 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763989.558518_215.1", 2022-09-09T22:59:48.652 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.652 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-34:4b05608c-7197-4e9b-bd29-7989607fa277.4354.34", 2022-09-09T22:59:48.652 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:09.558518140Z", 2022-09-09T22:59:48.652 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.653 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.653 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_4_jQEFJZa-xniCNap3y5x4k", 2022-09-09T22:59:48.653 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.653 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.654 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.654 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_4_jQEFJZa-xniCNap3y5x4k", 2022-09-09T22:59:48.654 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.654 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.655 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.655 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.655 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.655 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.656 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.656 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.656 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763989.567553_216.1", 2022-09-09T22:59:48.656 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.657 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-34:4b05608c-7197-4e9b-bd29-7989607fa277.4354.34", 2022-09-09T22:59:48.657 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:09.567553622Z", 2022-09-09T22:59:48.657 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.658 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.658 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_4_jQEFJZa-xniCNap3y5x4k", 2022-09-09T22:59:48.658 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.658 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.659 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.659 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_4_jQEFJZa-xniCNap3y5x4k", 2022-09-09T22:59:48.659 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.659 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.660 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.660 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.660 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.661 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.661 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.661 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.661 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763990.604998_217.1", 2022-09-09T22:59:48.662 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.662 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-35:4b05608c-7197-4e9b-bd29-7989607fa277.4354.35", 2022-09-09T22:59:48.662 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:10.604998041Z", 2022-09-09T22:59:48.662 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.663 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.663 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xarcxlcdFlngGuAvxUBGTlW", 2022-09-09T22:59:48.663 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.663 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.664 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.664 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xarcxlcdFlngGuAvxUBGTlW", 2022-09-09T22:59:48.664 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.664 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.665 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.665 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.665 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.665 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.666 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.666 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.666 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763990.613292_218.1", 2022-09-09T22:59:48.666 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.667 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-35:4b05608c-7197-4e9b-bd29-7989607fa277.4354.35", 2022-09-09T22:59:48.667 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:10.613292272Z", 2022-09-09T22:59:48.667 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.668 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.668 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xarcxlcdFlngGuAvxUBGTlW", 2022-09-09T22:59:48.668 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.669 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.669 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.669 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xarcxlcdFlngGuAvxUBGTlW", 2022-09-09T22:59:48.670 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:48.670 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.670 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.670 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.671 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.671 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.671 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.671 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.672 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763990.663150_219.1", 2022-09-09T22:59:48.672 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.672 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36", 2022-09-09T22:59:48.672 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:10.663150053Z", 2022-09-09T22:59:48.673 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.673 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.673 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.673 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.674 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.674 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.674 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_jPN1kAi0u5MPj33UDl-gVII", 2022-09-09T22:59:48.675 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.675 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.675 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.675 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.676 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.676 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.676 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.676 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.677 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763990.667756_220.1", 2022-09-09T22:59:48.677 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.677 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36", 2022-09-09T22:59:48.677 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:10.667756273Z", 2022-09-09T22:59:48.677 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.678 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.678 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.678 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.679 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.679 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.679 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_jPN1kAi0u5MPj33UDl-gVII", 2022-09-09T22:59:48.679 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.680 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.680 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.680 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.680 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.680 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.681 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.681 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.681 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763990.669560_221.1", 2022-09-09T22:59:48.682 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.682 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-36", 2022-09-09T22:59:48.682 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:10.669560632Z", 2022-09-09T22:59:48.682 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.683 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.683 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.683 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.684 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.684 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.684 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_gu46bZkwwhtar51KzoH8j_M", 2022-09-09T22:59:48.684 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.685 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.685 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.685 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.686 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.686 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.686 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.686 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.687 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763990.673725_222.1", 2022-09-09T22:59:48.687 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.687 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-36", 2022-09-09T22:59:48.687 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:10.673725202Z", 2022-09-09T22:59:48.688 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.688 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.688 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.688 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.689 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.689 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.689 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_gu46bZkwwhtar51KzoH8j_M", 2022-09-09T22:59:48.689 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.690 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.690 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.690 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.690 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.691 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.691 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.691 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.691 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763990.710819_223.1", 2022-09-09T22:59:48.692 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.692 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37", 2022-09-09T22:59:48.692 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:10.710819128Z", 2022-09-09T22:59:48.692 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.693 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.693 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.693 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.693 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.694 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.694 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_FX49dxEb4uM_d2ZifiXBHNo", 2022-09-09T22:59:48.694 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.694 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.695 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.695 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.695 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.695 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.696 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.696 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.696 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763990.715764_224.1", 2022-09-09T22:59:48.696 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.697 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37", 2022-09-09T22:59:48.697 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:10.715764014Z", 2022-09-09T22:59:48.697 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.697 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.698 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.698 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.698 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.699 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.699 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_FX49dxEb4uM_d2ZifiXBHNo", 2022-09-09T22:59:48.699 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.699 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.700 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.700 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.700 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.700 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.701 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.701 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.701 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763990.717500_225.1", 2022-09-09T22:59:48.701 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.702 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-37", 2022-09-09T22:59:48.702 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:10.717500107Z", 2022-09-09T22:59:48.702 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.702 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.703 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.703 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.703 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.703 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.703 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_hJzznYY2Hded3epYJ_abHVf", 2022-09-09T22:59:48.704 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.704 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.704 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.704 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.705 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.705 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.705 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.705 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.706 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763990.722195_226.1", 2022-09-09T22:59:48.706 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.706 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-37", 2022-09-09T22:59:48.707 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:10.722195904Z", 2022-09-09T22:59:48.707 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.707 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.707 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.708 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.708 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.708 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.708 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_hJzznYY2Hded3epYJ_abHVf", 2022-09-09T22:59:48.709 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.709 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.709 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.709 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.710 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.710 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.710 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.710 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.711 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763990.784611_227.1", 2022-09-09T22:59:48.711 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.711 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38", 2022-09-09T22:59:48.711 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:10.784611285Z", 2022-09-09T22:59:48.712 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.712 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.712 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.712 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.713 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.713 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.713 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_qDxaPPFFwg6ZX6ss6qndeGO", 2022-09-09T22:59:48.713 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.714 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.714 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.714 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.714 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.715 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.715 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.715 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.715 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763990.788678_228.1", 2022-09-09T22:59:48.716 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.716 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38", 2022-09-09T22:59:48.716 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:10.788678416Z", 2022-09-09T22:59:48.716 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.717 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.717 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.717 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.717 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.718 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.718 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_qDxaPPFFwg6ZX6ss6qndeGO", 2022-09-09T22:59:48.718 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.718 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.719 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.719 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.719 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.719 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.720 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.720 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.720 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763990.792062_229.1", 2022-09-09T22:59:48.720 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.721 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-38", 2022-09-09T22:59:48.721 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:10.792062116Z", 2022-09-09T22:59:48.721 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.721 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.722 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.722 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.722 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.723 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.723 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_gCNfZZlq_uw7Nsy58KndY3K", 2022-09-09T22:59:48.723 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.723 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.723 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.724 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.724 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.724 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.725 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.725 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.725 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763990.796002_230.1", 2022-09-09T22:59:48.725 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.726 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-38", 2022-09-09T22:59:48.726 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:10.796002253Z", 2022-09-09T22:59:48.726 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.726 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.727 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.727 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.727 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.727 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.728 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_gCNfZZlq_uw7Nsy58KndY3K", 2022-09-09T22:59:48.728 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.728 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.728 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.729 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.729 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.729 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.729 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.730 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.730 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764240.029920_231.1", 2022-09-09T22:59:48.730 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.730 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-39:4b05608c-7197-4e9b-bd29-7989607fa277.4354.39", 2022-09-09T22:59:48.731 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:57:20.029920453Z", 2022-09-09T22:59:48.731 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.731 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.731 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.732 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.732 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.732 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.732 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_yHfFFXCLbn_I9-CIE0fRcSE", 2022-09-09T22:59:48.733 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.733 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.733 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.733 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.734 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.734 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.734 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.735 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.735 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764240.034539_232.1", 2022-09-09T22:59:48.735 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.735 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-39:4b05608c-7197-4e9b-bd29-7989607fa277.4354.39", 2022-09-09T22:59:48.736 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:57:20.034539448Z", 2022-09-09T22:59:48.736 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.736 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.736 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.737 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.737 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.737 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.737 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_yHfFFXCLbn_I9-CIE0fRcSE", 2022-09-09T22:59:48.738 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.738 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.738 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.738 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.739 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.739 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.739 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.739 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.740 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764240.036306_233.1", 2022-09-09T22:59:48.740 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.740 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-39", 2022-09-09T22:59:48.740 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:57:20.036306500Z", 2022-09-09T22:59:48.741 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.741 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.741 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.741 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.742 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.742 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.742 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_kuQlL_FG34kY8wCt0puQ02d", 2022-09-09T22:59:48.742 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.743 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.743 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.743 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.743 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.744 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.744 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.744 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.744 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764240.040726_234.1", 2022-09-09T22:59:48.745 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.745 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-39", 2022-09-09T22:59:48.745 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:57:20.040726112Z", 2022-09-09T22:59:48.745 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.746 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.746 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.746 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.746 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.747 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.747 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_kuQlL_FG34kY8wCt0puQ02d", 2022-09-09T22:59:48.747 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.747 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.748 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.748 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.748 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.748 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.749 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.749 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.750 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764240.078400_235.1", 2022-09-09T22:59:48.750 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.750 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-40:4b05608c-7197-4e9b-bd29-7989607fa277.4354.40", 2022-09-09T22:59:48.751 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:57:20.078400843Z", 2022-09-09T22:59:48.751 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.751 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.752 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.752 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.752 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.752 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.753 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_vfgdN60g8MjxPCCEgs2R_vv", 2022-09-09T22:59:48.753 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.753 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.753 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.754 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.754 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.754 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.755 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.755 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.755 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764240.084545_236.1", 2022-09-09T22:59:48.755 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.756 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-40:4b05608c-7197-4e9b-bd29-7989607fa277.4354.40", 2022-09-09T22:59:48.756 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:57:20.084545560Z", 2022-09-09T22:59:48.756 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.756 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.757 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.757 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.757 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.757 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.758 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_vfgdN60g8MjxPCCEgs2R_vv", 2022-09-09T22:59:48.758 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.758 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.758 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.759 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.759 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.759 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.759 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.760 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.760 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764240.086165_237.1", 2022-09-09T22:59:48.760 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.760 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-40", 2022-09-09T22:59:48.761 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:57:20.086165638Z", 2022-09-09T22:59:48.761 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.761 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.761 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.762 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.762 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.762 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.762 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_TJv9-9zJFLq6wQcqNFr3ADa", 2022-09-09T22:59:48.763 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.763 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.763 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.763 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.764 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.764 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.764 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.764 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.765 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764240.090984_238.1", 2022-09-09T22:59:48.765 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.765 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-40", 2022-09-09T22:59:48.765 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:57:20.090984173Z", 2022-09-09T22:59:48.766 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.766 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.766 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.766 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.767 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.767 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.767 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_TJv9-9zJFLq6wQcqNFr3ADa", 2022-09-09T22:59:48.767 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.768 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.768 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.768 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.768 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.768 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.769 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.769 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.769 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764240.151909_239.1", 2022-09-09T22:59:48.769 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.770 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-41:4b05608c-7197-4e9b-bd29-7989607fa277.4354.41", 2022-09-09T22:59:48.770 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:57:20.151909503Z", 2022-09-09T22:59:48.770 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.770 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.771 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.771 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.771 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.771 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.772 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_lh1D7Ui8P7X3ZyWeXN_Rbr6", 2022-09-09T22:59:48.772 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.772 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.772 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.773 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.773 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.773 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.773 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.774 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.774 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764240.157917_240.1", 2022-09-09T22:59:48.774 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.774 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-41:4b05608c-7197-4e9b-bd29-7989607fa277.4354.41", 2022-09-09T22:59:48.775 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:57:20.157917182Z", 2022-09-09T22:59:48.775 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.775 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.775 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.776 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.776 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.776 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.776 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_lh1D7Ui8P7X3ZyWeXN_Rbr6", 2022-09-09T22:59:48.777 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.777 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.777 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.778 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.778 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.778 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.778 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.779 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.779 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764240.159693_241.1", 2022-09-09T22:59:48.779 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.779 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-41", 2022-09-09T22:59:48.780 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:57:20.159693443Z", 2022-09-09T22:59:48.780 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.780 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.780 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.781 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.781 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.781 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.781 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xxvbuFhSEtP4oKwQrN2eVjC", 2022-09-09T22:59:48.782 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.782 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.782 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.782 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:48.783 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.783 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.783 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.783 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:48.784 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764240.168869_243.1", 2022-09-09T22:59:48.784 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:48.784 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-41", 2022-09-09T22:59:48.784 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:57:20.168869275Z", 2022-09-09T22:59:48.785 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:48.785 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:48.785 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:48.785 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:48.786 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.786 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:48.786 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xxvbuFhSEtP4oKwQrN2eVjC", 2022-09-09T22:59:48.786 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:48.787 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:48.787 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:48.787 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:48.787 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.788 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.788 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:48.788 INFO:teuthology.orchestra.run.smithi080.stdout:] 2022-09-09T22:59:48.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '503b1d61-4513-48c8-a308-17b76219610c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:48.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '503b1d61-4513-48c8-a308-17b76219610c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:48.791 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 503b1d61-4513-48c8-a308-17b76219610c --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:59:48.840 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:48.840 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:48.892 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T22:59:48.893 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:48.893 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762486.140348_44.1", 2022-09-09T22:59:48.893 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "user", 2022-09-09T22:59:48.894 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "rgw-multisite-test-user", 2022-09-09T22:59:48.894 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:28:06.140348282Z", 2022-09-09T22:59:48.894 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:48.895 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:48.895 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:48.895 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:48.895 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.896 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:48.896 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_hX52Pw6Vsy-toyD35b1bFsR", 2022-09-09T22:59:48.896 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:48.896 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.897 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:48.897 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:48.898 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.898 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.899 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.899 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:48.900 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762486.146147_45.1", 2022-09-09T22:59:48.900 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "user", 2022-09-09T22:59:48.900 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "rgw-multisite-test-user", 2022-09-09T22:59:48.900 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:28:06.146147984Z", 2022-09-09T22:59:48.901 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:48.901 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:48.901 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:48.901 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:48.902 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.902 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:48.902 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_hX52Pw6Vsy-toyD35b1bFsR", 2022-09-09T22:59:48.903 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:48.903 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.903 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:48.904 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:48.904 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.904 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.904 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.905 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:48.905 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762609.632258_38.1", 2022-09-09T22:59:48.905 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.905 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-1:4b05608c-7197-4e9b-bd29-7989607fa277.4354.1", 2022-09-09T22:59:48.906 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:09.632258573Z", 2022-09-09T22:59:48.906 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:48.906 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:48.907 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:48.907 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:48.907 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.907 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:48.908 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_S0f1bdC1tDuhcRUnkrIExa6", 2022-09-09T22:59:48.908 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:48.909 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.909 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:48.910 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:48.910 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.911 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.911 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.911 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:48.911 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762609.636081_39.1", 2022-09-09T22:59:48.912 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.912 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-1:4b05608c-7197-4e9b-bd29-7989607fa277.4354.1", 2022-09-09T22:59:48.912 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:09.636081844Z", 2022-09-09T22:59:48.912 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:48.913 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:48.913 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:48.913 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:48.913 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.914 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:48.914 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_S0f1bdC1tDuhcRUnkrIExa6", 2022-09-09T22:59:48.915 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:48.915 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.915 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:48.915 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:48.916 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.916 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.916 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.916 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:48.917 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762609.637417_40.1", 2022-09-09T22:59:48.917 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:48.917 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-1", 2022-09-09T22:59:48.917 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:09.637417779Z", 2022-09-09T22:59:48.918 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:48.918 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:48.919 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:48.919 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:48.920 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.920 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:48.921 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_ykXwHWWOzdeAKyn1PiulIFb", 2022-09-09T22:59:48.921 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:48.922 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.922 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:48.923 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:48.923 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.924 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.924 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.925 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:48.925 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762609.640815_41.1", 2022-09-09T22:59:48.926 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:48.926 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-1", 2022-09-09T22:59:48.927 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:09.640815201Z", 2022-09-09T22:59:48.927 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:48.928 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:48.928 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:48.929 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:48.929 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.930 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:48.930 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_ykXwHWWOzdeAKyn1PiulIFb", 2022-09-09T22:59:48.931 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:48.931 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.932 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:48.932 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:48.932 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.933 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.933 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.934 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:48.934 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762609.685710_42.1", 2022-09-09T22:59:48.935 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.935 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-2:4b05608c-7197-4e9b-bd29-7989607fa277.4354.2", 2022-09-09T22:59:48.936 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:09.685710351Z", 2022-09-09T22:59:48.936 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:48.937 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:48.937 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:48.938 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:48.938 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.939 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:48.939 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_p7pA6aFQb4GfhoLrAPoFo3L", 2022-09-09T22:59:48.940 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:48.940 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.941 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:48.941 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:48.942 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.942 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.942 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.943 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:48.943 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762609.691000_43.1", 2022-09-09T22:59:48.944 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.944 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-2:4b05608c-7197-4e9b-bd29-7989607fa277.4354.2", 2022-09-09T22:59:48.945 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:09.691000087Z", 2022-09-09T22:59:48.945 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:48.945 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:48.946 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:48.946 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:48.947 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.947 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:48.947 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_p7pA6aFQb4GfhoLrAPoFo3L", 2022-09-09T22:59:48.948 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:48.948 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.949 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:48.952 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:48.952 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.953 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.953 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.953 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:48.954 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762609.692503_44.1", 2022-09-09T22:59:48.954 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:48.954 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-2", 2022-09-09T22:59:48.954 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:09.692503564Z", 2022-09-09T22:59:48.955 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:48.955 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:48.955 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:48.955 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:48.956 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.956 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:48.956 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_ut5cP71qSrGfyAy9sYnUiGl", 2022-09-09T22:59:48.956 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:48.957 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.957 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:48.957 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:48.957 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.958 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.958 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.958 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:48.958 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762609.696482_45.1", 2022-09-09T22:59:48.959 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:48.959 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-2", 2022-09-09T22:59:48.959 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:09.696482596Z", 2022-09-09T22:59:48.959 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:48.960 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:48.960 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:48.960 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:48.960 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.961 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:48.961 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_ut5cP71qSrGfyAy9sYnUiGl", 2022-09-09T22:59:48.961 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:48.961 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.962 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:48.962 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:48.962 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.962 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.963 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.963 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:48.963 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762612.758261_46.1", 2022-09-09T22:59:48.963 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.964 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-3:4b05608c-7197-4e9b-bd29-7989607fa277.4354.3", 2022-09-09T22:59:48.964 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:12.758261347Z", 2022-09-09T22:59:48.964 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:48.964 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:48.965 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:48.965 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:48.965 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.965 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:48.966 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_DTaKu07VPRWGanYia-lsgfe", 2022-09-09T22:59:48.966 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:48.966 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.966 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:48.967 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:48.967 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.967 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.967 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.968 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:48.968 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762612.764541_47.1", 2022-09-09T22:59:48.968 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.968 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-3:4b05608c-7197-4e9b-bd29-7989607fa277.4354.3", 2022-09-09T22:59:48.969 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:12.764541107Z", 2022-09-09T22:59:48.969 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:48.969 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:48.969 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:48.970 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:48.970 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.970 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:48.970 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_DTaKu07VPRWGanYia-lsgfe", 2022-09-09T22:59:48.971 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:48.971 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.971 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:48.971 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:48.972 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.972 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.972 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.972 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:48.973 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762612.766069_48.1", 2022-09-09T22:59:48.973 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:48.973 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-3", 2022-09-09T22:59:48.973 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:12.766069866Z", 2022-09-09T22:59:48.974 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:48.974 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:48.974 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:48.974 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:48.975 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.975 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:48.975 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_LVgUDeF4cxLIns2No37bLQx", 2022-09-09T22:59:48.975 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:48.976 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.976 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:48.976 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:48.976 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.977 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.977 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.977 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:48.977 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762612.770414_49.1", 2022-09-09T22:59:48.978 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:48.978 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-3", 2022-09-09T22:59:48.978 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:12.770414961Z", 2022-09-09T22:59:48.978 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:48.979 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:48.979 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:48.979 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:48.979 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.980 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:48.980 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_LVgUDeF4cxLIns2No37bLQx", 2022-09-09T22:59:48.980 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:48.980 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.981 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:48.981 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:48.981 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.981 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.982 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.982 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:48.982 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762618.742117_50.1", 2022-09-09T22:59:48.982 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.983 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-4:4b05608c-7197-4e9b-bd29-7989607fa277.4354.4", 2022-09-09T22:59:48.983 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:18.742117846Z", 2022-09-09T22:59:48.983 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:48.983 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:48.984 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:48.984 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:48.984 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.984 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:48.985 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_QrfGp4QHJmNMDcTAMoEbJrk", 2022-09-09T22:59:48.985 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:48.985 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.986 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:48.986 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:48.986 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.986 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.987 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.987 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:48.987 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762618.747553_51.1", 2022-09-09T22:59:48.987 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:48.988 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-4:4b05608c-7197-4e9b-bd29-7989607fa277.4354.4", 2022-09-09T22:59:48.988 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:18.747553163Z", 2022-09-09T22:59:48.988 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:48.988 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:48.989 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:48.989 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:48.989 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.989 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:48.989 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_QrfGp4QHJmNMDcTAMoEbJrk", 2022-09-09T22:59:48.990 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:48.990 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.990 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:48.990 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:48.991 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.991 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.991 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.991 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:48.992 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762618.749036_52.1", 2022-09-09T22:59:48.992 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:48.992 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-4", 2022-09-09T22:59:48.992 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:18.749036760Z", 2022-09-09T22:59:48.993 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:48.993 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:48.993 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:48.993 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:48.994 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.994 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:48.994 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_pNYiU6yoVaLSavU2aw7eQra", 2022-09-09T22:59:48.994 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:48.995 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.995 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:48.995 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:48.995 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.996 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:48.996 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.996 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:48.996 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762618.752816_53.1", 2022-09-09T22:59:48.997 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:48.997 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-4", 2022-09-09T22:59:48.997 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:18.752816607Z", 2022-09-09T22:59:48.997 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:48.998 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:48.998 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:48.998 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:48.998 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:48.999 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:48.999 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_pNYiU6yoVaLSavU2aw7eQra", 2022-09-09T22:59:48.999 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:48.999 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.000 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.000 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.000 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.000 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.001 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.001 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.001 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762618.783525_54.1", 2022-09-09T22:59:49.001 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.002 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-5:4b05608c-7197-4e9b-bd29-7989607fa277.4354.5", 2022-09-09T22:59:49.002 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:18.783525474Z", 2022-09-09T22:59:49.002 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.003 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.003 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.003 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.003 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.003 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.004 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_PPSGwb27eWuU4vjS5103sBF", 2022-09-09T22:59:49.004 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.004 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.004 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.005 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.005 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.005 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.006 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.006 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.006 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762618.789318_55.1", 2022-09-09T22:59:49.006 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.007 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-5:4b05608c-7197-4e9b-bd29-7989607fa277.4354.5", 2022-09-09T22:59:49.007 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:18.789318940Z", 2022-09-09T22:59:49.007 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.007 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.008 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.008 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.008 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.008 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.008 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_PPSGwb27eWuU4vjS5103sBF", 2022-09-09T22:59:49.009 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.009 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.009 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.010 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.010 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.010 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.011 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.011 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.011 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762618.791554_56.1", 2022-09-09T22:59:49.012 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.012 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-5", 2022-09-09T22:59:49.012 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:18.791554821Z", 2022-09-09T22:59:49.013 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.013 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.013 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.014 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.014 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.014 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.015 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_rrNh6U2_REBalApJMN0ghUy", 2022-09-09T22:59:49.015 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.015 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.015 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.016 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.016 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.016 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.016 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.017 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.017 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762618.797453_57.1", 2022-09-09T22:59:49.017 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.017 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-5", 2022-09-09T22:59:49.018 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:18.797453446Z", 2022-09-09T22:59:49.018 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.018 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.018 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.019 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.019 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.019 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.019 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_rrNh6U2_REBalApJMN0ghUy", 2022-09-09T22:59:49.020 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.020 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.020 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.020 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.021 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.021 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.021 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.021 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.022 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762618.850246_58.1", 2022-09-09T22:59:49.022 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.022 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-6:4b05608c-7197-4e9b-bd29-7989607fa277.4354.6", 2022-09-09T22:59:49.022 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:18.850246676Z", 2022-09-09T22:59:49.023 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.023 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.023 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.023 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.024 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.024 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.024 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_cRk-wHT_rpr3K1T0yRJVf7Q", 2022-09-09T22:59:49.024 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.025 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.025 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.025 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.025 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.026 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.026 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.026 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.026 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762618.855531_59.1", 2022-09-09T22:59:49.027 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.027 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-6:4b05608c-7197-4e9b-bd29-7989607fa277.4354.6", 2022-09-09T22:59:49.027 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:18.855531993Z", 2022-09-09T22:59:49.027 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.028 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.028 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.028 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.028 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.029 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.029 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_cRk-wHT_rpr3K1T0yRJVf7Q", 2022-09-09T22:59:49.029 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.029 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.030 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.030 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.030 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.030 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.031 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.031 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.031 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762618.857124_60.1", 2022-09-09T22:59:49.032 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.032 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-6", 2022-09-09T22:59:49.032 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:18.857124257Z", 2022-09-09T22:59:49.032 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.032 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.033 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.033 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.033 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.033 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.034 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_uJKaJMt6G8HexUvWErzrMXK", 2022-09-09T22:59:49.034 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.034 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.034 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.035 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.035 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.035 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.035 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.036 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.036 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762618.860801_61.1", 2022-09-09T22:59:49.036 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.036 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-6", 2022-09-09T22:59:49.037 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:18.860801405Z", 2022-09-09T22:59:49.037 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.037 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.037 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.038 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.038 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.038 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.039 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_uJKaJMt6G8HexUvWErzrMXK", 2022-09-09T22:59:49.039 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.039 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.039 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.039 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.040 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.040 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.040 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.040 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.041 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762621.000610_62.1", 2022-09-09T22:59:49.041 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.041 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-7:4b05608c-7197-4e9b-bd29-7989607fa277.4354.7", 2022-09-09T22:59:49.041 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:21.000610746Z", 2022-09-09T22:59:49.042 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.042 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.042 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.043 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.043 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.043 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.043 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_RZXAZlTqoK-kSBG7i9Qb1iT", 2022-09-09T22:59:49.044 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.044 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.044 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.044 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.045 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.045 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.045 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.045 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.045 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762621.004495_63.1", 2022-09-09T22:59:49.046 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.046 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-7:4b05608c-7197-4e9b-bd29-7989607fa277.4354.7", 2022-09-09T22:59:49.046 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:21.004495603Z", 2022-09-09T22:59:49.047 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.047 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.047 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.047 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.047 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.048 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.048 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_RZXAZlTqoK-kSBG7i9Qb1iT", 2022-09-09T22:59:49.048 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.048 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.049 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.049 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.049 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.049 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.050 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.050 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.050 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762621.005900_64.1", 2022-09-09T22:59:49.050 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.051 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-7", 2022-09-09T22:59:49.051 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:21.005900841Z", 2022-09-09T22:59:49.051 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.051 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.052 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.052 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.052 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.052 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.053 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_kxHFo1nRBQfhvLRoWDL9llN", 2022-09-09T22:59:49.053 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.053 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.053 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.054 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.054 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.054 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.054 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.055 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.055 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762621.009874_65.1", 2022-09-09T22:59:49.055 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.055 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-7", 2022-09-09T22:59:49.056 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:21.009874532Z", 2022-09-09T22:59:49.056 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.056 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.056 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.057 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.057 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.057 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.057 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_kxHFo1nRBQfhvLRoWDL9llN", 2022-09-09T22:59:49.058 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.058 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.058 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.058 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.059 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.059 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.059 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.059 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.060 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762621.036747_66.1", 2022-09-09T22:59:49.060 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.060 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-8:4b05608c-7197-4e9b-bd29-7989607fa277.4354.8", 2022-09-09T22:59:49.060 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:21.036747351Z", 2022-09-09T22:59:49.061 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.061 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.061 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.061 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.062 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.062 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.062 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_CE4D4FZvn3xK_Q51293KOFn", 2022-09-09T22:59:49.062 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.063 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.063 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.063 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.063 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.064 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.064 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.064 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.064 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762621.042100_67.1", 2022-09-09T22:59:49.065 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.065 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-8:4b05608c-7197-4e9b-bd29-7989607fa277.4354.8", 2022-09-09T22:59:49.065 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:21.042100215Z", 2022-09-09T22:59:49.065 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.066 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.066 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.066 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.066 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.067 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.067 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_CE4D4FZvn3xK_Q51293KOFn", 2022-09-09T22:59:49.067 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.067 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.068 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.068 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.068 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.068 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.069 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.069 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.069 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762621.043792_68.1", 2022-09-09T22:59:49.069 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.069 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-8", 2022-09-09T22:59:49.070 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:21.043792412Z", 2022-09-09T22:59:49.070 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.070 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.071 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.071 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.071 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.071 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.072 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_LemC9Nq4ijGLYcUn_qbYGY1", 2022-09-09T22:59:49.072 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.072 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.072 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.073 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.073 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.073 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.073 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.074 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.074 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762621.047923_69.1", 2022-09-09T22:59:49.074 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.074 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-8", 2022-09-09T22:59:49.075 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:21.047923611Z", 2022-09-09T22:59:49.075 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.075 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.075 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.076 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.076 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.076 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.076 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_LemC9Nq4ijGLYcUn_qbYGY1", 2022-09-09T22:59:49.077 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.077 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.077 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.077 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.077 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.078 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.078 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.078 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.079 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762621.100882_70.1", 2022-09-09T22:59:49.079 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.079 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-9:4b05608c-7197-4e9b-bd29-7989607fa277.4354.9", 2022-09-09T22:59:49.079 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:21.100882035Z", 2022-09-09T22:59:49.079 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.080 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.080 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.080 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.081 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.081 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.081 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_mXmeXhs1mUho7dSTp_n1nkI", 2022-09-09T22:59:49.081 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.082 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.082 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.082 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.082 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.083 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.083 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.083 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.083 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762621.105043_71.1", 2022-09-09T22:59:49.084 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.084 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-9:4b05608c-7197-4e9b-bd29-7989607fa277.4354.9", 2022-09-09T22:59:49.084 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:21.105043152Z", 2022-09-09T22:59:49.084 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.085 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.085 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.085 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.085 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.086 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.086 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_mXmeXhs1mUho7dSTp_n1nkI", 2022-09-09T22:59:49.086 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.086 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.087 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.087 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.087 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.087 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.088 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.088 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.088 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762621.106530_72.1", 2022-09-09T22:59:49.088 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.089 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-9", 2022-09-09T22:59:49.089 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:21.106530704Z", 2022-09-09T22:59:49.089 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.089 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.090 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.090 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.090 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.090 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.091 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_4XQiLk5HQiFc0Zuqos3dBaU", 2022-09-09T22:59:49.091 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.091 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.091 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.092 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.092 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.092 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.092 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.093 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.093 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762621.110791_73.1", 2022-09-09T22:59:49.093 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.093 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-9", 2022-09-09T22:59:49.094 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:21.110791110Z", 2022-09-09T22:59:49.094 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.094 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.094 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.095 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.095 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.095 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.095 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_4XQiLk5HQiFc0Zuqos3dBaU", 2022-09-09T22:59:49.096 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.096 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.096 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.096 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.097 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.097 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.097 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.097 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.098 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762637.476803_74.1", 2022-09-09T22:59:49.098 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.098 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-7:4b05608c-7197-4e9b-bd29-7989607fa277.4354.7", 2022-09-09T22:59:49.098 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.476803832Z", 2022-09-09T22:59:49.099 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.099 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.099 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_RZXAZlTqoK-kSBG7i9Qb1iT", 2022-09-09T22:59:49.099 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.100 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.100 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.100 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_RZXAZlTqoK-kSBG7i9Qb1iT", 2022-09-09T22:59:49.100 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.101 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.101 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.101 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.101 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.102 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.102 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.102 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.102 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762637.483128_75.1", 2022-09-09T22:59:49.103 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.103 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-7:4b05608c-7197-4e9b-bd29-7989607fa277.4354.7", 2022-09-09T22:59:49.103 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.483128307Z", 2022-09-09T22:59:49.103 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.104 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.104 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_RZXAZlTqoK-kSBG7i9Qb1iT", 2022-09-09T22:59:49.104 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.104 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.105 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.105 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_RZXAZlTqoK-kSBG7i9Qb1iT", 2022-09-09T22:59:49.105 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.105 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.106 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.106 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.106 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.107 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.107 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.107 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.107 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762637.506467_76.1", 2022-09-09T22:59:49.108 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.108 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-7", 2022-09-09T22:59:49.108 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.506467744Z", 2022-09-09T22:59:49.108 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.109 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.109 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_kxHFo1nRBQfhvLRoWDL9llN", 2022-09-09T22:59:49.109 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.109 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.110 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.110 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_kxHFo1nRBQfhvLRoWDL9llN", 2022-09-09T22:59:49.110 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.110 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.111 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.111 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "remove" 2022-09-09T22:59:49.111 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.111 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.112 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.112 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.112 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762637.511752_77.1", 2022-09-09T22:59:49.112 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.113 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-7", 2022-09-09T22:59:49.113 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.511752143Z", 2022-09-09T22:59:49.113 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.113 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.114 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_kxHFo1nRBQfhvLRoWDL9llN", 2022-09-09T22:59:49.114 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.114 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.114 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.115 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_kxHFo1nRBQfhvLRoWDL9llN", 2022-09-09T22:59:49.115 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.115 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.115 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.116 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.116 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.116 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.116 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.117 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.117 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762637.579226_78.1", 2022-09-09T22:59:49.117 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.117 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-8:4b05608c-7197-4e9b-bd29-7989607fa277.4354.8", 2022-09-09T22:59:49.118 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.579226395Z", 2022-09-09T22:59:49.118 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.118 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.118 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_CE4D4FZvn3xK_Q51293KOFn", 2022-09-09T22:59:49.119 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.119 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.119 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.119 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_CE4D4FZvn3xK_Q51293KOFn", 2022-09-09T22:59:49.120 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.120 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.120 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.120 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.121 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.121 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.121 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.121 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.122 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762637.584225_79.1", 2022-09-09T22:59:49.122 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.122 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-8:4b05608c-7197-4e9b-bd29-7989607fa277.4354.8", 2022-09-09T22:59:49.122 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.584225809Z", 2022-09-09T22:59:49.123 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.123 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.123 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_CE4D4FZvn3xK_Q51293KOFn", 2022-09-09T22:59:49.123 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.124 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.124 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.124 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_CE4D4FZvn3xK_Q51293KOFn", 2022-09-09T22:59:49.124 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.125 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.125 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.125 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.125 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.126 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.126 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.126 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.126 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762637.601579_80.1", 2022-09-09T22:59:49.127 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.127 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-8", 2022-09-09T22:59:49.127 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.601579852Z", 2022-09-09T22:59:49.127 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.128 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.128 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_LemC9Nq4ijGLYcUn_qbYGY1", 2022-09-09T22:59:49.128 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.128 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.129 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.129 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_LemC9Nq4ijGLYcUn_qbYGY1", 2022-09-09T22:59:49.129 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.129 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.130 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.130 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "remove" 2022-09-09T22:59:49.130 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.130 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.131 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.131 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.131 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762637.608474_81.1", 2022-09-09T22:59:49.131 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.132 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-8", 2022-09-09T22:59:49.132 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.608474828Z", 2022-09-09T22:59:49.132 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.132 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.133 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_LemC9Nq4ijGLYcUn_qbYGY1", 2022-09-09T22:59:49.133 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.133 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.133 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.134 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_LemC9Nq4ijGLYcUn_qbYGY1", 2022-09-09T22:59:49.134 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.134 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.134 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.135 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.135 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.135 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.135 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.136 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.136 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762637.726627_82.1", 2022-09-09T22:59:49.136 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.136 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-9:4b05608c-7197-4e9b-bd29-7989607fa277.4354.9", 2022-09-09T22:59:49.137 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.726627171Z", 2022-09-09T22:59:49.137 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.137 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.137 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_mXmeXhs1mUho7dSTp_n1nkI", 2022-09-09T22:59:49.138 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.138 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.138 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.138 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_mXmeXhs1mUho7dSTp_n1nkI", 2022-09-09T22:59:49.139 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.139 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.139 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.139 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.140 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.140 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.140 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.140 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.141 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762637.731530_83.1", 2022-09-09T22:59:49.141 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.141 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-9:4b05608c-7197-4e9b-bd29-7989607fa277.4354.9", 2022-09-09T22:59:49.141 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.731530094Z", 2022-09-09T22:59:49.142 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.142 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.142 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_mXmeXhs1mUho7dSTp_n1nkI", 2022-09-09T22:59:49.142 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.143 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.143 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.143 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_mXmeXhs1mUho7dSTp_n1nkI", 2022-09-09T22:59:49.143 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.144 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.144 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.144 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.144 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.145 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.145 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.145 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.145 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762637.751637_84.1", 2022-09-09T22:59:49.146 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.146 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-9", 2022-09-09T22:59:49.146 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.751637260Z", 2022-09-09T22:59:49.146 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.147 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.147 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_4XQiLk5HQiFc0Zuqos3dBaU", 2022-09-09T22:59:49.147 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.147 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.148 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.148 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_4XQiLk5HQiFc0Zuqos3dBaU", 2022-09-09T22:59:49.148 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.148 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.149 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.149 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "remove" 2022-09-09T22:59:49.149 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.149 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.150 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.150 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.150 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762637.757014_85.1", 2022-09-09T22:59:49.150 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.151 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-9", 2022-09-09T22:59:49.151 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.757014147Z", 2022-09-09T22:59:49.151 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.151 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.152 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_4XQiLk5HQiFc0Zuqos3dBaU", 2022-09-09T22:59:49.152 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.152 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.152 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.153 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_4XQiLk5HQiFc0Zuqos3dBaU", 2022-09-09T22:59:49.153 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.153 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.153 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.154 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.154 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.154 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.154 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.155 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.155 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762643.827940_36.1", 2022-09-09T22:59:49.155 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.155 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10", 2022-09-09T22:59:49.156 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:43.827940737Z", 2022-09-09T22:59:49.156 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.156 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.156 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.157 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.157 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.157 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.157 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_H19u2ytwno45CS85TVUwNCX", 2022-09-09T22:59:49.158 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.158 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.158 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.158 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.159 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.159 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.159 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.159 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.160 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762643.831705_37.1", 2022-09-09T22:59:49.160 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.160 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10", 2022-09-09T22:59:49.160 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:43.831705348Z", 2022-09-09T22:59:49.161 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.161 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.161 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.161 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.162 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.162 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.162 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_H19u2ytwno45CS85TVUwNCX", 2022-09-09T22:59:49.162 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.163 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.163 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.163 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.163 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.164 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.164 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.164 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.164 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762643.833115_38.1", 2022-09-09T22:59:49.165 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.165 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-10", 2022-09-09T22:59:49.165 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:43.833115806Z", 2022-09-09T22:59:49.165 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.166 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.166 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.166 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.166 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.167 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.167 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_fQCrFkZsSXo9ojIGdsOR1AR", 2022-09-09T22:59:49.167 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.167 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.167 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.168 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.168 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.168 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.168 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.169 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.169 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762643.836655_39.1", 2022-09-09T22:59:49.169 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.170 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-10", 2022-09-09T22:59:49.170 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:43.836655955Z", 2022-09-09T22:59:49.170 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.170 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.171 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.171 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.171 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.171 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.171 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_fQCrFkZsSXo9ojIGdsOR1AR", 2022-09-09T22:59:49.172 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.172 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.172 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.172 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.173 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.173 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.173 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.174 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.174 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762643.862368_40.1", 2022-09-09T22:59:49.174 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.174 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11", 2022-09-09T22:59:49.175 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:43.862368801Z", 2022-09-09T22:59:49.175 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.175 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.175 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.176 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.176 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.176 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.176 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_zzZ2TMBNyJ82w6D_gHE19mR", 2022-09-09T22:59:49.177 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.177 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.177 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.177 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.178 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.178 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.178 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.178 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.179 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762643.867631_41.1", 2022-09-09T22:59:49.179 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.179 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11", 2022-09-09T22:59:49.179 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:43.867631329Z", 2022-09-09T22:59:49.180 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.180 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.180 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.180 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.181 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.181 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.181 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_zzZ2TMBNyJ82w6D_gHE19mR", 2022-09-09T22:59:49.181 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.182 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.182 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.182 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.182 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.183 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.183 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.183 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.183 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762643.869504_42.1", 2022-09-09T22:59:49.184 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.184 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-11", 2022-09-09T22:59:49.184 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:43.869504153Z", 2022-09-09T22:59:49.184 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.185 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.185 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.185 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.185 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.186 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.186 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_TbMj1OrULohgBriaPgsn-L0", 2022-09-09T22:59:49.186 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.186 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.187 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.187 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.187 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.187 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.188 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.188 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.188 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762643.874819_43.1", 2022-09-09T22:59:49.188 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.189 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-11", 2022-09-09T22:59:49.189 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:43.874819824Z", 2022-09-09T22:59:49.189 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.189 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.190 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.190 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.190 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.190 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.191 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_TbMj1OrULohgBriaPgsn-L0", 2022-09-09T22:59:49.191 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.191 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.191 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.192 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.192 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.192 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.192 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.193 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.193 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762643.923552_44.1", 2022-09-09T22:59:49.193 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.193 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12", 2022-09-09T22:59:49.194 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:43.923552476Z", 2022-09-09T22:59:49.194 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.194 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.194 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.195 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.195 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.195 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.196 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_BNcMrJxZygGN9lZLTtBZnDT", 2022-09-09T22:59:49.196 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.196 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.196 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.196 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.197 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.197 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.197 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.197 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.198 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762643.928237_45.1", 2022-09-09T22:59:49.198 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.198 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12", 2022-09-09T22:59:49.198 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:43.928237283Z", 2022-09-09T22:59:49.199 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.199 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.199 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.199 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.200 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.200 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.200 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_BNcMrJxZygGN9lZLTtBZnDT", 2022-09-09T22:59:49.200 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.201 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.201 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.201 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.201 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.202 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.202 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.202 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.202 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762643.929723_46.1", 2022-09-09T22:59:49.203 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.203 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-12", 2022-09-09T22:59:49.203 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:43.929723240Z", 2022-09-09T22:59:49.203 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.204 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.204 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.204 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.204 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.205 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.205 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_yd7O3Le1YA9DE_Q8UDH4F_4", 2022-09-09T22:59:49.205 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.205 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.206 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.206 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.206 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.206 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.207 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.207 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.207 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762643.933454_47.1", 2022-09-09T22:59:49.207 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.208 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-12", 2022-09-09T22:59:49.208 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:43.933454382Z", 2022-09-09T22:59:49.208 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.208 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.209 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.209 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.209 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.209 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.210 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_yd7O3Le1YA9DE_Q8UDH4F_4", 2022-09-09T22:59:49.210 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.210 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.210 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.211 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.211 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.211 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.211 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.212 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.212 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762786.612586_48.1", 2022-09-09T22:59:49.212 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.212 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13", 2022-09-09T22:59:49.213 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:33:06.612586069Z", 2022-09-09T22:59:49.213 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.213 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.213 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.214 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.214 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.214 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.214 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_dqgbU-URapyJOJCglaP6dNs", 2022-09-09T22:59:49.215 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.215 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.215 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.215 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.216 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.216 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.216 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.216 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.217 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762786.616793_49.1", 2022-09-09T22:59:49.217 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.217 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13", 2022-09-09T22:59:49.217 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:33:06.616793174Z", 2022-09-09T22:59:49.218 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.218 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.218 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.218 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.219 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.219 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.219 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_dqgbU-URapyJOJCglaP6dNs", 2022-09-09T22:59:49.219 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.220 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.220 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.220 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.220 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.221 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.221 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.221 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.221 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762786.618187_50.1", 2022-09-09T22:59:49.222 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.222 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-13", 2022-09-09T22:59:49.222 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:33:06.618187872Z", 2022-09-09T22:59:49.222 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.223 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.223 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.223 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.223 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.224 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.224 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_JNgA-Mcbf5d_uTKjSiKsu05", 2022-09-09T22:59:49.224 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.224 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.224 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.225 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.225 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.225 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.225 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.226 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.226 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762786.621862_51.1", 2022-09-09T22:59:49.226 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.226 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-13", 2022-09-09T22:59:49.227 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:33:06.621862110Z", 2022-09-09T22:59:49.227 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.227 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.227 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.228 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.228 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.228 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.228 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_JNgA-Mcbf5d_uTKjSiKsu05", 2022-09-09T22:59:49.229 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.229 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.229 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.229 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.230 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.230 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.230 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.230 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.231 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762786.651673_52.1", 2022-09-09T22:59:49.231 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.231 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14", 2022-09-09T22:59:49.231 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:33:06.651673719Z", 2022-09-09T22:59:49.232 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.232 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.232 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.232 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.233 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.233 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.233 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_f0wDKO7d5qM7TS5fzfH6prz", 2022-09-09T22:59:49.233 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.234 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.234 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.234 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.235 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.235 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.235 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.235 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.235 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762786.656784_53.1", 2022-09-09T22:59:49.236 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.236 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14", 2022-09-09T22:59:49.236 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:33:06.656784091Z", 2022-09-09T22:59:49.237 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.237 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.237 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.237 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.238 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.238 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.238 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_f0wDKO7d5qM7TS5fzfH6prz", 2022-09-09T22:59:49.238 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.239 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.239 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.239 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.239 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.240 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.240 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.240 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.240 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762786.658400_54.1", 2022-09-09T22:59:49.241 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.241 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-14", 2022-09-09T22:59:49.241 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:33:06.658400465Z", 2022-09-09T22:59:49.241 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.242 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.242 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.242 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.242 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.243 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.243 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_tDp27FdXJgMOU8rSYSjQb_Y", 2022-09-09T22:59:49.243 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.243 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.244 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.244 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.244 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.244 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.245 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.245 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.245 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762786.662396_55.1", 2022-09-09T22:59:49.245 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.246 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-14", 2022-09-09T22:59:49.246 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:33:06.662396356Z", 2022-09-09T22:59:49.246 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.246 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.247 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.247 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.247 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.247 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.248 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_tDp27FdXJgMOU8rSYSjQb_Y", 2022-09-09T22:59:49.248 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.248 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.248 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.249 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.249 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.249 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.249 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.250 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.250 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762786.710171_56.1", 2022-09-09T22:59:49.250 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.250 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15", 2022-09-09T22:59:49.251 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:33:06.710171968Z", 2022-09-09T22:59:49.251 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.251 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.251 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.252 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.252 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.252 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.253 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_ogbkKIIiKYU8oWXJ_RMpS5O", 2022-09-09T22:59:49.253 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.253 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.253 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.253 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.254 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.254 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.254 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.254 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.255 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762786.713926_57.1", 2022-09-09T22:59:49.255 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.255 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15", 2022-09-09T22:59:49.256 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:33:06.713926458Z", 2022-09-09T22:59:49.256 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.256 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.256 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.257 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.257 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.257 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.257 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_ogbkKIIiKYU8oWXJ_RMpS5O", 2022-09-09T22:59:49.258 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.258 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.258 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.258 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.259 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.259 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.259 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.259 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.260 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762786.715462_58.1", 2022-09-09T22:59:49.260 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.260 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-15", 2022-09-09T22:59:49.260 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:33:06.715462548Z", 2022-09-09T22:59:49.261 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.261 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.261 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.261 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.262 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.262 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.262 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_Hk_J6cauBfq0403R0YNkLtB", 2022-09-09T22:59:49.262 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.263 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.263 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.263 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.263 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.264 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.264 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.264 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.264 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762786.718887_59.1", 2022-09-09T22:59:49.265 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.265 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-15", 2022-09-09T22:59:49.265 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:33:06.718887015Z", 2022-09-09T22:59:49.265 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.266 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.266 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.266 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.266 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.267 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.267 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_Hk_J6cauBfq0403R0YNkLtB", 2022-09-09T22:59:49.267 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.267 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.268 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.268 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.268 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.268 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.269 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.269 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.269 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.055821_61.1", 2022-09-09T22:59:49.269 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.270 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16", 2022-09-09T22:59:49.270 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.055821864Z", 2022-09-09T22:59:49.270 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.270 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.271 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.271 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.271 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.271 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.272 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_jgmlG80S-C76e7qmPCPXWwY", 2022-09-09T22:59:49.272 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.272 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.272 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.273 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.273 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.273 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.273 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.274 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.274 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.060244_62.1", 2022-09-09T22:59:49.274 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.274 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16", 2022-09-09T22:59:49.275 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.060244432Z", 2022-09-09T22:59:49.275 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.275 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.275 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.276 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.276 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.276 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.276 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_jgmlG80S-C76e7qmPCPXWwY", 2022-09-09T22:59:49.277 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.277 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.277 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.277 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.278 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.278 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.278 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.278 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.279 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.061724_63.1", 2022-09-09T22:59:49.279 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.279 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-16", 2022-09-09T22:59:49.279 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.061724454Z", 2022-09-09T22:59:49.280 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.280 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.280 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.280 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.281 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.281 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.281 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_QzVpUqaeVNRQiAA9WAn6beJ", 2022-09-09T22:59:49.281 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.282 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.282 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.282 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.282 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.283 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.283 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.283 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.283 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.065560_64.1", 2022-09-09T22:59:49.284 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.284 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-16", 2022-09-09T22:59:49.284 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.065560207Z", 2022-09-09T22:59:49.284 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.285 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.285 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.285 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.285 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.286 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.286 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_QzVpUqaeVNRQiAA9WAn6beJ", 2022-09-09T22:59:49.286 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.286 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.287 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.287 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.287 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.287 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.288 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.288 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.288 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.094558_65.1", 2022-09-09T22:59:49.288 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.289 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17", 2022-09-09T22:59:49.289 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.094558838Z", 2022-09-09T22:59:49.289 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.289 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.290 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.290 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.290 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.290 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.291 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_MtuyPaZmCzfHOcgXjri3F-V", 2022-09-09T22:59:49.291 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.291 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.291 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.292 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.292 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.292 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.292 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.293 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.293 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.099477_66.1", 2022-09-09T22:59:49.293 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.293 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17", 2022-09-09T22:59:49.294 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.099477016Z", 2022-09-09T22:59:49.294 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.294 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.294 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.295 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.295 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.295 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.295 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_MtuyPaZmCzfHOcgXjri3F-V", 2022-09-09T22:59:49.296 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.296 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.296 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.296 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.297 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.297 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.297 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.298 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.101086_67.1", 2022-09-09T22:59:49.298 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.299 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-17", 2022-09-09T22:59:49.299 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.101086117Z", 2022-09-09T22:59:49.299 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.300 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.300 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.301 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.301 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.302 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.302 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_eneLpQRdXpsBeKeIZdm9Fz7", 2022-09-09T22:59:49.303 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.303 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.304 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.304 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.305 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.305 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.306 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.306 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.307 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.105046_68.1", 2022-09-09T22:59:49.307 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.308 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-17", 2022-09-09T22:59:49.308 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.105046294Z", 2022-09-09T22:59:49.309 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.309 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.310 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.311 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.311 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.312 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.312 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_eneLpQRdXpsBeKeIZdm9Fz7", 2022-09-09T22:59:49.313 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.313 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.314 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.314 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.315 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.315 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.316 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.316 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.317 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.149665_69.1", 2022-09-09T22:59:49.317 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.318 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18", 2022-09-09T22:59:49.318 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.149665741Z", 2022-09-09T22:59:49.319 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.319 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.320 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.320 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.321 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.321 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.322 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_bCS5W1GGiGEz-gWOOFrVrp4", 2022-09-09T22:59:49.322 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.323 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.323 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.324 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.324 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.325 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.325 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.326 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.326 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.156765_70.1", 2022-09-09T22:59:49.327 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.327 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18", 2022-09-09T22:59:49.328 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.156765172Z", 2022-09-09T22:59:49.328 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.329 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.329 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.330 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.330 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.330 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.331 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_bCS5W1GGiGEz-gWOOFrVrp4", 2022-09-09T22:59:49.331 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.331 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.331 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.332 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.332 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.332 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.332 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.333 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.333 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.161880_71.1", 2022-09-09T22:59:49.333 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.333 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-18", 2022-09-09T22:59:49.334 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.161880990Z", 2022-09-09T22:59:49.334 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.334 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.335 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.335 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.335 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.335 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.336 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_vVrGNRKUHx_Nc7AgE45d0q7", 2022-09-09T22:59:49.336 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.336 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.337 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.337 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.337 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.338 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.338 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.338 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.339 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.166872_72.1", 2022-09-09T22:59:49.339 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.339 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-18", 2022-09-09T22:59:49.339 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.166872706Z", 2022-09-09T22:59:49.340 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.340 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.340 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.340 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.341 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.341 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.341 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_vVrGNRKUHx_Nc7AgE45d0q7", 2022-09-09T22:59:49.341 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.342 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.342 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.342 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.342 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.343 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.343 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.343 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.343 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.191982_73.1", 2022-09-09T22:59:49.344 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.344 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16", 2022-09-09T22:59:49.344 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.191982077Z", 2022-09-09T22:59:49.344 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.345 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.345 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_jgmlG80S-C76e7qmPCPXWwY", 2022-09-09T22:59:49.345 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.346 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.346 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.346 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_jgmlG80S-C76e7qmPCPXWwY", 2022-09-09T22:59:49.346 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.347 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.347 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.347 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.348 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.348 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.348 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.348 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.349 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.196936_74.1", 2022-09-09T22:59:49.349 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.349 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16", 2022-09-09T22:59:49.349 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.196936218Z", 2022-09-09T22:59:49.350 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.350 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.350 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_jgmlG80S-C76e7qmPCPXWwY", 2022-09-09T22:59:49.350 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.351 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.351 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.351 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_jgmlG80S-C76e7qmPCPXWwY", 2022-09-09T22:59:49.352 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.352 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.352 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.352 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.353 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.353 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.353 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.353 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.354 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.211489_75.1", 2022-09-09T22:59:49.354 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.354 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17", 2022-09-09T22:59:49.354 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.211489751Z", 2022-09-09T22:59:49.355 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.355 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.355 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_MtuyPaZmCzfHOcgXjri3F-V", 2022-09-09T22:59:49.355 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.356 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.356 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.356 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_MtuyPaZmCzfHOcgXjri3F-V", 2022-09-09T22:59:49.356 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.357 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.357 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.357 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.357 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.358 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.358 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.358 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.359 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.217670_76.1", 2022-09-09T22:59:49.359 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.359 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17", 2022-09-09T22:59:49.359 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.217670475Z", 2022-09-09T22:59:49.360 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.360 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.360 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_MtuyPaZmCzfHOcgXjri3F-V", 2022-09-09T22:59:49.360 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.361 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.361 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.361 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_MtuyPaZmCzfHOcgXjri3F-V", 2022-09-09T22:59:49.361 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.362 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.362 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.362 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.362 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.363 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.363 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.363 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.363 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.236343_77.1", 2022-09-09T22:59:49.364 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.364 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18", 2022-09-09T22:59:49.365 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.236343391Z", 2022-09-09T22:59:49.365 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.365 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.366 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_bCS5W1GGiGEz-gWOOFrVrp4", 2022-09-09T22:59:49.366 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.367 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.367 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.367 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_bCS5W1GGiGEz-gWOOFrVrp4", 2022-09-09T22:59:49.368 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.368 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.368 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.368 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.369 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.369 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.369 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.369 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.370 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662762884.244083_78.1", 2022-09-09T22:59:49.370 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.370 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18", 2022-09-09T22:59:49.371 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:34:44.244083785Z", 2022-09-09T22:59:49.371 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.371 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.371 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_bCS5W1GGiGEz-gWOOFrVrp4", 2022-09-09T22:59:49.372 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.372 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.372 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.372 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_bCS5W1GGiGEz-gWOOFrVrp4", 2022-09-09T22:59:49.373 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.373 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.373 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.373 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.374 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.374 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.374 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.375 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.375 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763177.690963_87.1", 2022-09-09T22:59:49.375 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.375 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19", 2022-09-09T22:59:49.376 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:39:37.690963866Z", 2022-09-09T22:59:49.376 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.377 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.377 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.377 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.378 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.378 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.378 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_ePT9JnPeiTDUMUQthdKtkWA", 2022-09-09T22:59:49.379 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.379 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.379 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.380 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.380 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.381 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.381 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.381 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.382 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763177.695434_88.1", 2022-09-09T22:59:49.382 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.383 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19", 2022-09-09T22:59:49.383 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:39:37.695434017Z", 2022-09-09T22:59:49.384 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.384 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.384 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.385 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.385 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.385 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.386 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_ePT9JnPeiTDUMUQthdKtkWA", 2022-09-09T22:59:49.386 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.386 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.386 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.387 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.387 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.387 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.388 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.388 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.388 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763177.697042_89.1", 2022-09-09T22:59:49.389 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.389 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-19", 2022-09-09T22:59:49.389 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:39:37.697042999Z", 2022-09-09T22:59:49.389 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.390 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.390 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.390 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.391 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.391 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.391 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_tYLgE0kYjoh4utIhhPKouFZ", 2022-09-09T22:59:49.392 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.392 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.392 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.392 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.393 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.393 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.393 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.394 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.394 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763177.701435_90.1", 2022-09-09T22:59:49.394 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.395 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-19", 2022-09-09T22:59:49.395 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:39:37.701435888Z", 2022-09-09T22:59:49.395 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.396 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.396 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.396 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.397 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.397 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.397 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_tYLgE0kYjoh4utIhhPKouFZ", 2022-09-09T22:59:49.398 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.398 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.398 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.398 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.399 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.399 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.399 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.399 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.400 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763177.715694_91.1", 2022-09-09T22:59:49.400 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.400 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19", 2022-09-09T22:59:49.401 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:39:37.715694533Z", 2022-09-09T22:59:49.401 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.401 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.401 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_ePT9JnPeiTDUMUQthdKtkWA", 2022-09-09T22:59:49.402 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.402 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.402 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.402 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_ePT9JnPeiTDUMUQthdKtkWA", 2022-09-09T22:59:49.403 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.403 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.403 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.403 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.404 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.404 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.404 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.404 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.405 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763177.721058_92.1", 2022-09-09T22:59:49.405 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.405 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19", 2022-09-09T22:59:49.405 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:39:37.721058937Z", 2022-09-09T22:59:49.406 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.406 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.406 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_ePT9JnPeiTDUMUQthdKtkWA", 2022-09-09T22:59:49.406 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.407 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.407 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.407 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_ePT9JnPeiTDUMUQthdKtkWA", 2022-09-09T22:59:49.407 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.408 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.408 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.408 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.409 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.409 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.409 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.409 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.410 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763618.848484_94.1", 2022-09-09T22:59:49.410 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.410 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20", 2022-09-09T22:59:49.411 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:46:58.848484735Z", 2022-09-09T22:59:49.411 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.411 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.411 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.412 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.412 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.412 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.412 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:49.413 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.413 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.413 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.413 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.414 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.414 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.414 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.414 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.415 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763618.852755_95.1", 2022-09-09T22:59:49.415 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.415 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20", 2022-09-09T22:59:49.415 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:46:58.852755664Z", 2022-09-09T22:59:49.416 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.416 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.416 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.416 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.417 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.417 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.417 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:49.417 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.418 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.418 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.418 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.419 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.419 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.419 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.419 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.420 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763618.854410_96.1", 2022-09-09T22:59:49.420 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.420 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-20", 2022-09-09T22:59:49.420 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:46:58.854410052Z", 2022-09-09T22:59:49.421 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.421 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.421 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.421 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.422 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.422 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.422 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_4dW2sFlblKAnYCXSGCT7lId", 2022-09-09T22:59:49.422 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.423 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.423 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.423 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.423 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.424 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.424 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.424 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.424 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763618.858372_97.1", 2022-09-09T22:59:49.425 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.425 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-20", 2022-09-09T22:59:49.425 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:46:58.858372321Z", 2022-09-09T22:59:49.426 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.426 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.426 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.427 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.427 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.427 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.428 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_4dW2sFlblKAnYCXSGCT7lId", 2022-09-09T22:59:49.428 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.428 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.428 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.429 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.429 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.429 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.429 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.430 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.430 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763692.030172_110.1", 2022-09-09T22:59:49.430 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.431 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20", 2022-09-09T22:59:49.431 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:48:12.030172339Z", 2022-09-09T22:59:49.431 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.431 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.432 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:49.432 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.432 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.432 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.433 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:49.433 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.433 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.433 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.434 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.434 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.434 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.435 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.435 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.435 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763692.038589_111.1", 2022-09-09T22:59:49.435 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.436 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20", 2022-09-09T22:59:49.436 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:48:12.038589581Z", 2022-09-09T22:59:49.436 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.437 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.437 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:49.437 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.437 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.438 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.438 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:49.438 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.438 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.439 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.439 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.439 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.439 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.440 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.440 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.440 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763726.102012_112.1", 2022-09-09T22:59:49.440 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.441 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20", 2022-09-09T22:59:49.441 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:48:46.102012638Z", 2022-09-09T22:59:49.441 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.441 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.442 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:49.442 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.442 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.442 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.443 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:49.443 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 3 2022-09-09T22:59:49.443 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.443 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.444 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.444 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.444 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.444 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.445 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.445 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763726.109956_113.1", 2022-09-09T22:59:49.445 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.445 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20", 2022-09-09T22:59:49.446 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:48:46.109956678Z", 2022-09-09T22:59:49.446 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.446 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.447 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:49.447 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.447 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.447 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.448 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:49.448 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 3 2022-09-09T22:59:49.448 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.448 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.449 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.449 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.449 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.449 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.450 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.450 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.234301_114.1", 2022-09-09T22:59:49.450 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.450 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21", 2022-09-09T22:59:49.451 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.234301916Z", 2022-09-09T22:59:49.451 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.451 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.452 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.452 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.452 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.453 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.453 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_vO_bfm5gHdzZApgtPNMJdSt", 2022-09-09T22:59:49.453 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.454 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.454 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.454 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.455 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.455 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.455 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.456 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.456 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.239008_115.1", 2022-09-09T22:59:49.456 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.457 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21", 2022-09-09T22:59:49.457 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.239008120Z", 2022-09-09T22:59:49.457 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.457 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.458 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.458 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.458 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.459 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.459 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_vO_bfm5gHdzZApgtPNMJdSt", 2022-09-09T22:59:49.459 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.459 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.460 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.460 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.460 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.460 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.461 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.461 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.461 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.240497_116.1", 2022-09-09T22:59:49.461 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.462 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-21", 2022-09-09T22:59:49.462 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.240497138Z", 2022-09-09T22:59:49.462 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.462 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.463 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.463 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.463 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.463 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.464 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_K_Aww59oBrAI7mXNiMdGk-A", 2022-09-09T22:59:49.464 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.464 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.464 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.465 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.465 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.465 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.465 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.466 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.466 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.244134_117.1", 2022-09-09T22:59:49.466 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.467 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-21", 2022-09-09T22:59:49.467 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.244134829Z", 2022-09-09T22:59:49.467 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.467 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.468 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.468 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.468 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.469 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.469 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_K_Aww59oBrAI7mXNiMdGk-A", 2022-09-09T22:59:49.469 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.469 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.470 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.470 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.470 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.470 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.471 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.471 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.471 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.271364_118.1", 2022-09-09T22:59:49.471 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.472 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22", 2022-09-09T22:59:49.472 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.271364170Z", 2022-09-09T22:59:49.472 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.473 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.473 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.473 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.473 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.474 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.474 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_oq35luOSMiKR6aRP4lKrHCx", 2022-09-09T22:59:49.474 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.474 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.475 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.475 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.475 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.476 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.476 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.476 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.476 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.276910_119.1", 2022-09-09T22:59:49.477 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.477 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22", 2022-09-09T22:59:49.477 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.276910204Z", 2022-09-09T22:59:49.478 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.478 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.478 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.478 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.479 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.479 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.479 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_oq35luOSMiKR6aRP4lKrHCx", 2022-09-09T22:59:49.480 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.480 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.480 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.481 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.481 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.481 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.481 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.482 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.482 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.278731_120.1", 2022-09-09T22:59:49.482 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.483 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-22", 2022-09-09T22:59:49.483 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.278731955Z", 2022-09-09T22:59:49.483 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.484 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.484 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.484 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.485 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.485 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.485 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_oBwxyUPEtECOF6uEAIe2K67", 2022-09-09T22:59:49.486 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.486 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.486 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.487 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.487 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.487 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.488 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.488 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.488 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.284005_121.1", 2022-09-09T22:59:49.489 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.489 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-22", 2022-09-09T22:59:49.489 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.284005465Z", 2022-09-09T22:59:49.489 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.490 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.490 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.490 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.491 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.491 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.491 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_oBwxyUPEtECOF6uEAIe2K67", 2022-09-09T22:59:49.492 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.492 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.492 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.492 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.493 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.493 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.493 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.494 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.494 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.330455_122.1", 2022-09-09T22:59:49.494 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.495 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23", 2022-09-09T22:59:49.495 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.330455662Z", 2022-09-09T22:59:49.496 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.496 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.496 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.496 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.497 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.497 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.497 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_TBNrJnEgH9G6t0MeX057y1f", 2022-09-09T22:59:49.497 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.498 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.504 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.504 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.505 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.505 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.505 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.506 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.506 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.335107_123.1", 2022-09-09T22:59:49.506 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.506 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23", 2022-09-09T22:59:49.507 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.335107502Z", 2022-09-09T22:59:49.507 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.507 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.507 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.508 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.508 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.508 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.508 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_TBNrJnEgH9G6t0MeX057y1f", 2022-09-09T22:59:49.509 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.509 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.509 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.509 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.510 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.510 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.510 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.510 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.511 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.336855_124.1", 2022-09-09T22:59:49.511 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.511 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-23", 2022-09-09T22:59:49.511 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.336855486Z", 2022-09-09T22:59:49.512 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.512 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.512 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.513 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.513 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.513 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.513 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_DGKEfxrBasYDPbGTJt7zCFo", 2022-09-09T22:59:49.514 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.514 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.514 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.515 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.515 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.515 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.515 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.516 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.516 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.340827_125.1", 2022-09-09T22:59:49.516 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.516 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-23", 2022-09-09T22:59:49.517 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.340827648Z", 2022-09-09T22:59:49.517 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.517 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.517 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.518 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.518 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.518 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.518 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_DGKEfxrBasYDPbGTJt7zCFo", 2022-09-09T22:59:49.519 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.519 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.519 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.519 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.520 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.520 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.520 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.521 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.521 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.368888_126.1", 2022-09-09T22:59:49.521 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.522 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21", 2022-09-09T22:59:49.522 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.368888518Z", 2022-09-09T22:59:49.522 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.522 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.523 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_vO_bfm5gHdzZApgtPNMJdSt", 2022-09-09T22:59:49.523 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.523 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.523 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.524 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_vO_bfm5gHdzZApgtPNMJdSt", 2022-09-09T22:59:49.524 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.524 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.524 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.525 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.525 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.525 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.526 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.526 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.526 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.374441_127.1", 2022-09-09T22:59:49.527 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.527 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21", 2022-09-09T22:59:49.527 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.374441110Z", 2022-09-09T22:59:49.528 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.528 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.528 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_vO_bfm5gHdzZApgtPNMJdSt", 2022-09-09T22:59:49.529 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.529 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.529 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.529 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_vO_bfm5gHdzZApgtPNMJdSt", 2022-09-09T22:59:49.530 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.530 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.530 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.531 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.531 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.531 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.531 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.532 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.532 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.389455_128.1", 2022-09-09T22:59:49.532 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.532 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22", 2022-09-09T22:59:49.533 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.389455212Z", 2022-09-09T22:59:49.533 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.533 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.533 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_oq35luOSMiKR6aRP4lKrHCx", 2022-09-09T22:59:49.534 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.534 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.534 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.534 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_oq35luOSMiKR6aRP4lKrHCx", 2022-09-09T22:59:49.535 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.535 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.535 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.535 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.536 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.536 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.536 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.536 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.537 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.396062_129.1", 2022-09-09T22:59:49.537 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.537 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22", 2022-09-09T22:59:49.537 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.396062782Z", 2022-09-09T22:59:49.538 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.538 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.538 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_oq35luOSMiKR6aRP4lKrHCx", 2022-09-09T22:59:49.538 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.539 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.539 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.539 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_oq35luOSMiKR6aRP4lKrHCx", 2022-09-09T22:59:49.539 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.540 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.540 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.540 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.540 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.541 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.541 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.541 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.541 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.417475_130.1", 2022-09-09T22:59:49.542 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.542 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23", 2022-09-09T22:59:49.542 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.417475123Z", 2022-09-09T22:59:49.542 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.543 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.543 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_TBNrJnEgH9G6t0MeX057y1f", 2022-09-09T22:59:49.543 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.543 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.544 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.544 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_TBNrJnEgH9G6t0MeX057y1f", 2022-09-09T22:59:49.544 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.544 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.545 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.545 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.545 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.546 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.546 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.546 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.546 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763763.426933_131.1", 2022-09-09T22:59:49.547 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.547 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23", 2022-09-09T22:59:49.547 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:49:23.426933442Z", 2022-09-09T22:59:49.547 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.548 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.548 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_TBNrJnEgH9G6t0MeX057y1f", 2022-09-09T22:59:49.548 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.548 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.549 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.549 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_TBNrJnEgH9G6t0MeX057y1f", 2022-09-09T22:59:49.549 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.549 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.550 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.550 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.550 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.550 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.551 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.551 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.551 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.243136_133.1", 2022-09-09T22:59:49.551 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.552 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24", 2022-09-09T22:59:49.552 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.243136650Z", 2022-09-09T22:59:49.552 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.552 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.553 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.553 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.553 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.553 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.554 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:49.554 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.554 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.554 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.555 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.555 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.555 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.555 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.556 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.556 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.247428_134.1", 2022-09-09T22:59:49.556 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.557 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24", 2022-09-09T22:59:49.557 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.247428396Z", 2022-09-09T22:59:49.557 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.557 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.558 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.558 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.558 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.558 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.559 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:49.559 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.559 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.559 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.560 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.560 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.560 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.560 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.561 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.561 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.249044_135.1", 2022-09-09T22:59:49.561 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.561 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-24", 2022-09-09T22:59:49.562 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.249044082Z", 2022-09-09T22:59:49.562 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.562 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.562 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.563 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.563 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.563 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.563 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_GPbPaiqkbVjC5D9wQQQP2jZ", 2022-09-09T22:59:49.564 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.564 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.564 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.564 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.565 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.565 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.565 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.565 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.566 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.253611_136.1", 2022-09-09T22:59:49.566 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.566 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-24", 2022-09-09T22:59:49.566 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.253611226Z", 2022-09-09T22:59:49.567 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.567 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.567 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.567 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.568 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.568 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.568 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_GPbPaiqkbVjC5D9wQQQP2jZ", 2022-09-09T22:59:49.568 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.569 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.569 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.569 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.570 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.570 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.570 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.570 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.571 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.282622_137.1", 2022-09-09T22:59:49.571 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.571 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25", 2022-09-09T22:59:49.571 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.282622692Z", 2022-09-09T22:59:49.572 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.572 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.572 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.572 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.573 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.573 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.573 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:49.574 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.574 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.574 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.574 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.575 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.575 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.575 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.575 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.576 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.287087_138.1", 2022-09-09T22:59:49.576 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.576 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25", 2022-09-09T22:59:49.576 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.287087929Z", 2022-09-09T22:59:49.577 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.577 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.577 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.577 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.578 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.578 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.578 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:49.578 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.579 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.579 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.579 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.579 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.580 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.580 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.580 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.580 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.288605_139.1", 2022-09-09T22:59:49.581 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.581 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-25", 2022-09-09T22:59:49.581 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.288605630Z", 2022-09-09T22:59:49.581 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.582 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.582 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.582 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.582 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.583 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.583 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_ZPxDZ2lg7jpxqasfBs-kwky", 2022-09-09T22:59:49.583 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.583 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.584 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.584 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.584 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.584 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.585 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.585 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.585 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.292651_140.1", 2022-09-09T22:59:49.585 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.586 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-25", 2022-09-09T22:59:49.586 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.292651882Z", 2022-09-09T22:59:49.586 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.586 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.587 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.587 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.587 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.587 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.588 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_ZPxDZ2lg7jpxqasfBs-kwky", 2022-09-09T22:59:49.588 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.588 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.588 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.589 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.589 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.589 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.589 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.590 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.590 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.347788_141.1", 2022-09-09T22:59:49.590 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.590 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26", 2022-09-09T22:59:49.591 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.347788063Z", 2022-09-09T22:59:49.591 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.591 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.591 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.592 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.592 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.592 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.592 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:49.593 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.593 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.593 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.594 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.594 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.594 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.594 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.595 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.595 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.352949_142.1", 2022-09-09T22:59:49.595 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.595 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26", 2022-09-09T22:59:49.596 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.352949747Z", 2022-09-09T22:59:49.596 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.596 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.596 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.597 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.597 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.597 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.597 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:49.598 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.598 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.598 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.598 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.599 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.599 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.599 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.599 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.600 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.354746_143.1", 2022-09-09T22:59:49.600 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.600 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-26", 2022-09-09T22:59:49.600 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.354746994Z", 2022-09-09T22:59:49.601 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.601 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.601 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.601 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.602 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.602 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_sgAtkuvW-X_3Moy5BHrhaBp", 2022-09-09T22:59:49.602 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.603 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.603 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.603 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.603 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.604 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.604 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.604 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.604 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.358613_144.1", 2022-09-09T22:59:49.605 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.605 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-26", 2022-09-09T22:59:49.605 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.358613174Z", 2022-09-09T22:59:49.605 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.606 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.606 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.606 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.606 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.607 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.607 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_sgAtkuvW-X_3Moy5BHrhaBp", 2022-09-09T22:59:49.607 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.608 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.608 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.608 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.608 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.609 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.609 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.609 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.386362_145.1", 2022-09-09T22:59:49.609 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.610 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24", 2022-09-09T22:59:49.610 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.386362667Z", 2022-09-09T22:59:49.610 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.610 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.611 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:49.611 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.611 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.612 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.612 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:49.612 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.612 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.613 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.613 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.613 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.613 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.614 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.614 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.614 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.392952_146.1", 2022-09-09T22:59:49.614 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.615 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24", 2022-09-09T22:59:49.615 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.392952544Z", 2022-09-09T22:59:49.615 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.615 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.616 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:49.616 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.616 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.616 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.617 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:49.617 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.617 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.617 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.618 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.618 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.618 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.618 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.619 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.619 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.405357_147.1", 2022-09-09T22:59:49.619 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.619 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24", 2022-09-09T22:59:49.620 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.405357667Z", 2022-09-09T22:59:49.620 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.620 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.620 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:49.621 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.621 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.621 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.621 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:49.622 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 3 2022-09-09T22:59:49.622 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.622 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.623 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.623 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.623 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.623 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.624 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.624 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.413164_148.1", 2022-09-09T22:59:49.624 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.624 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24", 2022-09-09T22:59:49.625 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.413164542Z", 2022-09-09T22:59:49.625 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.625 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.625 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:49.626 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.626 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.626 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.626 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:49.627 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 3 2022-09-09T22:59:49.627 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.627 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.627 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.628 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.628 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.628 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.628 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.628 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.428806_149.1", 2022-09-09T22:59:49.629 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.629 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25", 2022-09-09T22:59:49.629 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.428806301Z", 2022-09-09T22:59:49.630 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.630 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.630 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:49.630 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.631 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.631 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.631 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:49.631 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.632 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.632 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.632 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.632 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.633 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.633 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.633 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.633 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.436953_150.1", 2022-09-09T22:59:49.634 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.634 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25", 2022-09-09T22:59:49.634 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.436953749Z", 2022-09-09T22:59:49.634 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.635 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.635 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:49.635 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.635 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.636 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.636 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:49.636 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.637 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.637 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.637 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.637 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.638 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.638 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.638 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.638 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.460100_151.1", 2022-09-09T22:59:49.639 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.639 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25", 2022-09-09T22:59:49.639 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.460100101Z", 2022-09-09T22:59:49.639 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.640 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.640 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:49.640 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.640 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.641 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.641 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:49.641 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 3 2022-09-09T22:59:49.641 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.642 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.642 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.642 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.642 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.643 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.643 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.643 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.471650_152.1", 2022-09-09T22:59:49.643 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.644 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25", 2022-09-09T22:59:49.644 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.471650692Z", 2022-09-09T22:59:49.644 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.644 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.645 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:49.645 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.645 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.645 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.646 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:49.646 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 3 2022-09-09T22:59:49.646 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.646 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.647 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.647 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.647 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.647 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.648 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.648 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.513547_153.1", 2022-09-09T22:59:49.648 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.648 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26", 2022-09-09T22:59:49.649 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.513547080Z", 2022-09-09T22:59:49.649 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.649 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.649 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:49.650 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.650 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.650 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.650 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:49.651 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.651 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.651 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.651 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.652 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.652 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.652 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.652 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.653 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.520151_154.1", 2022-09-09T22:59:49.653 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.653 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26", 2022-09-09T22:59:49.653 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.520151282Z", 2022-09-09T22:59:49.654 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.654 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.654 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:49.654 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.655 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.655 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.655 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:49.655 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.656 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.656 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.656 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.656 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.657 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.657 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.657 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.657 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.543475_155.1", 2022-09-09T22:59:49.658 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.658 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26", 2022-09-09T22:59:49.658 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.543475276Z", 2022-09-09T22:59:49.659 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.659 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.659 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:49.659 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.660 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.660 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.660 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:49.660 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 3 2022-09-09T22:59:49.661 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.661 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.661 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.661 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.662 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.662 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.662 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.663 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763826.550710_156.1", 2022-09-09T22:59:49.663 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.663 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26", 2022-09-09T22:59:49.663 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:26.550710511Z", 2022-09-09T22:59:49.664 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.664 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.664 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:49.664 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.665 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.665 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.665 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:49.665 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 3 2022-09-09T22:59:49.666 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.666 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.666 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.666 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.667 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.667 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.667 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.667 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.587061_165.1", 2022-09-09T22:59:49.668 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.668 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-27:4b05608c-7197-4e9b-bd29-7989607fa277.4354.27", 2022-09-09T22:59:49.668 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.587061072Z", 2022-09-09T22:59:49.668 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.669 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.669 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.669 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.669 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.670 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.670 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_idGRpIZD_cBqHmSr0C8anAE", 2022-09-09T22:59:49.670 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.670 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.671 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.671 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.671 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.671 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.672 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.672 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.672 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.591517_166.1", 2022-09-09T22:59:49.672 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.673 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-27:4b05608c-7197-4e9b-bd29-7989607fa277.4354.27", 2022-09-09T22:59:49.673 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.591517503Z", 2022-09-09T22:59:49.673 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.673 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.674 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.674 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.674 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.675 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.675 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_idGRpIZD_cBqHmSr0C8anAE", 2022-09-09T22:59:49.675 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.675 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.675 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.676 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.676 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.676 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.676 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.677 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.677 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.593291_167.1", 2022-09-09T22:59:49.677 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.677 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-27", 2022-09-09T22:59:49.678 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.593291391Z", 2022-09-09T22:59:49.678 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.678 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.679 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.679 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.679 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.679 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.680 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_fil3WVqAmCZtFEsYnU9kPha", 2022-09-09T22:59:49.680 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.680 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.680 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.681 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.681 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.681 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.681 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.682 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.682 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.597347_168.1", 2022-09-09T22:59:49.682 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.682 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-27", 2022-09-09T22:59:49.683 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.597347657Z", 2022-09-09T22:59:49.683 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.683 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.683 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.684 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.684 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.684 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.684 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_fil3WVqAmCZtFEsYnU9kPha", 2022-09-09T22:59:49.685 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.685 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.685 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.685 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.686 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.686 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.686 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.686 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.629370_169.1", 2022-09-09T22:59:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-28:4b05608c-7197-4e9b-bd29-7989607fa277.4354.28", 2022-09-09T22:59:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.629370653Z", 2022-09-09T22:59:49.688 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.688 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.688 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.689 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.689 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.689 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.689 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_hWVXDIs2ffmJdaY5i2EUA49", 2022-09-09T22:59:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.692 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.634323_170.1", 2022-09-09T22:59:49.692 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.692 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-28:4b05608c-7197-4e9b-bd29-7989607fa277.4354.28", 2022-09-09T22:59:49.692 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.634323436Z", 2022-09-09T22:59:49.693 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.693 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.693 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.693 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_hWVXDIs2ffmJdaY5i2EUA49", 2022-09-09T22:59:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.695 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.695 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.695 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.636142_171.1", 2022-09-09T22:59:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-28", 2022-09-09T22:59:49.698 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.636142661Z", 2022-09-09T22:59:49.698 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.698 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.699 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.699 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.699 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.700 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.700 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_Rr2OoVgGeS8W2Ih7H5QimUB", 2022-09-09T22:59:49.700 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.701 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.701 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.701 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.702 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.702 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.702 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.702 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.703 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.640366_172.1", 2022-09-09T22:59:49.703 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.703 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-28", 2022-09-09T22:59:49.703 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.640366460Z", 2022-09-09T22:59:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.705 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.705 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.705 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_Rr2OoVgGeS8W2Ih7H5QimUB", 2022-09-09T22:59:49.705 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.708 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.691642_173.1", 2022-09-09T22:59:49.708 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.708 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-29:4b05608c-7197-4e9b-bd29-7989607fa277.4354.29", 2022-09-09T22:59:49.708 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.691642561Z", 2022-09-09T22:59:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.710 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.710 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_c7Bu5fk9awm3ApVCIJ8jMq6", 2022-09-09T22:59:49.710 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.695580_174.1", 2022-09-09T22:59:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-29:4b05608c-7197-4e9b-bd29-7989607fa277.4354.29", 2022-09-09T22:59:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.695580156Z", 2022-09-09T22:59:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_c7Bu5fk9awm3ApVCIJ8jMq6", 2022-09-09T22:59:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.696997_175.1", 2022-09-09T22:59:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.718 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-29", 2022-09-09T22:59:49.718 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.696997195Z", 2022-09-09T22:59:49.718 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.718 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_oMWT857Kx25Jy8rmeTqaRTI", 2022-09-09T22:59:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.722 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.722 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.701289_176.1", 2022-09-09T22:59:49.722 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.722 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-29", 2022-09-09T22:59:49.723 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.701289662Z", 2022-09-09T22:59:49.723 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.723 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.723 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.724 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.724 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.724 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.724 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_oMWT857Kx25Jy8rmeTqaRTI", 2022-09-09T22:59:49.725 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.725 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.725 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.725 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.733662_177.1", 2022-09-09T22:59:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-27:4b05608c-7197-4e9b-bd29-7989607fa277.4354.27", 2022-09-09T22:59:49.728 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.733662223Z", 2022-09-09T22:59:49.728 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.728 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.728 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_idGRpIZD_cBqHmSr0C8anAE", 2022-09-09T22:59:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_idGRpIZD_cBqHmSr0C8anAE", 2022-09-09T22:59:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.731 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.731 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.731 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.731 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.739796_178.1", 2022-09-09T22:59:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-27:4b05608c-7197-4e9b-bd29-7989607fa277.4354.27", 2022-09-09T22:59:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.739796617Z", 2022-09-09T22:59:49.733 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.733 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.733 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_idGRpIZD_cBqHmSr0C8anAE", 2022-09-09T22:59:49.733 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_idGRpIZD_cBqHmSr0C8anAE", 2022-09-09T22:59:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.735 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.735 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.735 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.735 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.736 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.736 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.736 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.736 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.815014_179.1", 2022-09-09T22:59:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-28:4b05608c-7197-4e9b-bd29-7989607fa277.4354.28", 2022-09-09T22:59:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.815014271Z", 2022-09-09T22:59:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.738 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.738 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_hWVXDIs2ffmJdaY5i2EUA49", 2022-09-09T22:59:49.738 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_hWVXDIs2ffmJdaY5i2EUA49", 2022-09-09T22:59:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.740 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.740 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.740 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.740 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.740 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.741 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.741 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.741 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.823504_180.1", 2022-09-09T22:59:49.742 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.742 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-28:4b05608c-7197-4e9b-bd29-7989607fa277.4354.28", 2022-09-09T22:59:49.742 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.823504725Z", 2022-09-09T22:59:49.742 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.743 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.743 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_hWVXDIs2ffmJdaY5i2EUA49", 2022-09-09T22:59:49.743 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.743 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_hWVXDIs2ffmJdaY5i2EUA49", 2022-09-09T22:59:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.746 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.746 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.746 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.861162_181.1", 2022-09-09T22:59:49.747 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.747 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-29:4b05608c-7197-4e9b-bd29-7989607fa277.4354.29", 2022-09-09T22:59:49.747 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.861162453Z", 2022-09-09T22:59:49.747 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.748 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.748 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_c7Bu5fk9awm3ApVCIJ8jMq6", 2022-09-09T22:59:49.748 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.748 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_c7Bu5fk9awm3ApVCIJ8jMq6", 2022-09-09T22:59:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.750 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.750 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.750 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.750 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.868518_182.1", 2022-09-09T22:59:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-29:4b05608c-7197-4e9b-bd29-7989607fa277.4354.29", 2022-09-09T22:59:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.868518409Z", 2022-09-09T22:59:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.753 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_c7Bu5fk9awm3ApVCIJ8jMq6", 2022-09-09T22:59:49.753 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.753 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_c7Bu5fk9awm3ApVCIJ8jMq6", 2022-09-09T22:59:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.755 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.755 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.755 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.755 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.905416_183.1", 2022-09-09T22:59:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-30:4b05608c-7197-4e9b-bd29-7989607fa277.4354.30", 2022-09-09T22:59:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.905416190Z", 2022-09-09T22:59:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.759 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.759 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xuD0Sb75U_5LCkKCpVp22OC", 2022-09-09T22:59:49.759 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.759 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.759 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.909698_184.1", 2022-09-09T22:59:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-30:4b05608c-7197-4e9b-bd29-7989607fa277.4354.30", 2022-09-09T22:59:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.909698021Z", 2022-09-09T22:59:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xuD0Sb75U_5LCkKCpVp22OC", 2022-09-09T22:59:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.765 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.765 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.765 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.765 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.911381_185.1", 2022-09-09T22:59:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-30", 2022-09-09T22:59:49.767 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.911381624Z", 2022-09-09T22:59:49.767 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.767 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_rADCtMkJa27KS0fuztGkOJ4", 2022-09-09T22:59:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.915543_186.1", 2022-09-09T22:59:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-30", 2022-09-09T22:59:49.772 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.915543203Z", 2022-09-09T22:59:49.772 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.772 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.772 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_rADCtMkJa27KS0fuztGkOJ4", 2022-09-09T22:59:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.775 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.775 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.775 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.775 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.957457_187.1", 2022-09-09T22:59:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-31:4b05608c-7197-4e9b-bd29-7989607fa277.4354.31", 2022-09-09T22:59:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.957457179Z", 2022-09-09T22:59:49.777 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.777 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.777 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.777 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.778 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.778 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.778 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_RG9oYzLzZKilowVQDrZqfZp", 2022-09-09T22:59:49.778 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.779 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.779 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.779 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.780 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.780 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.780 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.780 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.961791_188.1", 2022-09-09T22:59:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-31:4b05608c-7197-4e9b-bd29-7989607fa277.4354.31", 2022-09-09T22:59:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.961791758Z", 2022-09-09T22:59:49.782 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.782 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.782 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.782 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_RG9oYzLzZKilowVQDrZqfZp", 2022-09-09T22:59:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.963310_189.1", 2022-09-09T22:59:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-31", 2022-09-09T22:59:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.963310744Z", 2022-09-09T22:59:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_u7CGf9AlYIHVBDPwmE_8lDH", 2022-09-09T22:59:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.789 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.789 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.789 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.790 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.790 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.790 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763987.966754_190.1", 2022-09-09T22:59:49.790 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.791 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-31", 2022-09-09T22:59:49.791 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:07.966754250Z", 2022-09-09T22:59:49.791 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.791 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.792 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.792 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.792 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.792 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.793 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_u7CGf9AlYIHVBDPwmE_8lDH", 2022-09-09T22:59:49.793 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.793 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.793 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.794 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.794 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.794 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.794 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.795 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.795 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.029534_191.1", 2022-09-09T22:59:49.795 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.795 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-32:4b05608c-7197-4e9b-bd29-7989607fa277.4354.32", 2022-09-09T22:59:49.796 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.029534998Z", 2022-09-09T22:59:49.796 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.796 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.796 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.797 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.797 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.797 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.797 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_Uke9qkxor72jlh3_r-i6E2w", 2022-09-09T22:59:49.798 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.798 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.798 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.798 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.799 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.799 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.799 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.799 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.800 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.036126_192.1", 2022-09-09T22:59:49.800 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.800 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-32:4b05608c-7197-4e9b-bd29-7989607fa277.4354.32", 2022-09-09T22:59:49.800 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.036126389Z", 2022-09-09T22:59:49.801 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.801 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.801 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.801 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.802 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.802 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.802 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_Uke9qkxor72jlh3_r-i6E2w", 2022-09-09T22:59:49.802 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.803 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.803 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.803 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.803 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.804 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.804 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.804 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.804 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.040740_193.1", 2022-09-09T22:59:49.805 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.805 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-32", 2022-09-09T22:59:49.805 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.040740810Z", 2022-09-09T22:59:49.805 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.806 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.806 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.806 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.806 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.807 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.807 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_SYdiOqvbUDZ5DpIXlyitzm5", 2022-09-09T22:59:49.807 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.807 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.808 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.808 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.808 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.808 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.809 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.809 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.809 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.045233_194.1", 2022-09-09T22:59:49.809 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.810 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-32", 2022-09-09T22:59:49.810 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.045233474Z", 2022-09-09T22:59:49.810 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.810 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.811 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.811 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.811 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.811 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.812 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_SYdiOqvbUDZ5DpIXlyitzm5", 2022-09-09T22:59:49.812 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.812 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.813 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.813 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.813 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.813 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.814 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.814 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.814 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.086181_195.1", 2022-09-09T22:59:49.814 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.815 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-30:4b05608c-7197-4e9b-bd29-7989607fa277.4354.30", 2022-09-09T22:59:49.815 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.086181840Z", 2022-09-09T22:59:49.815 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.815 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.816 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xuD0Sb75U_5LCkKCpVp22OC", 2022-09-09T22:59:49.816 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.816 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.816 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.817 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xuD0Sb75U_5LCkKCpVp22OC", 2022-09-09T22:59:49.817 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.817 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.817 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.818 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.818 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.818 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.818 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.819 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.819 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.092819_196.1", 2022-09-09T22:59:49.819 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.819 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-30:4b05608c-7197-4e9b-bd29-7989607fa277.4354.30", 2022-09-09T22:59:49.820 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.092819829Z", 2022-09-09T22:59:49.820 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.820 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.820 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xuD0Sb75U_5LCkKCpVp22OC", 2022-09-09T22:59:49.821 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.821 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.821 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.821 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xuD0Sb75U_5LCkKCpVp22OC", 2022-09-09T22:59:49.822 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.822 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.822 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.823 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.823 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.823 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.823 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.824 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.824 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.131992_197.1", 2022-09-09T22:59:49.824 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.824 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-31:4b05608c-7197-4e9b-bd29-7989607fa277.4354.31", 2022-09-09T22:59:49.825 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.131992122Z", 2022-09-09T22:59:49.825 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.825 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.825 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_RG9oYzLzZKilowVQDrZqfZp", 2022-09-09T22:59:49.826 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.826 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.826 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.826 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_RG9oYzLzZKilowVQDrZqfZp", 2022-09-09T22:59:49.827 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.827 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.827 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.827 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.828 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.828 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.828 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.828 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.829 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.140406_198.1", 2022-09-09T22:59:49.829 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.829 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-31:4b05608c-7197-4e9b-bd29-7989607fa277.4354.31", 2022-09-09T22:59:49.829 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.140406171Z", 2022-09-09T22:59:49.830 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.830 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.830 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_RG9oYzLzZKilowVQDrZqfZp", 2022-09-09T22:59:49.830 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.831 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.831 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.831 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_RG9oYzLzZKilowVQDrZqfZp", 2022-09-09T22:59:49.831 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.832 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.832 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.832 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.832 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.833 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.833 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.833 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.833 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.234470_199.1", 2022-09-09T22:59:49.834 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.834 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-32:4b05608c-7197-4e9b-bd29-7989607fa277.4354.32", 2022-09-09T22:59:49.834 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.234470787Z", 2022-09-09T22:59:49.835 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.835 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.835 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_Uke9qkxor72jlh3_r-i6E2w", 2022-09-09T22:59:49.835 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.836 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.836 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.836 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_Uke9qkxor72jlh3_r-i6E2w", 2022-09-09T22:59:49.836 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.837 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.837 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.837 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.837 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.838 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.838 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.838 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.838 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.242680_200.1", 2022-09-09T22:59:49.839 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.839 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-32:4b05608c-7197-4e9b-bd29-7989607fa277.4354.32", 2022-09-09T22:59:49.839 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.242680646Z", 2022-09-09T22:59:49.839 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.840 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.840 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_Uke9qkxor72jlh3_r-i6E2w", 2022-09-09T22:59:49.840 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.840 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.841 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.841 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_Uke9qkxor72jlh3_r-i6E2w", 2022-09-09T22:59:49.841 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.841 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.842 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.842 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.842 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.842 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.843 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.843 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.843 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.342621_201.1", 2022-09-09T22:59:49.843 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.844 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-33:4b05608c-7197-4e9b-bd29-7989607fa277.4354.33", 2022-09-09T22:59:49.844 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.342621802Z", 2022-09-09T22:59:49.844 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.844 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.845 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.845 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.845 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.845 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.846 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xQnGVMAoouaLAz4qR87BBiu", 2022-09-09T22:59:49.846 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.846 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.846 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.847 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.847 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.847 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.847 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.848 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.848 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.348018_202.1", 2022-09-09T22:59:49.848 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.848 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-33:4b05608c-7197-4e9b-bd29-7989607fa277.4354.33", 2022-09-09T22:59:49.849 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.348018686Z", 2022-09-09T22:59:49.849 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.849 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.849 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.850 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.850 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.850 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.850 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xQnGVMAoouaLAz4qR87BBiu", 2022-09-09T22:59:49.851 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.851 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.851 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.851 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.852 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.852 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.852 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.852 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.853 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.352316_203.1", 2022-09-09T22:59:49.853 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.853 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-33", 2022-09-09T22:59:49.853 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.352316326Z", 2022-09-09T22:59:49.854 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.854 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.854 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.854 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.855 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.855 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.855 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_py9xB0upsDT3rp1yJKL0fW7", 2022-09-09T22:59:49.855 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.856 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.856 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.856 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.856 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.857 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.857 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.857 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.857 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.367301_204.1", 2022-09-09T22:59:49.858 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.858 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-33", 2022-09-09T22:59:49.858 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.367301625Z", 2022-09-09T22:59:49.858 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.859 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.859 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.859 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.859 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.860 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.860 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_py9xB0upsDT3rp1yJKL0fW7", 2022-09-09T22:59:49.860 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.860 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.861 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.861 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.861 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.861 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.862 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.862 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.862 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.433156_205.1", 2022-09-09T22:59:49.862 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.863 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-34:4b05608c-7197-4e9b-bd29-7989607fa277.4354.34", 2022-09-09T22:59:49.863 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.433156697Z", 2022-09-09T22:59:49.863 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.863 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.864 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.864 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.864 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.864 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.865 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_4_jQEFJZa-xniCNap3y5x4k", 2022-09-09T22:59:49.865 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.865 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.865 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.865 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.866 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.866 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.866 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.867 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.867 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.438238_206.1", 2022-09-09T22:59:49.867 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.867 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-34:4b05608c-7197-4e9b-bd29-7989607fa277.4354.34", 2022-09-09T22:59:49.868 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.438238112Z", 2022-09-09T22:59:49.868 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.868 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.868 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.869 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.869 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.869 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.869 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_4_jQEFJZa-xniCNap3y5x4k", 2022-09-09T22:59:49.870 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.870 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.870 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.870 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.871 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.871 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.871 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.871 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.872 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.439911_207.1", 2022-09-09T22:59:49.872 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.872 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-34", 2022-09-09T22:59:49.872 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.439911797Z", 2022-09-09T22:59:49.873 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.873 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.873 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.873 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.874 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.874 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.874 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_i4pt2ScTLCpaSy-sTWzcDsg", 2022-09-09T22:59:49.874 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.875 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.875 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.875 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.875 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.876 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.876 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.876 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.876 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.444301_208.1", 2022-09-09T22:59:49.877 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.877 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-34", 2022-09-09T22:59:49.877 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.444301272Z", 2022-09-09T22:59:49.877 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.878 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.878 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.878 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.878 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.879 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.879 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_i4pt2ScTLCpaSy-sTWzcDsg", 2022-09-09T22:59:49.879 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.879 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.880 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.880 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.880 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.880 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.881 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.881 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.881 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.490350_209.1", 2022-09-09T22:59:49.881 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.882 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-35:4b05608c-7197-4e9b-bd29-7989607fa277.4354.35", 2022-09-09T22:59:49.882 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.490350184Z", 2022-09-09T22:59:49.882 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.882 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.883 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.883 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.883 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.883 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.884 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xarcxlcdFlngGuAvxUBGTlW", 2022-09-09T22:59:49.884 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.884 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.884 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.885 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.885 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.885 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.885 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.886 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.886 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.495064_210.1", 2022-09-09T22:59:49.886 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.887 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-35:4b05608c-7197-4e9b-bd29-7989607fa277.4354.35", 2022-09-09T22:59:49.887 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.495064159Z", 2022-09-09T22:59:49.887 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.887 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.887 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.888 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.888 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.888 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.888 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xarcxlcdFlngGuAvxUBGTlW", 2022-09-09T22:59:49.889 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.889 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.889 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.890 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.890 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.890 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.890 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.891 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.891 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.496525_211.1", 2022-09-09T22:59:49.891 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.891 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-35", 2022-09-09T22:59:49.892 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.496525235Z", 2022-09-09T22:59:49.892 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.892 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.892 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.893 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.893 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.893 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.893 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_b22QH41XyKk67qZyODLiaOx", 2022-09-09T22:59:49.894 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.894 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.894 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.894 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.895 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.895 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.895 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.895 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.896 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.500218_212.1", 2022-09-09T22:59:49.896 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.896 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-35", 2022-09-09T22:59:49.896 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.500218886Z", 2022-09-09T22:59:49.897 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.897 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.897 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.897 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.898 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.898 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.898 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_b22QH41XyKk67qZyODLiaOx", 2022-09-09T22:59:49.898 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.899 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.899 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.899 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.899 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.900 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.900 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.900 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.900 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.527104_213.1", 2022-09-09T22:59:49.901 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.901 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-33:4b05608c-7197-4e9b-bd29-7989607fa277.4354.33", 2022-09-09T22:59:49.901 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.527104670Z", 2022-09-09T22:59:49.901 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.902 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.902 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xQnGVMAoouaLAz4qR87BBiu", 2022-09-09T22:59:49.902 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.903 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.903 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.903 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xQnGVMAoouaLAz4qR87BBiu", 2022-09-09T22:59:49.903 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.904 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.904 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.904 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.904 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.905 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.905 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.905 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.905 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763988.533014_214.1", 2022-09-09T22:59:49.906 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.906 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-33:4b05608c-7197-4e9b-bd29-7989607fa277.4354.33", 2022-09-09T22:59:49.906 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:08.533014771Z", 2022-09-09T22:59:49.906 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.907 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.907 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xQnGVMAoouaLAz4qR87BBiu", 2022-09-09T22:59:49.907 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.907 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.908 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.908 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xQnGVMAoouaLAz4qR87BBiu", 2022-09-09T22:59:49.908 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.908 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.909 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.909 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.909 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.909 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.910 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.910 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.910 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763989.558518_215.1", 2022-09-09T22:59:49.910 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.911 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-34:4b05608c-7197-4e9b-bd29-7989607fa277.4354.34", 2022-09-09T22:59:49.911 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:09.558518140Z", 2022-09-09T22:59:49.911 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.911 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.912 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_4_jQEFJZa-xniCNap3y5x4k", 2022-09-09T22:59:49.912 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.912 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.912 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.913 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_4_jQEFJZa-xniCNap3y5x4k", 2022-09-09T22:59:49.913 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.913 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.913 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.914 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.914 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.914 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.914 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.915 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.915 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763989.567553_216.1", 2022-09-09T22:59:49.915 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.915 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-34:4b05608c-7197-4e9b-bd29-7989607fa277.4354.34", 2022-09-09T22:59:49.916 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:09.567553622Z", 2022-09-09T22:59:49.916 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.916 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.916 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_4_jQEFJZa-xniCNap3y5x4k", 2022-09-09T22:59:49.917 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.917 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.917 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.917 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_4_jQEFJZa-xniCNap3y5x4k", 2022-09-09T22:59:49.918 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.918 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.918 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.918 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.919 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.919 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.919 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.919 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.920 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763990.604998_217.1", 2022-09-09T22:59:49.920 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.920 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-35:4b05608c-7197-4e9b-bd29-7989607fa277.4354.35", 2022-09-09T22:59:49.920 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:10.604998041Z", 2022-09-09T22:59:49.921 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.921 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.921 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xarcxlcdFlngGuAvxUBGTlW", 2022-09-09T22:59:49.921 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.922 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.922 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.922 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xarcxlcdFlngGuAvxUBGTlW", 2022-09-09T22:59:49.922 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.923 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.923 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.923 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.923 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.924 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.924 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.924 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.924 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763990.613292_218.1", 2022-09-09T22:59:49.925 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.925 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-35:4b05608c-7197-4e9b-bd29-7989607fa277.4354.35", 2022-09-09T22:59:49.925 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:10.613292272Z", 2022-09-09T22:59:49.926 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.926 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.926 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xarcxlcdFlngGuAvxUBGTlW", 2022-09-09T22:59:49.926 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.927 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.927 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.927 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xarcxlcdFlngGuAvxUBGTlW", 2022-09-09T22:59:49.927 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 2 2022-09-09T22:59:49.928 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.928 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.928 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.928 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.929 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.929 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.929 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.929 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763990.663150_219.1", 2022-09-09T22:59:49.930 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.930 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36", 2022-09-09T22:59:49.930 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:10.663150053Z", 2022-09-09T22:59:49.930 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.931 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.931 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.931 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.931 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.932 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.932 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_jPN1kAi0u5MPj33UDl-gVII", 2022-09-09T22:59:49.932 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.932 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.933 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.933 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.933 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.933 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.934 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.934 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.934 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763990.667756_220.1", 2022-09-09T22:59:49.934 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.935 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36", 2022-09-09T22:59:49.935 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:10.667756273Z", 2022-09-09T22:59:49.935 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.935 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.936 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.936 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.936 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.936 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.937 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_jPN1kAi0u5MPj33UDl-gVII", 2022-09-09T22:59:49.937 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.937 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.937 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.938 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.938 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.938 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.938 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.939 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.939 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763990.669560_221.1", 2022-09-09T22:59:49.939 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.939 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-36", 2022-09-09T22:59:49.940 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:10.669560632Z", 2022-09-09T22:59:49.940 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.940 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.940 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.941 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.941 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.941 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.941 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_gu46bZkwwhtar51KzoH8j_M", 2022-09-09T22:59:49.942 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.942 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.942 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.942 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.943 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.943 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.943 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.943 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.944 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763990.673725_222.1", 2022-09-09T22:59:49.944 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.944 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-36", 2022-09-09T22:59:49.944 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:10.673725202Z", 2022-09-09T22:59:49.945 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.945 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.945 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.945 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.946 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.946 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.946 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_gu46bZkwwhtar51KzoH8j_M", 2022-09-09T22:59:49.946 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.947 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.947 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.947 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.947 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.948 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.948 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.948 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.948 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763990.710819_223.1", 2022-09-09T22:59:49.949 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.949 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37", 2022-09-09T22:59:49.949 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:10.710819128Z", 2022-09-09T22:59:49.949 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.950 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.950 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.950 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.950 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.951 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.951 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_FX49dxEb4uM_d2ZifiXBHNo", 2022-09-09T22:59:49.951 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.951 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.952 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.952 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.952 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.952 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.953 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.953 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.953 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763990.715764_224.1", 2022-09-09T22:59:49.953 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.954 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37", 2022-09-09T22:59:49.954 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:10.715764014Z", 2022-09-09T22:59:49.954 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.954 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.955 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.955 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.955 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.955 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.956 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_FX49dxEb4uM_d2ZifiXBHNo", 2022-09-09T22:59:49.956 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.956 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.956 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.957 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.957 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.957 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.957 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.958 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.958 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763990.717500_225.1", 2022-09-09T22:59:49.958 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.958 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-37", 2022-09-09T22:59:49.959 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:10.717500107Z", 2022-09-09T22:59:49.959 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.959 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.959 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.960 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.960 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.960 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.960 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_hJzznYY2Hded3epYJ_abHVf", 2022-09-09T22:59:49.961 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.961 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.961 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.961 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.962 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.962 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.962 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.962 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.963 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763990.722195_226.1", 2022-09-09T22:59:49.963 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.963 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-37", 2022-09-09T22:59:49.963 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:10.722195904Z", 2022-09-09T22:59:49.964 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.964 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.964 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.965 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.965 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.965 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.965 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_hJzznYY2Hded3epYJ_abHVf", 2022-09-09T22:59:49.966 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.966 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.966 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.966 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.967 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.967 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.967 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.967 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.968 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763990.784611_227.1", 2022-09-09T22:59:49.968 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.968 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38", 2022-09-09T22:59:49.968 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:10.784611285Z", 2022-09-09T22:59:49.969 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.969 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.969 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.969 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.970 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.970 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.970 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_qDxaPPFFwg6ZX6ss6qndeGO", 2022-09-09T22:59:49.971 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.971 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.971 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.971 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.972 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.972 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.972 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.972 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.973 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763990.788678_228.1", 2022-09-09T22:59:49.973 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.973 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38", 2022-09-09T22:59:49.973 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:10.788678416Z", 2022-09-09T22:59:49.974 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.974 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.974 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.974 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.975 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.975 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.975 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_qDxaPPFFwg6ZX6ss6qndeGO", 2022-09-09T22:59:49.975 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.976 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.976 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.976 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.976 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.977 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.977 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.977 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.978 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763990.792062_229.1", 2022-09-09T22:59:49.978 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.978 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-38", 2022-09-09T22:59:49.978 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:10.792062116Z", 2022-09-09T22:59:49.979 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.979 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.979 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.979 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.980 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.980 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.980 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_gCNfZZlq_uw7Nsy58KndY3K", 2022-09-09T22:59:49.980 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.981 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.981 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.981 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.981 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.982 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.982 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.982 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.982 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662763990.796002_230.1", 2022-09-09T22:59:49.983 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.983 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-38", 2022-09-09T22:59:49.983 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:53:10.796002253Z", 2022-09-09T22:59:49.983 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.984 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.984 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.984 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.984 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.985 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.985 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_gCNfZZlq_uw7Nsy58KndY3K", 2022-09-09T22:59:49.985 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.985 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.986 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.986 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.986 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.987 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.987 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.987 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.987 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764240.029920_231.1", 2022-09-09T22:59:49.988 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.988 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-39:4b05608c-7197-4e9b-bd29-7989607fa277.4354.39", 2022-09-09T22:59:49.988 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:57:20.029920453Z", 2022-09-09T22:59:49.988 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.989 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.989 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.989 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.989 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.990 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.990 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_yHfFFXCLbn_I9-CIE0fRcSE", 2022-09-09T22:59:49.990 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.990 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.991 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.991 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:49.991 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.991 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.992 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.992 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.992 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764240.034539_232.1", 2022-09-09T22:59:49.992 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:49.993 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-39:4b05608c-7197-4e9b-bd29-7989607fa277.4354.39", 2022-09-09T22:59:49.993 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:57:20.034539448Z", 2022-09-09T22:59:49.993 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.993 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.994 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.994 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.994 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.994 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.995 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_yHfFFXCLbn_I9-CIE0fRcSE", 2022-09-09T22:59:49.995 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:49.995 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.995 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:49.996 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:49.996 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.996 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:49.996 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.997 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:49.997 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764240.036306_233.1", 2022-09-09T22:59:49.997 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:49.997 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-39", 2022-09-09T22:59:49.998 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:57:20.036306500Z", 2022-09-09T22:59:49.998 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:49.998 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:49.998 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:49.999 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:49.999 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:49.999 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:49.999 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_kuQlL_FG34kY8wCt0puQ02d", 2022-09-09T22:59:50.000 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:50.000 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:50.000 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:50.000 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:50.001 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:50.001 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:50.001 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:50.001 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:50.002 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764240.040726_234.1", 2022-09-09T22:59:50.002 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:50.002 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-39", 2022-09-09T22:59:50.002 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:57:20.040726112Z", 2022-09-09T22:59:50.003 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:50.003 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:50.003 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:50.003 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:50.004 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:50.004 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:50.004 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_kuQlL_FG34kY8wCt0puQ02d", 2022-09-09T22:59:50.004 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:50.005 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:50.005 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:50.005 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:50.006 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:50.006 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:50.006 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:50.006 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:50.007 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764240.078400_235.1", 2022-09-09T22:59:50.007 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.007 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-40:4b05608c-7197-4e9b-bd29-7989607fa277.4354.40", 2022-09-09T22:59:50.007 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:57:20.078400843Z", 2022-09-09T22:59:50.008 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:50.008 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:50.008 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:50.008 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:50.009 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:50.009 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:50.010 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_vfgdN60g8MjxPCCEgs2R_vv", 2022-09-09T22:59:50.010 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:50.011 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:50.011 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:50.012 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:50.012 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:50.013 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:50.013 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:50.014 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:50.014 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764240.084545_236.1", 2022-09-09T22:59:50.014 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.015 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-40:4b05608c-7197-4e9b-bd29-7989607fa277.4354.40", 2022-09-09T22:59:50.015 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:57:20.084545560Z", 2022-09-09T22:59:50.015 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:50.016 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:50.016 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:50.016 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:50.016 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:50.017 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:50.017 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_vfgdN60g8MjxPCCEgs2R_vv", 2022-09-09T22:59:50.017 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:50.017 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:50.018 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:50.018 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:50.018 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:50.018 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:50.019 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:50.019 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:50.019 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764240.086165_237.1", 2022-09-09T22:59:50.019 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:50.020 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-40", 2022-09-09T22:59:50.020 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:57:20.086165638Z", 2022-09-09T22:59:50.020 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:50.020 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:50.021 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:50.021 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:50.021 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:50.021 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:50.022 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_TJv9-9zJFLq6wQcqNFr3ADa", 2022-09-09T22:59:50.022 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:50.022 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:50.023 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:50.023 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:50.023 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:50.023 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:50.024 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:50.024 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:50.024 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764240.090984_238.1", 2022-09-09T22:59:50.024 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:50.025 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-40", 2022-09-09T22:59:50.025 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:57:20.090984173Z", 2022-09-09T22:59:50.025 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:50.025 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:50.026 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:50.026 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:50.026 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:50.026 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:50.027 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_TJv9-9zJFLq6wQcqNFr3ADa", 2022-09-09T22:59:50.027 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:50.027 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:50.027 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:50.028 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:50.028 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:50.028 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:50.028 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:50.029 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:50.029 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764240.151909_239.1", 2022-09-09T22:59:50.029 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.029 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-41:4b05608c-7197-4e9b-bd29-7989607fa277.4354.41", 2022-09-09T22:59:50.030 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:57:20.151909503Z", 2022-09-09T22:59:50.030 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:50.030 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:50.030 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:50.031 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:50.031 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:50.031 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:50.031 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_lh1D7Ui8P7X3ZyWeXN_Rbr6", 2022-09-09T22:59:50.032 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:50.032 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:50.032 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:50.032 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:50.033 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:50.033 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:50.033 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:50.033 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:50.034 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764240.157917_240.1", 2022-09-09T22:59:50.034 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.034 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-41:4b05608c-7197-4e9b-bd29-7989607fa277.4354.41", 2022-09-09T22:59:50.034 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:57:20.157917182Z", 2022-09-09T22:59:50.035 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:50.035 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:50.035 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:50.035 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:50.036 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:50.036 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:50.036 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_lh1D7Ui8P7X3ZyWeXN_Rbr6", 2022-09-09T22:59:50.036 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:50.037 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:50.037 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:50.037 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:50.037 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:50.038 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:50.038 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:50.038 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:50.038 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764240.159693_241.1", 2022-09-09T22:59:50.039 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:50.039 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-41", 2022-09-09T22:59:50.039 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:57:20.159693443Z", 2022-09-09T22:59:50.039 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:50.040 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:50.040 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:50.040 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:50.040 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:50.041 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:50.041 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xxvbuFhSEtP4oKwQrN2eVjC", 2022-09-09T22:59:50.041 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:50.041 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:50.042 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:50.042 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:50.042 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:50.042 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:50.043 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:50.043 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:50.043 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764240.168869_243.1", 2022-09-09T22:59:50.043 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:50.044 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-41", 2022-09-09T22:59:50.044 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:57:20.168869275Z", 2022-09-09T22:59:50.044 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:50.045 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:50.045 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:50.045 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:50.045 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:50.045 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:50.046 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_xxvbuFhSEtP4oKwQrN2eVjC", 2022-09-09T22:59:50.046 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:50.046 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:50.047 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:50.047 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:50.047 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:50.047 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:50.048 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:50.048 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T22:59:50.049 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '503b1d61-4513-48c8-a308-17b76219610c', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:50.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '503b1d61-4513-48c8-a308-17b76219610c', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:50.051 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 503b1d61-4513-48c8-a308-17b76219610c --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:59:50.093 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:50.094 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:50.150 INFO:teuthology.orchestra.run.smithi080.stdout:[ 2022-09-09T22:59:50.151 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.151 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762486.140348_44.1", 2022-09-09T22:59:50.151 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "user", 2022-09-09T22:59:50.151 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "rgw-multisite-test-user", 2022-09-09T22:59:50.152 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.140348282Z", 2022-09-09T22:59:50.152 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.152 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.152 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.153 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.153 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.153 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.154 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_hX52Pw6Vsy-toyD35b1bFsR", 2022-09-09T22:59:50.154 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.154 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.154 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.155 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.155 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.156 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.156 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.156 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.156 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762486.146147_45.1", 2022-09-09T22:59:50.157 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "user", 2022-09-09T22:59:50.157 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "rgw-multisite-test-user", 2022-09-09T22:59:50.157 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147984Z", 2022-09-09T22:59:50.157 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.158 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.158 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.158 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.159 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.159 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.160 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_hX52Pw6Vsy-toyD35b1bFsR", 2022-09-09T22:59:50.160 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.160 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.161 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.161 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.162 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.162 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.163 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.163 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.164 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762609.632258_38.1", 2022-09-09T22:59:50.164 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.164 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-1:4b05608c-7197-4e9b-bd29-7989607fa277.4354.1", 2022-09-09T22:59:50.165 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:09.632258573Z", 2022-09-09T22:59:50.165 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.166 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.166 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.166 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.167 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.167 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.168 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_S0f1bdC1tDuhcRUnkrIExa6", 2022-09-09T22:59:50.168 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.168 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.168 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.169 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.169 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.169 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.169 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.170 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.170 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762609.636081_39.1", 2022-09-09T22:59:50.170 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.171 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-1:4b05608c-7197-4e9b-bd29-7989607fa277.4354.1", 2022-09-09T22:59:50.171 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:09.636081844Z", 2022-09-09T22:59:50.171 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.171 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.172 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.172 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.172 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.172 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.173 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_S0f1bdC1tDuhcRUnkrIExa6", 2022-09-09T22:59:50.173 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.173 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.173 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.174 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.174 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.174 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.175 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.175 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.175 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762609.637417_40.1", 2022-09-09T22:59:50.175 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.176 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-1", 2022-09-09T22:59:50.176 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:09.637417779Z", 2022-09-09T22:59:50.176 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.176 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.177 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.177 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.177 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.177 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.178 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_ykXwHWWOzdeAKyn1PiulIFb", 2022-09-09T22:59:50.178 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.178 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.179 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.179 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.179 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.179 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.180 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.180 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.180 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762609.640815_41.1", 2022-09-09T22:59:50.180 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.181 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-1", 2022-09-09T22:59:50.181 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:09.640815201Z", 2022-09-09T22:59:50.181 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.181 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.182 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.182 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.182 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.182 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.183 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_ykXwHWWOzdeAKyn1PiulIFb", 2022-09-09T22:59:50.183 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.183 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.183 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.184 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.184 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.184 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.184 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.185 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.185 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762609.685710_42.1", 2022-09-09T22:59:50.185 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.185 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-2:4b05608c-7197-4e9b-bd29-7989607fa277.4354.2", 2022-09-09T22:59:50.186 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:09.685710351Z", 2022-09-09T22:59:50.186 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.186 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.187 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.187 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.187 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.187 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.188 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_p7pA6aFQb4GfhoLrAPoFo3L", 2022-09-09T22:59:50.188 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.188 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.188 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.189 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.189 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.189 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.190 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.190 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.190 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762609.691000_43.1", 2022-09-09T22:59:50.190 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.191 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-2:4b05608c-7197-4e9b-bd29-7989607fa277.4354.2", 2022-09-09T22:59:50.191 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:09.691000087Z", 2022-09-09T22:59:50.191 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.191 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.192 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.192 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.192 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.192 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.193 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_p7pA6aFQb4GfhoLrAPoFo3L", 2022-09-09T22:59:50.193 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.193 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.193 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.196 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.197 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.197 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.197 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.198 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.198 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762609.692503_44.1", 2022-09-09T22:59:50.198 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.198 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-2", 2022-09-09T22:59:50.199 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:09.692503564Z", 2022-09-09T22:59:50.199 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.199 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.199 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.200 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.200 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.200 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.200 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_ut5cP71qSrGfyAy9sYnUiGl", 2022-09-09T22:59:50.201 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.201 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.201 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.201 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.202 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.202 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.202 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.203 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.203 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762609.696482_45.1", 2022-09-09T22:59:50.203 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.203 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-2", 2022-09-09T22:59:50.204 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:09.696482596Z", 2022-09-09T22:59:50.204 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.204 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.204 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.205 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.205 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.205 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.205 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_ut5cP71qSrGfyAy9sYnUiGl", 2022-09-09T22:59:50.206 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.206 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.206 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.207 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.207 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.207 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.207 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.208 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.208 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762612.758261_46.1", 2022-09-09T22:59:50.208 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.208 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-3:4b05608c-7197-4e9b-bd29-7989607fa277.4354.3", 2022-09-09T22:59:50.209 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:12.758261347Z", 2022-09-09T22:59:50.209 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.209 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.209 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.210 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.210 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.210 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.211 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_DTaKu07VPRWGanYia-lsgfe", 2022-09-09T22:59:50.211 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.211 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.211 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.212 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.212 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.212 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.212 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.213 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.213 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762612.764541_47.1", 2022-09-09T22:59:50.213 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.213 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-3:4b05608c-7197-4e9b-bd29-7989607fa277.4354.3", 2022-09-09T22:59:50.214 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:12.764541107Z", 2022-09-09T22:59:50.214 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.214 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.215 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.215 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.215 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.215 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.216 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_DTaKu07VPRWGanYia-lsgfe", 2022-09-09T22:59:50.216 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.217 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.217 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.218 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.218 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.219 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.219 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.220 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.220 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762612.766069_48.1", 2022-09-09T22:59:50.221 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.221 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-3", 2022-09-09T22:59:50.222 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:12.766069866Z", 2022-09-09T22:59:50.222 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.223 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.223 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.224 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.224 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.225 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.225 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_LVgUDeF4cxLIns2No37bLQx", 2022-09-09T22:59:50.226 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.226 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.227 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.227 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.228 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.228 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.229 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.229 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.230 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762612.770414_49.1", 2022-09-09T22:59:50.230 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.231 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-3", 2022-09-09T22:59:50.231 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:12.770414961Z", 2022-09-09T22:59:50.232 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.232 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.233 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.234 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.234 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.235 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.235 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_LVgUDeF4cxLIns2No37bLQx", 2022-09-09T22:59:50.236 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.236 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.237 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.237 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.238 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.238 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.239 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.239 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.240 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762618.742117_50.1", 2022-09-09T22:59:50.240 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.241 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-4:4b05608c-7197-4e9b-bd29-7989607fa277.4354.4", 2022-09-09T22:59:50.241 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:18.742117846Z", 2022-09-09T22:59:50.242 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.242 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.243 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.243 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.244 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.244 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.245 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_QrfGp4QHJmNMDcTAMoEbJrk", 2022-09-09T22:59:50.245 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.246 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.246 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.247 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.247 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.248 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.248 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.249 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.249 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762618.747553_51.1", 2022-09-09T22:59:50.250 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.250 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-4:4b05608c-7197-4e9b-bd29-7989607fa277.4354.4", 2022-09-09T22:59:50.251 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:18.747553163Z", 2022-09-09T22:59:50.251 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.252 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.252 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.253 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.253 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.254 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.254 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_QrfGp4QHJmNMDcTAMoEbJrk", 2022-09-09T22:59:50.255 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.255 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.256 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.257 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.258 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.258 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.259 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.259 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.260 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762618.749036_52.1", 2022-09-09T22:59:50.260 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.261 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-4", 2022-09-09T22:59:50.261 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:18.749036760Z", 2022-09-09T22:59:50.262 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.262 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.263 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.263 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.264 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.264 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.265 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_pNYiU6yoVaLSavU2aw7eQra", 2022-09-09T22:59:50.265 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.266 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.266 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.266 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.267 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.267 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.268 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.268 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.268 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762618.752816_53.1", 2022-09-09T22:59:50.269 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.269 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-4", 2022-09-09T22:59:50.270 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:18.752816607Z", 2022-09-09T22:59:50.270 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.270 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.271 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.271 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.271 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.272 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.272 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_pNYiU6yoVaLSavU2aw7eQra", 2022-09-09T22:59:50.272 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.272 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.273 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.273 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.273 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.273 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.274 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.274 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.274 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762618.783525_54.1", 2022-09-09T22:59:50.274 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.275 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-5:4b05608c-7197-4e9b-bd29-7989607fa277.4354.5", 2022-09-09T22:59:50.275 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:18.783525474Z", 2022-09-09T22:59:50.275 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.275 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.276 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.276 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.276 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.276 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.277 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_PPSGwb27eWuU4vjS5103sBF", 2022-09-09T22:59:50.277 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.277 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.277 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.278 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.278 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.278 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.279 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.279 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.279 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762618.789318_55.1", 2022-09-09T22:59:50.279 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.280 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-5:4b05608c-7197-4e9b-bd29-7989607fa277.4354.5", 2022-09-09T22:59:50.280 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:18.789318940Z", 2022-09-09T22:59:50.280 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.280 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.281 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.281 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.281 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.281 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.282 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_PPSGwb27eWuU4vjS5103sBF", 2022-09-09T22:59:50.282 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.282 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.282 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.283 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.283 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.283 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.283 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.284 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.284 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762618.791554_56.1", 2022-09-09T22:59:50.284 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.284 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-5", 2022-09-09T22:59:50.285 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:18.791554821Z", 2022-09-09T22:59:50.285 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.285 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.285 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.286 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.286 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.286 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.287 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_rrNh6U2_REBalApJMN0ghUy", 2022-09-09T22:59:50.287 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.287 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.287 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.288 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.288 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.288 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.289 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.289 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.289 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762618.797453_57.1", 2022-09-09T22:59:50.290 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.290 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-5", 2022-09-09T22:59:50.290 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:18.797453446Z", 2022-09-09T22:59:50.290 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.291 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.291 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.291 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.292 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.292 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.292 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_rrNh6U2_REBalApJMN0ghUy", 2022-09-09T22:59:50.292 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.293 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.293 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.293 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.293 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.293 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.294 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.294 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.294 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762618.850246_58.1", 2022-09-09T22:59:50.295 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.295 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-6:4b05608c-7197-4e9b-bd29-7989607fa277.4354.6", 2022-09-09T22:59:50.295 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:18.850246676Z", 2022-09-09T22:59:50.295 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.296 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.296 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.296 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.296 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.297 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.297 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_cRk-wHT_rpr3K1T0yRJVf7Q", 2022-09-09T22:59:50.297 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.297 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.298 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.298 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.298 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.298 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.299 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.299 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.299 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762618.855531_59.1", 2022-09-09T22:59:50.299 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.300 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-6:4b05608c-7197-4e9b-bd29-7989607fa277.4354.6", 2022-09-09T22:59:50.300 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:18.855531993Z", 2022-09-09T22:59:50.300 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.300 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.301 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.301 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.301 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.301 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.302 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_cRk-wHT_rpr3K1T0yRJVf7Q", 2022-09-09T22:59:50.302 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.302 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.303 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.303 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.303 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.303 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.304 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.304 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.304 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762618.857124_60.1", 2022-09-09T22:59:50.304 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.305 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-6", 2022-09-09T22:59:50.305 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:18.857124257Z", 2022-09-09T22:59:50.305 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.305 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.306 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.306 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.306 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.306 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.307 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_uJKaJMt6G8HexUvWErzrMXK", 2022-09-09T22:59:50.307 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.307 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.307 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.308 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.308 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.308 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.308 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.309 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.309 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762618.860801_61.1", 2022-09-09T22:59:50.309 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.309 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-6", 2022-09-09T22:59:50.310 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:18.860801405Z", 2022-09-09T22:59:50.310 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.310 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.310 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.311 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.311 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.311 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.311 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_uJKaJMt6G8HexUvWErzrMXK", 2022-09-09T22:59:50.312 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.312 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.312 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.312 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.313 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.313 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.313 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.313 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.314 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762621.000610_62.1", 2022-09-09T22:59:50.314 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.314 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-7:4b05608c-7197-4e9b-bd29-7989607fa277.4354.7", 2022-09-09T22:59:50.314 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:21.000610746Z", 2022-09-09T22:59:50.315 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.315 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.315 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.315 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.316 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.316 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.316 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_RZXAZlTqoK-kSBG7i9Qb1iT", 2022-09-09T22:59:50.316 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.317 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.317 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.317 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.317 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.318 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.318 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.318 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.319 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762621.004495_63.1", 2022-09-09T22:59:50.319 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.319 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-7:4b05608c-7197-4e9b-bd29-7989607fa277.4354.7", 2022-09-09T22:59:50.319 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:21.004495603Z", 2022-09-09T22:59:50.320 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.320 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.320 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.320 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.320 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.321 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.321 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_RZXAZlTqoK-kSBG7i9Qb1iT", 2022-09-09T22:59:50.321 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.322 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.322 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.322 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.322 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.323 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.323 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.323 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.323 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762621.005900_64.1", 2022-09-09T22:59:50.324 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.324 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-7", 2022-09-09T22:59:50.324 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:21.005900841Z", 2022-09-09T22:59:50.324 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.325 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.325 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.325 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.325 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.326 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.326 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_kxHFo1nRBQfhvLRoWDL9llN", 2022-09-09T22:59:50.326 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.326 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.327 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.327 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.327 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.327 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.328 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.328 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.328 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762621.009874_65.1", 2022-09-09T22:59:50.328 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.328 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-7", 2022-09-09T22:59:50.329 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:21.009874532Z", 2022-09-09T22:59:50.329 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.329 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.330 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.330 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.330 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.330 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.331 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_kxHFo1nRBQfhvLRoWDL9llN", 2022-09-09T22:59:50.331 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.331 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.331 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.332 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.332 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.332 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.332 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.333 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.333 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762621.036747_66.1", 2022-09-09T22:59:50.333 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.333 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-8:4b05608c-7197-4e9b-bd29-7989607fa277.4354.8", 2022-09-09T22:59:50.334 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:21.036747351Z", 2022-09-09T22:59:50.334 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.334 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.334 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.335 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.335 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.335 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.335 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_CE4D4FZvn3xK_Q51293KOFn", 2022-09-09T22:59:50.336 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.336 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.336 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.336 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.337 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.337 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.337 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.337 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.338 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762621.042100_67.1", 2022-09-09T22:59:50.338 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.338 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-8:4b05608c-7197-4e9b-bd29-7989607fa277.4354.8", 2022-09-09T22:59:50.338 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:21.042100215Z", 2022-09-09T22:59:50.339 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.339 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.339 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.339 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.340 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.340 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.340 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_CE4D4FZvn3xK_Q51293KOFn", 2022-09-09T22:59:50.341 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.341 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.341 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.341 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.342 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.342 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.342 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.342 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.343 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762621.043792_68.1", 2022-09-09T22:59:50.343 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.343 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-8", 2022-09-09T22:59:50.343 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:21.043792412Z", 2022-09-09T22:59:50.344 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.344 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.344 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.344 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.345 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.345 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.345 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_LemC9Nq4ijGLYcUn_qbYGY1", 2022-09-09T22:59:50.345 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.346 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.346 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.346 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.347 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.347 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.347 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.347 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.347 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762621.047923_69.1", 2022-09-09T22:59:50.348 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.348 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-8", 2022-09-09T22:59:50.348 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:21.047923611Z", 2022-09-09T22:59:50.348 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.349 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.349 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.349 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.349 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.350 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.350 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_LemC9Nq4ijGLYcUn_qbYGY1", 2022-09-09T22:59:50.350 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.350 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.351 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.351 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.351 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.351 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.352 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.352 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.352 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762621.100882_70.1", 2022-09-09T22:59:50.352 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.353 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-9:4b05608c-7197-4e9b-bd29-7989607fa277.4354.9", 2022-09-09T22:59:50.353 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:21.100882035Z", 2022-09-09T22:59:50.353 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.353 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.354 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.354 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.354 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.354 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.355 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_mXmeXhs1mUho7dSTp_n1nkI", 2022-09-09T22:59:50.355 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.355 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.355 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.356 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.356 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.356 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.356 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.357 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.357 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762621.105043_71.1", 2022-09-09T22:59:50.357 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.357 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-9:4b05608c-7197-4e9b-bd29-7989607fa277.4354.9", 2022-09-09T22:59:50.358 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:21.105043152Z", 2022-09-09T22:59:50.358 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.358 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.358 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.359 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.359 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.359 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.359 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_mXmeXhs1mUho7dSTp_n1nkI", 2022-09-09T22:59:50.360 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.360 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.360 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.360 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.361 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.361 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.361 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.362 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.362 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762621.106530_72.1", 2022-09-09T22:59:50.362 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.362 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-9", 2022-09-09T22:59:50.363 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:21.106530704Z", 2022-09-09T22:59:50.363 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.363 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.363 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.364 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.364 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.364 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.364 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_4XQiLk5HQiFc0Zuqos3dBaU", 2022-09-09T22:59:50.365 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.365 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.365 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.365 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.366 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.366 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.366 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.366 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.367 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762621.110791_73.1", 2022-09-09T22:59:50.367 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.367 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-9", 2022-09-09T22:59:50.368 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:21.110791110Z", 2022-09-09T22:59:50.368 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.368 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.368 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.369 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.369 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.369 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.369 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_4XQiLk5HQiFc0Zuqos3dBaU", 2022-09-09T22:59:50.370 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.370 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.370 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.370 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.371 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.371 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.371 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.372 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.372 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762637.476803_74.1", 2022-09-09T22:59:50.372 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.372 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-7:4b05608c-7197-4e9b-bd29-7989607fa277.4354.7", 2022-09-09T22:59:50.373 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.476803832Z", 2022-09-09T22:59:50.373 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.373 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.373 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_RZXAZlTqoK-kSBG7i9Qb1iT", 2022-09-09T22:59:50.374 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.374 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.374 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.374 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_RZXAZlTqoK-kSBG7i9Qb1iT", 2022-09-09T22:59:50.375 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.375 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.375 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.375 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.376 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.376 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.376 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.376 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.377 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762637.483128_75.1", 2022-09-09T22:59:50.377 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.377 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-7:4b05608c-7197-4e9b-bd29-7989607fa277.4354.7", 2022-09-09T22:59:50.377 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.483128307Z", 2022-09-09T22:59:50.378 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.378 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.378 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_RZXAZlTqoK-kSBG7i9Qb1iT", 2022-09-09T22:59:50.378 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.379 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.379 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.379 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_RZXAZlTqoK-kSBG7i9Qb1iT", 2022-09-09T22:59:50.379 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.380 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.380 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.380 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.381 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.381 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.381 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.381 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.382 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762637.506467_76.1", 2022-09-09T22:59:50.382 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.382 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-7", 2022-09-09T22:59:50.382 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.506467744Z", 2022-09-09T22:59:50.383 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.383 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.383 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_kxHFo1nRBQfhvLRoWDL9llN", 2022-09-09T22:59:50.383 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.384 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.384 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.384 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_kxHFo1nRBQfhvLRoWDL9llN", 2022-09-09T22:59:50.385 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.385 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.385 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.385 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "remove" 2022-09-09T22:59:50.386 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.386 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.386 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.386 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.387 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762637.511752_77.1", 2022-09-09T22:59:50.387 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.387 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-7", 2022-09-09T22:59:50.387 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.511752143Z", 2022-09-09T22:59:50.388 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.388 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.388 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_kxHFo1nRBQfhvLRoWDL9llN", 2022-09-09T22:59:50.388 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.388 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.389 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.389 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_kxHFo1nRBQfhvLRoWDL9llN", 2022-09-09T22:59:50.389 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.389 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.390 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.390 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.390 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.391 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.391 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.391 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.391 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762637.579226_78.1", 2022-09-09T22:59:50.392 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.392 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-8:4b05608c-7197-4e9b-bd29-7989607fa277.4354.8", 2022-09-09T22:59:50.392 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.579226395Z", 2022-09-09T22:59:50.392 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.393 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.393 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_CE4D4FZvn3xK_Q51293KOFn", 2022-09-09T22:59:50.393 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.393 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.394 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.394 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_CE4D4FZvn3xK_Q51293KOFn", 2022-09-09T22:59:50.394 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.394 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.395 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.395 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.395 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.395 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.396 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.396 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.396 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762637.584225_79.1", 2022-09-09T22:59:50.396 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.397 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-8:4b05608c-7197-4e9b-bd29-7989607fa277.4354.8", 2022-09-09T22:59:50.397 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.584225809Z", 2022-09-09T22:59:50.397 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.397 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.398 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_CE4D4FZvn3xK_Q51293KOFn", 2022-09-09T22:59:50.398 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.398 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.399 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.399 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_CE4D4FZvn3xK_Q51293KOFn", 2022-09-09T22:59:50.399 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.399 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.400 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.400 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.400 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.400 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.401 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.401 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.401 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762637.601579_80.1", 2022-09-09T22:59:50.401 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.401 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-8", 2022-09-09T22:59:50.402 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.601579852Z", 2022-09-09T22:59:50.402 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.402 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.403 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_LemC9Nq4ijGLYcUn_qbYGY1", 2022-09-09T22:59:50.403 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.403 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.403 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.404 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_LemC9Nq4ijGLYcUn_qbYGY1", 2022-09-09T22:59:50.404 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.404 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.404 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.405 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "remove" 2022-09-09T22:59:50.405 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.405 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.405 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.406 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.406 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762637.608474_81.1", 2022-09-09T22:59:50.406 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.407 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-8", 2022-09-09T22:59:50.407 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.608474828Z", 2022-09-09T22:59:50.407 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.407 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.408 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_LemC9Nq4ijGLYcUn_qbYGY1", 2022-09-09T22:59:50.408 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.408 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.408 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.409 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_LemC9Nq4ijGLYcUn_qbYGY1", 2022-09-09T22:59:50.409 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.409 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.410 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.410 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.410 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.411 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.411 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.411 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.411 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762637.726627_82.1", 2022-09-09T22:59:50.412 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.412 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-9:4b05608c-7197-4e9b-bd29-7989607fa277.4354.9", 2022-09-09T22:59:50.412 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.726627171Z", 2022-09-09T22:59:50.413 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.413 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.413 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_mXmeXhs1mUho7dSTp_n1nkI", 2022-09-09T22:59:50.413 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.414 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.414 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.414 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_mXmeXhs1mUho7dSTp_n1nkI", 2022-09-09T22:59:50.415 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.415 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.415 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.415 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.416 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.416 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.416 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.417 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.417 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762637.731530_83.1", 2022-09-09T22:59:50.417 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.417 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-9:4b05608c-7197-4e9b-bd29-7989607fa277.4354.9", 2022-09-09T22:59:50.418 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.731530094Z", 2022-09-09T22:59:50.418 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.418 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.419 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_mXmeXhs1mUho7dSTp_n1nkI", 2022-09-09T22:59:50.419 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.419 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.419 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.420 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_mXmeXhs1mUho7dSTp_n1nkI", 2022-09-09T22:59:50.420 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.420 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.420 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.421 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.421 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.421 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.421 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.422 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.422 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762637.751637_84.1", 2022-09-09T22:59:50.422 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.422 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-9", 2022-09-09T22:59:50.423 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.751637260Z", 2022-09-09T22:59:50.423 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.423 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.423 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_4XQiLk5HQiFc0Zuqos3dBaU", 2022-09-09T22:59:50.424 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.424 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.424 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.424 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_4XQiLk5HQiFc0Zuqos3dBaU", 2022-09-09T22:59:50.425 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.425 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.425 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.425 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "remove" 2022-09-09T22:59:50.426 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.426 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.426 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.426 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.427 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762637.757014_85.1", 2022-09-09T22:59:50.427 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.427 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-9", 2022-09-09T22:59:50.427 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014147Z", 2022-09-09T22:59:50.428 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.428 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.428 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_4XQiLk5HQiFc0Zuqos3dBaU", 2022-09-09T22:59:50.428 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.429 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.429 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.429 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_4XQiLk5HQiFc0Zuqos3dBaU", 2022-09-09T22:59:50.429 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.430 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.430 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.430 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.431 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.431 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.431 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.431 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.432 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762643.827940_36.1", 2022-09-09T22:59:50.432 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.432 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10", 2022-09-09T22:59:50.432 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:43.827940737Z", 2022-09-09T22:59:50.433 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.433 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.433 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.433 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.434 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.434 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.434 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_H19u2ytwno45CS85TVUwNCX", 2022-09-09T22:59:50.434 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.435 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.435 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.435 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.435 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.436 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.436 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.436 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.436 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762643.831705_37.1", 2022-09-09T22:59:50.437 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.437 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10", 2022-09-09T22:59:50.437 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:43.831705348Z", 2022-09-09T22:59:50.437 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.438 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.438 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.438 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.438 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.439 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.439 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_H19u2ytwno45CS85TVUwNCX", 2022-09-09T22:59:50.439 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.439 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.440 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.440 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.440 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.440 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.441 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.441 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.441 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762643.833115_38.1", 2022-09-09T22:59:50.441 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.442 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-10", 2022-09-09T22:59:50.442 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:43.833115806Z", 2022-09-09T22:59:50.442 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.442 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.443 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.443 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.443 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.444 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.444 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_fQCrFkZsSXo9ojIGdsOR1AR", 2022-09-09T22:59:50.444 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.444 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.445 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.445 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.445 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.446 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.446 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.446 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.447 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762643.836655_39.1", 2022-09-09T22:59:50.447 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.447 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-10", 2022-09-09T22:59:50.447 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:43.836655955Z", 2022-09-09T22:59:50.448 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.448 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.448 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.448 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.449 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.449 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.449 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_fQCrFkZsSXo9ojIGdsOR1AR", 2022-09-09T22:59:50.449 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.449 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.450 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.450 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.450 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.451 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.451 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.451 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.452 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762643.862368_40.1", 2022-09-09T22:59:50.452 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.452 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11", 2022-09-09T22:59:50.452 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:43.862368801Z", 2022-09-09T22:59:50.453 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.453 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.453 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.453 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.454 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.454 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.454 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_zzZ2TMBNyJ82w6D_gHE19mR", 2022-09-09T22:59:50.455 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.455 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.455 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.455 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.456 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.456 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.456 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.457 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.457 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762643.867631_41.1", 2022-09-09T22:59:50.457 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.457 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11", 2022-09-09T22:59:50.458 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:43.867631329Z", 2022-09-09T22:59:50.458 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.458 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.459 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.459 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.459 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.459 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.460 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_zzZ2TMBNyJ82w6D_gHE19mR", 2022-09-09T22:59:50.460 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.460 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.461 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.461 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.461 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.461 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.462 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.462 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.462 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762643.869504_42.1", 2022-09-09T22:59:50.462 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.463 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-11", 2022-09-09T22:59:50.463 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:43.869504153Z", 2022-09-09T22:59:50.463 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.464 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.464 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.464 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.464 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.465 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.465 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_TbMj1OrULohgBriaPgsn-L0", 2022-09-09T22:59:50.465 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.466 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.466 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.466 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.467 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.467 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.467 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.467 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.468 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762643.874819_43.1", 2022-09-09T22:59:50.468 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.468 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-11", 2022-09-09T22:59:50.469 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:43.874819824Z", 2022-09-09T22:59:50.469 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.469 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.469 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.470 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.470 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.470 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.471 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_TbMj1OrULohgBriaPgsn-L0", 2022-09-09T22:59:50.471 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.471 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.471 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.472 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.472 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.472 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.472 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.473 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.473 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762643.923552_44.1", 2022-09-09T22:59:50.473 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.473 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12", 2022-09-09T22:59:50.474 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:43.923552476Z", 2022-09-09T22:59:50.474 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.474 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.474 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.475 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.475 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.475 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.475 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_BNcMrJxZygGN9lZLTtBZnDT", 2022-09-09T22:59:50.476 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.476 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.476 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.476 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.477 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.477 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.477 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.477 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.478 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762643.928237_45.1", 2022-09-09T22:59:50.478 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.478 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12", 2022-09-09T22:59:50.478 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:43.928237283Z", 2022-09-09T22:59:50.479 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.479 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.479 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.479 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.480 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.480 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.480 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_BNcMrJxZygGN9lZLTtBZnDT", 2022-09-09T22:59:50.480 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.481 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.481 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.481 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.481 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.482 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.482 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.482 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.482 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762643.929723_46.1", 2022-09-09T22:59:50.483 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.483 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-12", 2022-09-09T22:59:50.483 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:43.929723240Z", 2022-09-09T22:59:50.483 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.484 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.484 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.484 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.484 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.484 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.485 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_yd7O3Le1YA9DE_Q8UDH4F_4", 2022-09-09T22:59:50.485 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.485 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.485 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.486 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.486 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.486 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.486 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.487 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.487 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762643.933454_47.1", 2022-09-09T22:59:50.487 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.487 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-12", 2022-09-09T22:59:50.488 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:43.933454382Z", 2022-09-09T22:59:50.488 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.488 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.488 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.489 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.489 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.489 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.489 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_yd7O3Le1YA9DE_Q8UDH4F_4", 2022-09-09T22:59:50.490 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.490 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.490 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.490 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.491 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.491 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.491 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.491 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.492 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762786.612586_48.1", 2022-09-09T22:59:50.492 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.492 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13", 2022-09-09T22:59:50.492 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:33:06.612586069Z", 2022-09-09T22:59:50.493 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.493 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.493 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.493 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.494 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.494 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.494 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_dqgbU-URapyJOJCglaP6dNs", 2022-09-09T22:59:50.494 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.495 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.495 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.495 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.495 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.496 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.496 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.496 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.496 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762786.616793_49.1", 2022-09-09T22:59:50.497 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.497 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13", 2022-09-09T22:59:50.497 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:33:06.616793174Z", 2022-09-09T22:59:50.498 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.498 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.498 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.498 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.499 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.499 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.499 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_dqgbU-URapyJOJCglaP6dNs", 2022-09-09T22:59:50.499 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.500 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.500 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.500 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.500 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.501 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.501 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.501 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.501 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762786.618187_50.1", 2022-09-09T22:59:50.502 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.502 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-13", 2022-09-09T22:59:50.502 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:33:06.618187872Z", 2022-09-09T22:59:50.502 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.503 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.503 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.503 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.503 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.504 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.504 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_JNgA-Mcbf5d_uTKjSiKsu05", 2022-09-09T22:59:50.504 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.504 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.505 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.505 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.505 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.505 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.505 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.506 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.506 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762786.621862_51.1", 2022-09-09T22:59:50.506 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.507 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-13", 2022-09-09T22:59:50.507 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:33:06.621862110Z", 2022-09-09T22:59:50.507 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.507 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.508 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.508 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.508 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.508 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.509 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_JNgA-Mcbf5d_uTKjSiKsu05", 2022-09-09T22:59:50.509 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.509 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.509 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.510 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.510 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.510 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.510 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.511 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.511 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762786.651673_52.1", 2022-09-09T22:59:50.511 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.511 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14", 2022-09-09T22:59:50.512 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:33:06.651673719Z", 2022-09-09T22:59:50.512 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.512 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.512 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.513 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.513 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.513 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.513 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_f0wDKO7d5qM7TS5fzfH6prz", 2022-09-09T22:59:50.514 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.514 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.514 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.514 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.515 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.515 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.515 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.515 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.516 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762786.656784_53.1", 2022-09-09T22:59:50.516 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.516 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14", 2022-09-09T22:59:50.516 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:33:06.656784091Z", 2022-09-09T22:59:50.517 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.517 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.517 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.517 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.518 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.518 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.518 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_f0wDKO7d5qM7TS5fzfH6prz", 2022-09-09T22:59:50.518 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.519 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.519 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.519 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.519 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.520 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.520 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.520 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.520 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762786.658400_54.1", 2022-09-09T22:59:50.521 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.521 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-14", 2022-09-09T22:59:50.521 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:33:06.658400465Z", 2022-09-09T22:59:50.521 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.521 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.522 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.522 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.522 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.523 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.523 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_tDp27FdXJgMOU8rSYSjQb_Y", 2022-09-09T22:59:50.523 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.523 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.523 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.524 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.524 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.524 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.524 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.525 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.525 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762786.662396_55.1", 2022-09-09T22:59:50.525 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.525 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-14", 2022-09-09T22:59:50.526 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:33:06.662396356Z", 2022-09-09T22:59:50.526 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.526 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.526 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.527 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.527 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.527 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.527 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_tDp27FdXJgMOU8rSYSjQb_Y", 2022-09-09T22:59:50.528 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.528 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.528 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.528 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.529 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.529 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.529 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.529 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.530 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762786.710171_56.1", 2022-09-09T22:59:50.530 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.530 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15", 2022-09-09T22:59:50.530 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:33:06.710171968Z", 2022-09-09T22:59:50.531 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.531 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.531 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.531 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.532 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.532 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.532 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_ogbkKIIiKYU8oWXJ_RMpS5O", 2022-09-09T22:59:50.532 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.533 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.533 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.533 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.533 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.534 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.534 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.534 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.534 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762786.713926_57.1", 2022-09-09T22:59:50.535 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.535 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15", 2022-09-09T22:59:50.535 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:33:06.713926458Z", 2022-09-09T22:59:50.535 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.536 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.536 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.536 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.536 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.537 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.537 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_ogbkKIIiKYU8oWXJ_RMpS5O", 2022-09-09T22:59:50.537 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.537 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.538 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.538 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.538 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.538 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.539 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.539 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.539 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762786.715462_58.1", 2022-09-09T22:59:50.539 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.540 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-15", 2022-09-09T22:59:50.540 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:33:06.715462548Z", 2022-09-09T22:59:50.540 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.540 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.541 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.541 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.541 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.541 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.542 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_Hk_J6cauBfq0403R0YNkLtB", 2022-09-09T22:59:50.542 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.542 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.542 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.543 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.543 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.543 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.543 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.544 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.544 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762786.718887_59.1", 2022-09-09T22:59:50.544 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.544 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-15", 2022-09-09T22:59:50.545 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:33:06.718887015Z", 2022-09-09T22:59:50.545 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.545 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.545 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.546 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.546 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.546 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.546 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_Hk_J6cauBfq0403R0YNkLtB", 2022-09-09T22:59:50.547 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.547 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.547 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.548 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.548 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.548 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.548 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.549 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.549 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.055821_61.1", 2022-09-09T22:59:50.549 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.549 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16", 2022-09-09T22:59:50.550 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.055821864Z", 2022-09-09T22:59:50.550 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.550 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.550 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.551 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.551 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.551 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.552 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_jgmlG80S-C76e7qmPCPXWwY", 2022-09-09T22:59:50.552 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.552 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.552 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.553 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.553 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.553 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.553 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.554 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.554 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.060244_62.1", 2022-09-09T22:59:50.554 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.554 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16", 2022-09-09T22:59:50.555 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.060244432Z", 2022-09-09T22:59:50.555 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.555 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.555 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.556 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.556 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.556 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.556 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_jgmlG80S-C76e7qmPCPXWwY", 2022-09-09T22:59:50.557 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.557 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.557 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.557 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.558 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.558 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.558 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.558 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.559 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.061724_63.1", 2022-09-09T22:59:50.559 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.559 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-16", 2022-09-09T22:59:50.559 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.061724454Z", 2022-09-09T22:59:50.560 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.560 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.560 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.560 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.560 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.561 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.561 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_QzVpUqaeVNRQiAA9WAn6beJ", 2022-09-09T22:59:50.561 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.561 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.562 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.562 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.562 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.562 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.563 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.563 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.563 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.065560_64.1", 2022-09-09T22:59:50.563 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.564 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-16", 2022-09-09T22:59:50.564 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.065560207Z", 2022-09-09T22:59:50.564 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.564 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.565 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.565 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.565 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.565 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.566 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_QzVpUqaeVNRQiAA9WAn6beJ", 2022-09-09T22:59:50.566 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.566 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.566 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.567 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.567 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.567 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.567 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.568 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.568 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.094558_65.1", 2022-09-09T22:59:50.568 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.568 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17", 2022-09-09T22:59:50.569 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.094558838Z", 2022-09-09T22:59:50.569 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.569 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.569 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.570 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.570 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.570 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.570 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_MtuyPaZmCzfHOcgXjri3F-V", 2022-09-09T22:59:50.571 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.571 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.571 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.571 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.572 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.572 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.572 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.572 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.573 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.099477_66.1", 2022-09-09T22:59:50.573 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.573 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17", 2022-09-09T22:59:50.573 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.099477016Z", 2022-09-09T22:59:50.573 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.574 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.574 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.574 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.574 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.575 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.575 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_MtuyPaZmCzfHOcgXjri3F-V", 2022-09-09T22:59:50.575 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.575 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.576 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.576 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.576 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.576 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.577 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.577 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.577 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.101086_67.1", 2022-09-09T22:59:50.577 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.578 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-17", 2022-09-09T22:59:50.578 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.101086117Z", 2022-09-09T22:59:50.578 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.578 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.579 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.579 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.579 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.579 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.580 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_eneLpQRdXpsBeKeIZdm9Fz7", 2022-09-09T22:59:50.580 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.580 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.580 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.581 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.581 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.581 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.581 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.582 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.582 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.105046_68.1", 2022-09-09T22:59:50.582 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.582 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-17", 2022-09-09T22:59:50.583 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.105046294Z", 2022-09-09T22:59:50.583 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.583 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.583 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.584 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.584 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.584 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.584 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_eneLpQRdXpsBeKeIZdm9Fz7", 2022-09-09T22:59:50.585 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.585 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.585 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.585 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.586 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.586 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.586 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.586 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.587 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.149665_69.1", 2022-09-09T22:59:50.587 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.587 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18", 2022-09-09T22:59:50.587 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.149665741Z", 2022-09-09T22:59:50.588 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.588 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.588 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.588 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.589 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.589 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.589 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_bCS5W1GGiGEz-gWOOFrVrp4", 2022-09-09T22:59:50.589 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.590 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.590 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.590 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.590 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.591 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.591 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.591 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.591 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.156765_70.1", 2022-09-09T22:59:50.592 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.592 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18", 2022-09-09T22:59:50.592 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.156765172Z", 2022-09-09T22:59:50.592 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.593 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.593 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.593 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.593 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.594 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.594 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_bCS5W1GGiGEz-gWOOFrVrp4", 2022-09-09T22:59:50.594 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.594 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.595 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.595 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.595 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.595 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.596 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.596 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.596 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.161880_71.1", 2022-09-09T22:59:50.596 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.597 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-18", 2022-09-09T22:59:50.597 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.161880990Z", 2022-09-09T22:59:50.597 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.597 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.598 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.598 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.598 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.598 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.599 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_vVrGNRKUHx_Nc7AgE45d0q7", 2022-09-09T22:59:50.599 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.599 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.599 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.600 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.600 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.600 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.600 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.600 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.601 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.166872_72.1", 2022-09-09T22:59:50.601 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.601 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-18", 2022-09-09T22:59:50.601 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.166872706Z", 2022-09-09T22:59:50.602 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.602 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.602 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.602 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.603 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.603 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.603 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_vVrGNRKUHx_Nc7AgE45d0q7", 2022-09-09T22:59:50.603 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.604 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.604 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.604 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.604 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.605 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.605 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.605 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.605 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.191982_73.1", 2022-09-09T22:59:50.606 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.606 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16", 2022-09-09T22:59:50.606 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.191982077Z", 2022-09-09T22:59:50.606 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.607 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.607 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_jgmlG80S-C76e7qmPCPXWwY", 2022-09-09T22:59:50.607 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.607 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.608 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.608 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_jgmlG80S-C76e7qmPCPXWwY", 2022-09-09T22:59:50.608 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.608 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.609 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.609 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.609 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.609 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.610 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.610 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.610 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.196936_74.1", 2022-09-09T22:59:50.610 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.611 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16", 2022-09-09T22:59:50.611 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.196936218Z", 2022-09-09T22:59:50.611 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.611 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.612 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_jgmlG80S-C76e7qmPCPXWwY", 2022-09-09T22:59:50.612 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.612 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.612 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.613 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_jgmlG80S-C76e7qmPCPXWwY", 2022-09-09T22:59:50.613 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.613 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.613 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.614 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.614 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.614 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.614 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.615 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.615 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.211489_75.1", 2022-09-09T22:59:50.615 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.615 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17", 2022-09-09T22:59:50.616 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.211489751Z", 2022-09-09T22:59:50.616 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.616 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.616 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_MtuyPaZmCzfHOcgXjri3F-V", 2022-09-09T22:59:50.617 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.617 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.617 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.617 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_MtuyPaZmCzfHOcgXjri3F-V", 2022-09-09T22:59:50.618 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.618 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.618 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.618 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.619 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.619 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.619 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.619 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.620 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.217670_76.1", 2022-09-09T22:59:50.620 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.620 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17", 2022-09-09T22:59:50.620 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.217670475Z", 2022-09-09T22:59:50.621 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.621 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.621 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_MtuyPaZmCzfHOcgXjri3F-V", 2022-09-09T22:59:50.621 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.622 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.622 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.622 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_MtuyPaZmCzfHOcgXjri3F-V", 2022-09-09T22:59:50.622 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.623 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.623 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.623 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.623 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.624 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.624 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.624 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.624 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.236343_77.1", 2022-09-09T22:59:50.625 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.625 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18", 2022-09-09T22:59:50.625 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.236343391Z", 2022-09-09T22:59:50.625 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.626 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.626 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_bCS5W1GGiGEz-gWOOFrVrp4", 2022-09-09T22:59:50.626 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.626 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.627 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.627 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_bCS5W1GGiGEz-gWOOFrVrp4", 2022-09-09T22:59:50.627 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.627 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.628 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.628 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.628 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.628 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.629 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.629 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.629 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662762884.244083_78.1", 2022-09-09T22:59:50.629 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.630 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18", 2022-09-09T22:59:50.630 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:34:44.244083785Z", 2022-09-09T22:59:50.630 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.630 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.631 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_bCS5W1GGiGEz-gWOOFrVrp4", 2022-09-09T22:59:50.631 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.631 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.631 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.632 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_bCS5W1GGiGEz-gWOOFrVrp4", 2022-09-09T22:59:50.632 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.632 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.632 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.633 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.633 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.633 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.633 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.633 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.634 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763177.690963_87.1", 2022-09-09T22:59:50.634 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.634 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19", 2022-09-09T22:59:50.635 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:39:37.690963866Z", 2022-09-09T22:59:50.635 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.635 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.635 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.636 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.636 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.636 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.636 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_ePT9JnPeiTDUMUQthdKtkWA", 2022-09-09T22:59:50.637 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.637 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.637 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.637 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.638 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.638 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.638 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.638 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.639 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763177.695434_88.1", 2022-09-09T22:59:50.639 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.639 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19", 2022-09-09T22:59:50.639 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:39:37.695434017Z", 2022-09-09T22:59:50.640 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.640 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.640 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.640 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.641 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.641 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.641 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_ePT9JnPeiTDUMUQthdKtkWA", 2022-09-09T22:59:50.641 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.642 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.642 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.642 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.642 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.643 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.643 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.643 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.643 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763177.697042_89.1", 2022-09-09T22:59:50.644 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.644 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-19", 2022-09-09T22:59:50.644 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:39:37.697042999Z", 2022-09-09T22:59:50.644 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.644 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.645 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.645 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.645 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.645 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.646 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_tYLgE0kYjoh4utIhhPKouFZ", 2022-09-09T22:59:50.646 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.646 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.646 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.647 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.647 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.647 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.647 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.648 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.648 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763177.701435_90.1", 2022-09-09T22:59:50.648 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.648 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-19", 2022-09-09T22:59:50.649 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:39:37.701435888Z", 2022-09-09T22:59:50.649 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.649 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.649 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.650 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.650 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.650 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.651 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_tYLgE0kYjoh4utIhhPKouFZ", 2022-09-09T22:59:50.651 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.651 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.651 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.651 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.652 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.652 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.652 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.652 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.653 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763177.715694_91.1", 2022-09-09T22:59:50.653 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.653 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19", 2022-09-09T22:59:50.653 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:39:37.715694533Z", 2022-09-09T22:59:50.654 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.654 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.654 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_ePT9JnPeiTDUMUQthdKtkWA", 2022-09-09T22:59:50.654 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.655 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.655 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.655 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_ePT9JnPeiTDUMUQthdKtkWA", 2022-09-09T22:59:50.655 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.656 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.656 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.656 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.656 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.657 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.657 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.657 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.657 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763177.721058_92.1", 2022-09-09T22:59:50.658 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.658 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19", 2022-09-09T22:59:50.658 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:39:37.721058937Z", 2022-09-09T22:59:50.658 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.659 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.659 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_ePT9JnPeiTDUMUQthdKtkWA", 2022-09-09T22:59:50.659 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.659 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.660 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.660 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_ePT9JnPeiTDUMUQthdKtkWA", 2022-09-09T22:59:50.660 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.660 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.661 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.661 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.661 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.661 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.662 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.662 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.662 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763618.848484_94.1", 2022-09-09T22:59:50.662 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.663 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20", 2022-09-09T22:59:50.663 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:46:58.848484735Z", 2022-09-09T22:59:50.663 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.663 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.664 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.664 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.664 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.664 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.665 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:50.665 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.665 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.665 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.666 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.666 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.666 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.666 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.667 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.667 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763618.852755_95.1", 2022-09-09T22:59:50.667 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.667 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20", 2022-09-09T22:59:50.668 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:46:58.852755664Z", 2022-09-09T22:59:50.668 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.668 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.668 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.669 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.669 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.669 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.669 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:50.670 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.670 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.670 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.670 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.671 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.671 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.671 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.671 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.672 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763618.854410_96.1", 2022-09-09T22:59:50.672 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.672 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-20", 2022-09-09T22:59:50.672 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:46:58.854410052Z", 2022-09-09T22:59:50.673 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.673 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.673 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.673 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.674 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.674 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.674 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_4dW2sFlblKAnYCXSGCT7lId", 2022-09-09T22:59:50.674 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.675 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.675 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.675 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.675 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.676 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.676 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.676 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.676 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763618.858372_97.1", 2022-09-09T22:59:50.677 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.677 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-20", 2022-09-09T22:59:50.677 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:46:58.858372321Z", 2022-09-09T22:59:50.677 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.678 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.678 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.678 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.678 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.679 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.679 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_4dW2sFlblKAnYCXSGCT7lId", 2022-09-09T22:59:50.679 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.679 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.680 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.680 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.680 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.680 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.681 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.681 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.681 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763692.030172_110.1", 2022-09-09T22:59:50.681 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.682 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20", 2022-09-09T22:59:50.682 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:48:12.030172339Z", 2022-09-09T22:59:50.682 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.682 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.683 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:50.683 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.683 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.683 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.684 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:50.684 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.684 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.684 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.685 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.685 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.685 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.685 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.686 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.686 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763692.038589_111.1", 2022-09-09T22:59:50.686 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.686 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20", 2022-09-09T22:59:50.687 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:48:12.038589581Z", 2022-09-09T22:59:50.687 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.687 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.687 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:50.688 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.688 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.688 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.688 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:50.689 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.689 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.689 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.689 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.690 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.690 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.690 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.690 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.691 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763726.102012_112.1", 2022-09-09T22:59:50.691 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.691 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20", 2022-09-09T22:59:50.691 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:48:46.102012638Z", 2022-09-09T22:59:50.692 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.692 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.692 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:50.692 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.693 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.693 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.693 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:50.693 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 3 2022-09-09T22:59:50.694 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.694 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.694 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.694 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.695 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.695 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.695 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.695 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763726.109956_113.1", 2022-09-09T22:59:50.696 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.696 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20", 2022-09-09T22:59:50.696 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:48:46.109956678Z", 2022-09-09T22:59:50.696 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.697 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.697 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:50.697 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.697 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.697 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.698 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_krIdEiF-Kba8o94ySBDlzoB", 2022-09-09T22:59:50.698 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 3 2022-09-09T22:59:50.698 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.699 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.699 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.699 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.699 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.699 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.700 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.700 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.234301_114.1", 2022-09-09T22:59:50.700 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.700 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21", 2022-09-09T22:59:50.701 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.234301916Z", 2022-09-09T22:59:50.701 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.701 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.701 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.702 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.702 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.702 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.702 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_vO_bfm5gHdzZApgtPNMJdSt", 2022-09-09T22:59:50.703 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.703 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.703 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.703 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.704 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.704 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.704 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.704 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.705 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.239008_115.1", 2022-09-09T22:59:50.705 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.705 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21", 2022-09-09T22:59:50.705 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.239008120Z", 2022-09-09T22:59:50.706 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.706 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.706 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.706 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.707 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.707 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.707 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_vO_bfm5gHdzZApgtPNMJdSt", 2022-09-09T22:59:50.707 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.708 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.708 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.708 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.708 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.709 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.709 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.709 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.709 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.240497_116.1", 2022-09-09T22:59:50.710 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.710 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-21", 2022-09-09T22:59:50.710 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.240497138Z", 2022-09-09T22:59:50.710 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.711 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.711 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.711 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.711 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.712 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.712 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_K_Aww59oBrAI7mXNiMdGk-A", 2022-09-09T22:59:50.712 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.712 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.713 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.713 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.713 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.713 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.714 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.714 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.714 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.244134_117.1", 2022-09-09T22:59:50.714 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.715 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-21", 2022-09-09T22:59:50.715 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.244134829Z", 2022-09-09T22:59:50.715 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.715 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.716 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.716 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.716 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.716 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.717 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_K_Aww59oBrAI7mXNiMdGk-A", 2022-09-09T22:59:50.717 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.717 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.717 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.718 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.718 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.718 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.718 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.719 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.719 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.271364_118.1", 2022-09-09T22:59:50.719 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.719 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22", 2022-09-09T22:59:50.720 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.271364170Z", 2022-09-09T22:59:50.720 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.720 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.720 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.721 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.721 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.721 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.721 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_oq35luOSMiKR6aRP4lKrHCx", 2022-09-09T22:59:50.722 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.722 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.722 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.722 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.723 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.723 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.723 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.723 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.724 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.276910_119.1", 2022-09-09T22:59:50.724 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.724 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22", 2022-09-09T22:59:50.724 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.276910204Z", 2022-09-09T22:59:50.725 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.725 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.725 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.725 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.726 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.726 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.726 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_oq35luOSMiKR6aRP4lKrHCx", 2022-09-09T22:59:50.726 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.727 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.727 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.727 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.727 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.728 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.728 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.728 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.728 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.278731_120.1", 2022-09-09T22:59:50.729 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.729 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-22", 2022-09-09T22:59:50.729 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.278731955Z", 2022-09-09T22:59:50.729 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.730 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.730 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.730 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.730 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.731 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.731 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_oBwxyUPEtECOF6uEAIe2K67", 2022-09-09T22:59:50.731 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.731 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.732 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.732 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.732 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.732 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.733 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.733 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.733 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.284005_121.1", 2022-09-09T22:59:50.733 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.734 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-22", 2022-09-09T22:59:50.734 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.284005465Z", 2022-09-09T22:59:50.734 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.734 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.735 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.735 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.735 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.735 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.736 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_oBwxyUPEtECOF6uEAIe2K67", 2022-09-09T22:59:50.736 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.736 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.736 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.737 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.737 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.737 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.737 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.738 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.738 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.330455_122.1", 2022-09-09T22:59:50.738 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.738 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23", 2022-09-09T22:59:50.739 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.330455662Z", 2022-09-09T22:59:50.739 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.739 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.739 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.740 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.740 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.740 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.740 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_TBNrJnEgH9G6t0MeX057y1f", 2022-09-09T22:59:50.741 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.741 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.741 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.741 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.742 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.742 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.742 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.742 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.743 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.335107_123.1", 2022-09-09T22:59:50.743 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.743 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23", 2022-09-09T22:59:50.743 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.335107502Z", 2022-09-09T22:59:50.744 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.744 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.744 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.744 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.745 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.745 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.745 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_TBNrJnEgH9G6t0MeX057y1f", 2022-09-09T22:59:50.745 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.746 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.746 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.746 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.746 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.747 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.747 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.747 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.747 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.336855_124.1", 2022-09-09T22:59:50.748 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.748 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-23", 2022-09-09T22:59:50.748 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.336855486Z", 2022-09-09T22:59:50.748 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.749 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.749 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.749 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.749 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.750 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.750 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_DGKEfxrBasYDPbGTJt7zCFo", 2022-09-09T22:59:50.750 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.750 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.751 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.751 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.751 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.751 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.752 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.752 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.752 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.340827_125.1", 2022-09-09T22:59:50.752 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.753 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-23", 2022-09-09T22:59:50.753 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.340827648Z", 2022-09-09T22:59:50.753 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.753 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.753 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.754 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.754 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.754 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.755 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_DGKEfxrBasYDPbGTJt7zCFo", 2022-09-09T22:59:50.755 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.755 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.755 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.756 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.756 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.756 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.756 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.757 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.757 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.368888_126.1", 2022-09-09T22:59:50.757 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.757 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21", 2022-09-09T22:59:50.758 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.368888518Z", 2022-09-09T22:59:50.758 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.758 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.758 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_vO_bfm5gHdzZApgtPNMJdSt", 2022-09-09T22:59:50.759 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.759 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.759 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.759 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_vO_bfm5gHdzZApgtPNMJdSt", 2022-09-09T22:59:50.760 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.760 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.760 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.760 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.761 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.761 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.761 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.761 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.762 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.374441_127.1", 2022-09-09T22:59:50.762 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.762 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21", 2022-09-09T22:59:50.762 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.374441110Z", 2022-09-09T22:59:50.763 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.763 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.763 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_vO_bfm5gHdzZApgtPNMJdSt", 2022-09-09T22:59:50.763 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.764 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.764 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.764 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_vO_bfm5gHdzZApgtPNMJdSt", 2022-09-09T22:59:50.764 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.765 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.765 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.765 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.765 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.765 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.766 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.766 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.766 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.389455_128.1", 2022-09-09T22:59:50.766 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.767 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22", 2022-09-09T22:59:50.767 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.389455212Z", 2022-09-09T22:59:50.767 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.767 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.768 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_oq35luOSMiKR6aRP4lKrHCx", 2022-09-09T22:59:50.768 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.768 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.768 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.769 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_oq35luOSMiKR6aRP4lKrHCx", 2022-09-09T22:59:50.769 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.769 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.769 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.770 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.770 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.770 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.770 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.771 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.771 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.396062_129.1", 2022-09-09T22:59:50.771 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.771 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22", 2022-09-09T22:59:50.772 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.396062782Z", 2022-09-09T22:59:50.772 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.772 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.772 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_oq35luOSMiKR6aRP4lKrHCx", 2022-09-09T22:59:50.773 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.773 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.773 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.773 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_oq35luOSMiKR6aRP4lKrHCx", 2022-09-09T22:59:50.774 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.774 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.774 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.774 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.775 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.775 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.775 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.775 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.776 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.417475_130.1", 2022-09-09T22:59:50.776 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.776 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23", 2022-09-09T22:59:50.776 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.417475123Z", 2022-09-09T22:59:50.777 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.777 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.777 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_TBNrJnEgH9G6t0MeX057y1f", 2022-09-09T22:59:50.777 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.778 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.778 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.778 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_TBNrJnEgH9G6t0MeX057y1f", 2022-09-09T22:59:50.778 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.779 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.779 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.779 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.779 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.780 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.780 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.780 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.780 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763763.426933_131.1", 2022-09-09T22:59:50.781 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.781 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23", 2022-09-09T22:59:50.781 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:49:23.426933442Z", 2022-09-09T22:59:50.781 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.782 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.782 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_TBNrJnEgH9G6t0MeX057y1f", 2022-09-09T22:59:50.782 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.782 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.783 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.783 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_TBNrJnEgH9G6t0MeX057y1f", 2022-09-09T22:59:50.783 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.783 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.784 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.784 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.784 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.784 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.785 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.785 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.785 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.243136_133.1", 2022-09-09T22:59:50.785 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.786 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24", 2022-09-09T22:59:50.786 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.243136650Z", 2022-09-09T22:59:50.786 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.786 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.787 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.787 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.787 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.787 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.788 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:50.788 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.788 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.788 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.789 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.789 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.789 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.789 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.790 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.790 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.247428_134.1", 2022-09-09T22:59:50.790 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.790 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24", 2022-09-09T22:59:50.791 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.247428396Z", 2022-09-09T22:59:50.791 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.791 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.791 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.792 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.792 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.792 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.793 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:50.793 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.793 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.794 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.794 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.794 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.795 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.795 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.795 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.795 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.249044_135.1", 2022-09-09T22:59:50.796 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.796 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-24", 2022-09-09T22:59:50.796 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.249044082Z", 2022-09-09T22:59:50.796 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.796 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.797 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.797 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.797 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.798 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.798 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_GPbPaiqkbVjC5D9wQQQP2jZ", 2022-09-09T22:59:50.798 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.798 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.799 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.799 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.799 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.799 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.800 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.800 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.800 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.253611_136.1", 2022-09-09T22:59:50.800 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.801 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-24", 2022-09-09T22:59:50.801 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.253611226Z", 2022-09-09T22:59:50.801 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.801 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.802 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.802 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.802 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.802 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.803 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_GPbPaiqkbVjC5D9wQQQP2jZ", 2022-09-09T22:59:50.803 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.803 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.803 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.804 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.804 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.804 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.804 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.805 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.805 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.282622_137.1", 2022-09-09T22:59:50.805 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.805 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25", 2022-09-09T22:59:50.806 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.282622692Z", 2022-09-09T22:59:50.806 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.806 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.806 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.807 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.807 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.807 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.807 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:50.808 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.808 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.808 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.808 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.809 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.809 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.809 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.809 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.810 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.287087_138.1", 2022-09-09T22:59:50.810 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.810 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25", 2022-09-09T22:59:50.810 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.287087929Z", 2022-09-09T22:59:50.811 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.811 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.811 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.811 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.812 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.812 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.812 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:50.812 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.813 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.813 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.813 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.813 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.814 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.814 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.814 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.814 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.288605_139.1", 2022-09-09T22:59:50.815 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.815 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-25", 2022-09-09T22:59:50.815 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.288605630Z", 2022-09-09T22:59:50.816 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.816 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.816 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.816 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.816 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.817 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.817 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_ZPxDZ2lg7jpxqasfBs-kwky", 2022-09-09T22:59:50.817 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.817 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.818 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.818 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.818 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.818 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.819 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.819 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.819 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.292651_140.1", 2022-09-09T22:59:50.819 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.820 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-25", 2022-09-09T22:59:50.820 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.292651882Z", 2022-09-09T22:59:50.820 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.820 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.821 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.821 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.821 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.821 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.822 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_ZPxDZ2lg7jpxqasfBs-kwky", 2022-09-09T22:59:50.822 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.822 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.822 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.823 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.823 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.823 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.823 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.824 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.824 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.347788_141.1", 2022-09-09T22:59:50.824 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.824 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26", 2022-09-09T22:59:50.825 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.347788063Z", 2022-09-09T22:59:50.825 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.825 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.825 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.826 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.826 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.826 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.826 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:50.827 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.827 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.827 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.827 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.828 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.828 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.828 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.828 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.829 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.352949_142.1", 2022-09-09T22:59:50.829 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.829 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26", 2022-09-09T22:59:50.829 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.352949747Z", 2022-09-09T22:59:50.830 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.830 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.830 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.830 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.831 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.831 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.831 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:50.831 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.832 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.832 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.832 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.832 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.833 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.833 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.833 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.833 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.354746_143.1", 2022-09-09T22:59:50.834 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.834 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-26", 2022-09-09T22:59:50.834 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.354746994Z", 2022-09-09T22:59:50.834 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.835 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.835 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.835 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.836 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.836 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.836 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_sgAtkuvW-X_3Moy5BHrhaBp", 2022-09-09T22:59:50.836 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.837 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.837 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.837 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.838 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.838 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.838 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.838 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.839 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.358613_144.1", 2022-09-09T22:59:50.839 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.839 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-26", 2022-09-09T22:59:50.839 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.358613174Z", 2022-09-09T22:59:50.840 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.840 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.840 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.841 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.841 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.841 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.841 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_sgAtkuvW-X_3Moy5BHrhaBp", 2022-09-09T22:59:50.842 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.842 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.842 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.843 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.843 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.843 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.843 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.844 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.844 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.386362_145.1", 2022-09-09T22:59:50.844 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.844 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24", 2022-09-09T22:59:50.845 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.386362667Z", 2022-09-09T22:59:50.845 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.845 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.845 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:50.846 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.846 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.846 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.847 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:50.847 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.847 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.847 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.847 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.848 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.848 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.848 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.848 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.849 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.392952_146.1", 2022-09-09T22:59:50.849 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.849 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24", 2022-09-09T22:59:50.849 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.392952544Z", 2022-09-09T22:59:50.850 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.850 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.850 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:50.851 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.851 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.851 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.851 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:50.852 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.852 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.852 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.852 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.852 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.853 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.853 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.853 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.853 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.405357_147.1", 2022-09-09T22:59:50.854 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.854 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24", 2022-09-09T22:59:50.854 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.405357667Z", 2022-09-09T22:59:50.855 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.855 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.855 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:50.855 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.855 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.856 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.856 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:50.856 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 3 2022-09-09T22:59:50.856 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.857 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.857 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.857 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.857 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.858 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.858 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.858 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.413164_148.1", 2022-09-09T22:59:50.858 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.859 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24", 2022-09-09T22:59:50.859 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.413164542Z", 2022-09-09T22:59:50.859 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.859 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.860 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:50.860 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.860 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.860 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.861 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_-xPKQ4VHqAyPwIXOPDgpLPn", 2022-09-09T22:59:50.861 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 3 2022-09-09T22:59:50.861 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.861 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.862 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.862 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.862 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.862 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.863 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.863 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.428806_149.1", 2022-09-09T22:59:50.863 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.863 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25", 2022-09-09T22:59:50.864 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.428806301Z", 2022-09-09T22:59:50.864 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.864 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.864 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:50.865 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.865 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.865 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.865 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:50.866 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.866 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.866 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.866 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.867 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.867 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.867 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.867 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.868 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.436953_150.1", 2022-09-09T22:59:50.868 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.868 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25", 2022-09-09T22:59:50.868 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.436953749Z", 2022-09-09T22:59:50.869 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.869 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.869 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:50.869 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.870 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.870 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.870 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:50.870 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.871 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.871 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.871 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.871 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.872 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.872 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.872 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.872 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.460100_151.1", 2022-09-09T22:59:50.873 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.873 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25", 2022-09-09T22:59:50.873 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.460100101Z", 2022-09-09T22:59:50.873 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.874 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.874 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:50.874 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.874 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.875 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.875 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:50.875 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 3 2022-09-09T22:59:50.875 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.876 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.876 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.876 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.876 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.877 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.877 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.877 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.471650_152.1", 2022-09-09T22:59:50.877 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.878 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25", 2022-09-09T22:59:50.878 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.471650692Z", 2022-09-09T22:59:50.878 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.878 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.879 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:50.879 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.879 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.879 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.880 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_g_dWd0a5CmSIxHtX3yQG2g1", 2022-09-09T22:59:50.880 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 3 2022-09-09T22:59:50.880 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.880 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.881 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.881 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.881 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.881 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.882 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.882 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.513547_153.1", 2022-09-09T22:59:50.882 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.882 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26", 2022-09-09T22:59:50.883 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.513547080Z", 2022-09-09T22:59:50.883 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.883 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.883 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:50.884 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.884 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.884 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.884 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:50.885 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.885 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.885 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.885 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.886 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.886 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.886 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.886 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.887 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.520151_154.1", 2022-09-09T22:59:50.887 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.887 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26", 2022-09-09T22:59:50.887 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.520151282Z", 2022-09-09T22:59:50.888 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.888 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.888 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:50.888 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.889 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.889 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.889 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:50.889 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.890 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.890 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.890 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.890 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.891 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.891 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.891 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.891 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.543475_155.1", 2022-09-09T22:59:50.892 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.892 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26", 2022-09-09T22:59:50.892 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.543475276Z", 2022-09-09T22:59:50.892 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.893 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.893 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:50.893 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.893 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.894 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.894 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:50.894 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 3 2022-09-09T22:59:50.894 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.895 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.895 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.895 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.895 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.896 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.896 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.896 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763826.550710_156.1", 2022-09-09T22:59:50.896 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.897 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26", 2022-09-09T22:59:50.897 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:26.550710511Z", 2022-09-09T22:59:50.897 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.897 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.898 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:50.898 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.898 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.898 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.899 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_1-z_5RLgXlVlsP9bNCn3Z64", 2022-09-09T22:59:50.899 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 3 2022-09-09T22:59:50.899 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.899 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.900 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.900 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.900 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.900 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.901 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.901 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.587061_165.1", 2022-09-09T22:59:50.901 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.901 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-27:4b05608c-7197-4e9b-bd29-7989607fa277.4354.27", 2022-09-09T22:59:50.902 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.587061072Z", 2022-09-09T22:59:50.902 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.902 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.902 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.903 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.903 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.903 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.903 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_idGRpIZD_cBqHmSr0C8anAE", 2022-09-09T22:59:50.904 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.904 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.904 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.904 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.905 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.905 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.905 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.905 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.906 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.591517_166.1", 2022-09-09T22:59:50.906 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.906 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-27:4b05608c-7197-4e9b-bd29-7989607fa277.4354.27", 2022-09-09T22:59:50.906 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.591517503Z", 2022-09-09T22:59:50.907 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.907 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.907 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.907 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.908 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.908 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.908 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_idGRpIZD_cBqHmSr0C8anAE", 2022-09-09T22:59:50.908 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.909 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.909 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.909 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.909 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.910 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.910 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.910 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.910 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.593291_167.1", 2022-09-09T22:59:50.911 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.911 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-27", 2022-09-09T22:59:50.911 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.593291391Z", 2022-09-09T22:59:50.911 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.912 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.912 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.912 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.912 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.913 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.913 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_fil3WVqAmCZtFEsYnU9kPha", 2022-09-09T22:59:50.913 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.913 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.914 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.914 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.914 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.914 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.915 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.915 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.915 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.597347_168.1", 2022-09-09T22:59:50.915 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.916 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-27", 2022-09-09T22:59:50.916 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.597347657Z", 2022-09-09T22:59:50.916 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.916 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.917 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.917 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.917 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.917 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.918 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_fil3WVqAmCZtFEsYnU9kPha", 2022-09-09T22:59:50.918 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.918 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.918 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.919 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.919 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.919 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.919 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.920 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.920 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.629370_169.1", 2022-09-09T22:59:50.920 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.920 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-28:4b05608c-7197-4e9b-bd29-7989607fa277.4354.28", 2022-09-09T22:59:50.921 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.629370653Z", 2022-09-09T22:59:50.921 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.921 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.921 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.922 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.922 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.922 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.922 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_hWVXDIs2ffmJdaY5i2EUA49", 2022-09-09T22:59:50.923 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.923 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.923 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.923 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.924 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.924 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.924 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.924 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.925 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.634323_170.1", 2022-09-09T22:59:50.925 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.925 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-28:4b05608c-7197-4e9b-bd29-7989607fa277.4354.28", 2022-09-09T22:59:50.925 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.634323436Z", 2022-09-09T22:59:50.926 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.926 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.926 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.926 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.927 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.927 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.927 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_hWVXDIs2ffmJdaY5i2EUA49", 2022-09-09T22:59:50.927 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.928 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.928 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.928 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.928 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.929 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.929 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.929 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.929 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.636142_171.1", 2022-09-09T22:59:50.930 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.930 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-28", 2022-09-09T22:59:50.930 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.636142661Z", 2022-09-09T22:59:50.930 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.931 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.931 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.931 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.931 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.932 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.932 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_Rr2OoVgGeS8W2Ih7H5QimUB", 2022-09-09T22:59:50.932 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.932 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.933 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.933 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.933 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.933 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.934 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.934 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.934 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.640366_172.1", 2022-09-09T22:59:50.934 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.935 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-28", 2022-09-09T22:59:50.935 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.640366460Z", 2022-09-09T22:59:50.935 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.935 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.936 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.936 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.936 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.936 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.937 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_Rr2OoVgGeS8W2Ih7H5QimUB", 2022-09-09T22:59:50.937 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.937 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.937 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.938 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.938 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.938 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.938 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.939 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.939 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.691642_173.1", 2022-09-09T22:59:50.939 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.939 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-29:4b05608c-7197-4e9b-bd29-7989607fa277.4354.29", 2022-09-09T22:59:50.940 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.691642561Z", 2022-09-09T22:59:50.940 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.940 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.940 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.941 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.941 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.941 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.941 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_c7Bu5fk9awm3ApVCIJ8jMq6", 2022-09-09T22:59:50.942 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.942 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.942 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.942 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.943 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.943 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.943 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.943 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.944 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.695580_174.1", 2022-09-09T22:59:50.944 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.944 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-29:4b05608c-7197-4e9b-bd29-7989607fa277.4354.29", 2022-09-09T22:59:50.944 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.695580156Z", 2022-09-09T22:59:50.945 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.945 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.945 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.945 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.946 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.946 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.946 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_c7Bu5fk9awm3ApVCIJ8jMq6", 2022-09-09T22:59:50.946 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.947 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.947 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.947 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.947 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.948 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.948 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.948 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.948 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.696997_175.1", 2022-09-09T22:59:50.949 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.949 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-29", 2022-09-09T22:59:50.949 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.696997195Z", 2022-09-09T22:59:50.949 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.950 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.950 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.950 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.950 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.951 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.951 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_oMWT857Kx25Jy8rmeTqaRTI", 2022-09-09T22:59:50.951 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.951 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.952 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.952 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.952 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.952 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.953 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.953 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.953 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.701289_176.1", 2022-09-09T22:59:50.953 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.954 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-29", 2022-09-09T22:59:50.954 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.701289662Z", 2022-09-09T22:59:50.954 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.954 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.955 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.955 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.955 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.955 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.956 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_oMWT857Kx25Jy8rmeTqaRTI", 2022-09-09T22:59:50.956 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.956 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.956 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.957 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.957 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.957 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.958 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.958 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.958 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.733662_177.1", 2022-09-09T22:59:50.958 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.959 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-27:4b05608c-7197-4e9b-bd29-7989607fa277.4354.27", 2022-09-09T22:59:50.959 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.733662223Z", 2022-09-09T22:59:50.959 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.959 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.960 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_idGRpIZD_cBqHmSr0C8anAE", 2022-09-09T22:59:50.960 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.960 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.960 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.961 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_idGRpIZD_cBqHmSr0C8anAE", 2022-09-09T22:59:50.961 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.961 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.961 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.962 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.962 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.962 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.962 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.963 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.963 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.739796_178.1", 2022-09-09T22:59:50.963 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.963 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-27:4b05608c-7197-4e9b-bd29-7989607fa277.4354.27", 2022-09-09T22:59:50.964 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.739796617Z", 2022-09-09T22:59:50.964 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.964 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.964 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_idGRpIZD_cBqHmSr0C8anAE", 2022-09-09T22:59:50.965 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.965 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.965 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.965 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_idGRpIZD_cBqHmSr0C8anAE", 2022-09-09T22:59:50.966 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.966 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.966 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.966 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.967 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.967 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.967 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.967 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.968 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.815014_179.1", 2022-09-09T22:59:50.968 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.968 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-28:4b05608c-7197-4e9b-bd29-7989607fa277.4354.28", 2022-09-09T22:59:50.968 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.815014271Z", 2022-09-09T22:59:50.969 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.969 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.969 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_hWVXDIs2ffmJdaY5i2EUA49", 2022-09-09T22:59:50.969 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.970 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.970 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.970 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_hWVXDIs2ffmJdaY5i2EUA49", 2022-09-09T22:59:50.970 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.971 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.971 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.971 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.971 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.972 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.972 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.972 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.972 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.823504_180.1", 2022-09-09T22:59:50.973 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.973 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-28:4b05608c-7197-4e9b-bd29-7989607fa277.4354.28", 2022-09-09T22:59:50.973 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.823504725Z", 2022-09-09T22:59:50.973 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.974 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.974 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_hWVXDIs2ffmJdaY5i2EUA49", 2022-09-09T22:59:50.974 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.974 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.975 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.975 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_hWVXDIs2ffmJdaY5i2EUA49", 2022-09-09T22:59:50.975 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.975 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.976 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.976 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.976 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.976 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.977 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.977 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.977 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.861162_181.1", 2022-09-09T22:59:50.977 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.978 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-29:4b05608c-7197-4e9b-bd29-7989607fa277.4354.29", 2022-09-09T22:59:50.978 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.861162453Z", 2022-09-09T22:59:50.978 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.978 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.979 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_c7Bu5fk9awm3ApVCIJ8jMq6", 2022-09-09T22:59:50.979 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.979 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.979 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.980 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_c7Bu5fk9awm3ApVCIJ8jMq6", 2022-09-09T22:59:50.980 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.980 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.980 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.981 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.981 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.981 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.981 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.982 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.982 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.868518_182.1", 2022-09-09T22:59:50.982 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.982 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-29:4b05608c-7197-4e9b-bd29-7989607fa277.4354.29", 2022-09-09T22:59:50.983 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.868518409Z", 2022-09-09T22:59:50.983 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.983 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.983 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_c7Bu5fk9awm3ApVCIJ8jMq6", 2022-09-09T22:59:50.984 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.984 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.984 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.984 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_c7Bu5fk9awm3ApVCIJ8jMq6", 2022-09-09T22:59:50.985 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:50.985 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.985 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.985 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.986 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.986 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.986 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.986 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.987 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.905416_183.1", 2022-09-09T22:59:50.987 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.987 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-30:4b05608c-7197-4e9b-bd29-7989607fa277.4354.30", 2022-09-09T22:59:50.987 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.905416190Z", 2022-09-09T22:59:50.988 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.988 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.988 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.988 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.989 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.989 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.989 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xuD0Sb75U_5LCkKCpVp22OC", 2022-09-09T22:59:50.989 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.990 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.990 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.990 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:50.991 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.991 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.991 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.991 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.991 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.909698_184.1", 2022-09-09T22:59:50.992 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:50.992 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-30:4b05608c-7197-4e9b-bd29-7989607fa277.4354.30", 2022-09-09T22:59:50.992 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.909698021Z", 2022-09-09T22:59:50.992 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.993 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.993 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.993 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.994 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.994 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.994 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xuD0Sb75U_5LCkKCpVp22OC", 2022-09-09T22:59:50.994 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.995 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.995 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:50.995 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:50.995 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.996 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:50.996 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.996 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:50.996 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.911381_185.1", 2022-09-09T22:59:50.996 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:50.997 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-30", 2022-09-09T22:59:50.997 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.911381624Z", 2022-09-09T22:59:50.997 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:50.997 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:50.998 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:50.998 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:50.998 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.998 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:50.999 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_rADCtMkJa27KS0fuztGkOJ4", 2022-09-09T22:59:50.999 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:50.999 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:50.999 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.000 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.000 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.000 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.000 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.001 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.001 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.915543_186.1", 2022-09-09T22:59:51.001 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:51.001 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-30", 2022-09-09T22:59:51.002 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.915543203Z", 2022-09-09T22:59:51.002 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.002 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.002 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.003 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.003 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.003 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.003 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_rADCtMkJa27KS0fuztGkOJ4", 2022-09-09T22:59:51.004 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.004 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.004 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.004 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.005 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.005 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.005 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.005 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.006 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.957457_187.1", 2022-09-09T22:59:51.006 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.006 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-31:4b05608c-7197-4e9b-bd29-7989607fa277.4354.31", 2022-09-09T22:59:51.006 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.957457179Z", 2022-09-09T22:59:51.007 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.007 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.007 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.007 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.008 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.008 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.008 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_RG9oYzLzZKilowVQDrZqfZp", 2022-09-09T22:59:51.008 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.009 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.009 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.009 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.009 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.010 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.010 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.010 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.011 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.961791_188.1", 2022-09-09T22:59:51.011 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.011 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-31:4b05608c-7197-4e9b-bd29-7989607fa277.4354.31", 2022-09-09T22:59:51.011 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.961791758Z", 2022-09-09T22:59:51.012 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.012 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.012 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.012 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.013 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.013 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.013 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_RG9oYzLzZKilowVQDrZqfZp", 2022-09-09T22:59:51.013 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.014 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.014 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.014 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.014 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.015 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.015 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.015 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.015 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.963310_189.1", 2022-09-09T22:59:51.016 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:51.016 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-31", 2022-09-09T22:59:51.016 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.963310744Z", 2022-09-09T22:59:51.016 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.017 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.017 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.017 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.017 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.018 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.018 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_u7CGf9AlYIHVBDPwmE_8lDH", 2022-09-09T22:59:51.018 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.018 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.019 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.019 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.019 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.020 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.020 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.020 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.020 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763987.966754_190.1", 2022-09-09T22:59:51.021 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:51.021 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-31", 2022-09-09T22:59:51.021 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:07.966754250Z", 2022-09-09T22:59:51.021 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.022 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.022 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.022 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.022 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.023 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.023 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_u7CGf9AlYIHVBDPwmE_8lDH", 2022-09-09T22:59:51.023 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.023 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.024 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.024 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.024 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.024 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.025 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.025 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.025 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.029534_191.1", 2022-09-09T22:59:51.026 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.026 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-32:4b05608c-7197-4e9b-bd29-7989607fa277.4354.32", 2022-09-09T22:59:51.026 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.029534998Z", 2022-09-09T22:59:51.026 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.027 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.027 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.027 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.027 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.028 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.028 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_Uke9qkxor72jlh3_r-i6E2w", 2022-09-09T22:59:51.028 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.028 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.029 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.029 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.029 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.029 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.030 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.030 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.030 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.036126_192.1", 2022-09-09T22:59:51.031 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.031 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-32:4b05608c-7197-4e9b-bd29-7989607fa277.4354.32", 2022-09-09T22:59:51.031 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.036126389Z", 2022-09-09T22:59:51.031 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.032 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.032 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.032 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.032 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.033 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.033 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_Uke9qkxor72jlh3_r-i6E2w", 2022-09-09T22:59:51.033 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.033 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.034 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.034 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.034 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.035 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.035 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.035 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.035 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.040740_193.1", 2022-09-09T22:59:51.036 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:51.036 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-32", 2022-09-09T22:59:51.036 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.040740810Z", 2022-09-09T22:59:51.036 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.037 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.037 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.037 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.037 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.038 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.038 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_SYdiOqvbUDZ5DpIXlyitzm5", 2022-09-09T22:59:51.038 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.038 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.039 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.039 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.039 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.039 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.040 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.040 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.040 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.045233_194.1", 2022-09-09T22:59:51.040 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:51.041 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-32", 2022-09-09T22:59:51.041 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.045233474Z", 2022-09-09T22:59:51.041 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.041 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.042 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.042 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.042 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.043 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.043 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_SYdiOqvbUDZ5DpIXlyitzm5", 2022-09-09T22:59:51.043 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.043 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.044 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.044 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.044 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.044 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.045 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.045 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.045 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.086181_195.1", 2022-09-09T22:59:51.045 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.046 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-30:4b05608c-7197-4e9b-bd29-7989607fa277.4354.30", 2022-09-09T22:59:51.046 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.086181840Z", 2022-09-09T22:59:51.046 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.046 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.047 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xuD0Sb75U_5LCkKCpVp22OC", 2022-09-09T22:59:51.047 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.047 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.047 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.048 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xuD0Sb75U_5LCkKCpVp22OC", 2022-09-09T22:59:51.048 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:51.048 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.048 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.049 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.049 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.049 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.049 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.050 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.050 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.092819_196.1", 2022-09-09T22:59:51.050 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.050 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-30:4b05608c-7197-4e9b-bd29-7989607fa277.4354.30", 2022-09-09T22:59:51.051 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.092819829Z", 2022-09-09T22:59:51.051 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.051 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.051 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xuD0Sb75U_5LCkKCpVp22OC", 2022-09-09T22:59:51.052 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.052 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.052 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.053 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xuD0Sb75U_5LCkKCpVp22OC", 2022-09-09T22:59:51.053 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:51.053 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.053 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.054 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.054 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.054 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.054 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.055 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.055 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.131992_197.1", 2022-09-09T22:59:51.055 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.055 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-31:4b05608c-7197-4e9b-bd29-7989607fa277.4354.31", 2022-09-09T22:59:51.056 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.131992122Z", 2022-09-09T22:59:51.056 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.056 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.056 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_RG9oYzLzZKilowVQDrZqfZp", 2022-09-09T22:59:51.057 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.057 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.057 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.057 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_RG9oYzLzZKilowVQDrZqfZp", 2022-09-09T22:59:51.058 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:51.058 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.058 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.058 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.059 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.059 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.059 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.059 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.060 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.140406_198.1", 2022-09-09T22:59:51.060 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.060 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-31:4b05608c-7197-4e9b-bd29-7989607fa277.4354.31", 2022-09-09T22:59:51.060 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.140406171Z", 2022-09-09T22:59:51.061 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.061 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.061 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_RG9oYzLzZKilowVQDrZqfZp", 2022-09-09T22:59:51.061 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.062 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.062 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.062 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_RG9oYzLzZKilowVQDrZqfZp", 2022-09-09T22:59:51.063 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:51.063 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.063 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.063 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.064 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.064 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.064 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.064 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.065 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.234470_199.1", 2022-09-09T22:59:51.065 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.065 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-32:4b05608c-7197-4e9b-bd29-7989607fa277.4354.32", 2022-09-09T22:59:51.065 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.234470787Z", 2022-09-09T22:59:51.066 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.066 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.066 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_Uke9qkxor72jlh3_r-i6E2w", 2022-09-09T22:59:51.067 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.067 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.067 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.067 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_Uke9qkxor72jlh3_r-i6E2w", 2022-09-09T22:59:51.068 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:51.068 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.068 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.068 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.069 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.069 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.069 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.069 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.070 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.242680_200.1", 2022-09-09T22:59:51.070 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.070 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-32:4b05608c-7197-4e9b-bd29-7989607fa277.4354.32", 2022-09-09T22:59:51.070 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.242680646Z", 2022-09-09T22:59:51.071 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.071 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.071 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_Uke9qkxor72jlh3_r-i6E2w", 2022-09-09T22:59:51.071 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.072 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.072 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.072 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_Uke9qkxor72jlh3_r-i6E2w", 2022-09-09T22:59:51.072 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:51.073 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.073 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.073 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.073 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.074 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.074 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.074 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.074 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.342621_201.1", 2022-09-09T22:59:51.075 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.075 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-33:4b05608c-7197-4e9b-bd29-7989607fa277.4354.33", 2022-09-09T22:59:51.075 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.342621802Z", 2022-09-09T22:59:51.075 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.076 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.076 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.076 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.077 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.077 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.077 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xQnGVMAoouaLAz4qR87BBiu", 2022-09-09T22:59:51.077 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.078 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.078 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.078 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.078 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.079 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.079 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.079 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.079 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.348018_202.1", 2022-09-09T22:59:51.080 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.080 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-33:4b05608c-7197-4e9b-bd29-7989607fa277.4354.33", 2022-09-09T22:59:51.080 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.348018686Z", 2022-09-09T22:59:51.080 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.081 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.081 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.081 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.081 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.082 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.082 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xQnGVMAoouaLAz4qR87BBiu", 2022-09-09T22:59:51.082 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.082 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.083 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.083 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.083 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.084 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.084 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.084 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.084 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.352316_203.1", 2022-09-09T22:59:51.085 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:51.085 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-33", 2022-09-09T22:59:51.085 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.352316326Z", 2022-09-09T22:59:51.085 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.086 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.086 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.086 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.086 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.087 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.087 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_py9xB0upsDT3rp1yJKL0fW7", 2022-09-09T22:59:51.087 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.087 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.087 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.088 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.088 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.088 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.088 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.089 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.089 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.367301_204.1", 2022-09-09T22:59:51.089 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:51.089 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-33", 2022-09-09T22:59:51.090 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.367301625Z", 2022-09-09T22:59:51.090 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.090 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.091 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.091 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.091 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.091 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.092 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_py9xB0upsDT3rp1yJKL0fW7", 2022-09-09T22:59:51.092 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.092 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.092 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.093 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.093 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.093 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.094 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.094 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.094 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.433156_205.1", 2022-09-09T22:59:51.094 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.095 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-34:4b05608c-7197-4e9b-bd29-7989607fa277.4354.34", 2022-09-09T22:59:51.095 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.433156697Z", 2022-09-09T22:59:51.095 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.095 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.096 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.096 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.096 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.096 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.097 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_4_jQEFJZa-xniCNap3y5x4k", 2022-09-09T22:59:51.097 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.097 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.097 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.098 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.098 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.098 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.098 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.099 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.099 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.438238_206.1", 2022-09-09T22:59:51.099 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.099 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-34:4b05608c-7197-4e9b-bd29-7989607fa277.4354.34", 2022-09-09T22:59:51.100 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.438238112Z", 2022-09-09T22:59:51.100 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.100 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.100 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.101 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.101 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.101 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.101 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_4_jQEFJZa-xniCNap3y5x4k", 2022-09-09T22:59:51.102 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.102 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.102 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.102 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.103 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.103 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.103 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.103 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.104 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.439911_207.1", 2022-09-09T22:59:51.104 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:51.104 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-34", 2022-09-09T22:59:51.104 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.439911797Z", 2022-09-09T22:59:51.105 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.105 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.105 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.105 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.106 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.106 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.106 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_i4pt2ScTLCpaSy-sTWzcDsg", 2022-09-09T22:59:51.106 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.107 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.107 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.107 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.107 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.108 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.108 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.108 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.108 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.444301_208.1", 2022-09-09T22:59:51.109 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:51.109 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-34", 2022-09-09T22:59:51.109 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.444301272Z", 2022-09-09T22:59:51.109 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.110 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.110 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.110 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.111 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.111 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.111 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_i4pt2ScTLCpaSy-sTWzcDsg", 2022-09-09T22:59:51.111 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.112 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.112 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.112 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.112 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.113 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.113 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.113 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.113 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.490350_209.1", 2022-09-09T22:59:51.114 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.114 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-35:4b05608c-7197-4e9b-bd29-7989607fa277.4354.35", 2022-09-09T22:59:51.114 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.490350184Z", 2022-09-09T22:59:51.114 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.115 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.115 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.115 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.115 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.116 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.116 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xarcxlcdFlngGuAvxUBGTlW", 2022-09-09T22:59:51.116 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.116 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.117 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.117 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.117 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.117 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.118 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.118 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.118 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.495064_210.1", 2022-09-09T22:59:51.118 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.119 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-35:4b05608c-7197-4e9b-bd29-7989607fa277.4354.35", 2022-09-09T22:59:51.119 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.495064159Z", 2022-09-09T22:59:51.119 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.120 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.120 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.120 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.120 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.121 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.121 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xarcxlcdFlngGuAvxUBGTlW", 2022-09-09T22:59:51.121 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.121 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.122 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.122 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.122 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.123 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.123 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.123 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.123 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.496525_211.1", 2022-09-09T22:59:51.124 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:51.124 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-35", 2022-09-09T22:59:51.124 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.496525235Z", 2022-09-09T22:59:51.124 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.125 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.125 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.125 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.125 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.126 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.126 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_b22QH41XyKk67qZyODLiaOx", 2022-09-09T22:59:51.126 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.126 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.127 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.127 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.127 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.128 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.128 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.128 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.128 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.500218_212.1", 2022-09-09T22:59:51.129 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:51.129 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-35", 2022-09-09T22:59:51.129 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.500218886Z", 2022-09-09T22:59:51.129 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.130 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.130 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.130 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.130 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.131 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.131 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_b22QH41XyKk67qZyODLiaOx", 2022-09-09T22:59:51.131 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.131 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.132 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.132 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.132 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.132 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.133 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.133 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.133 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.527104_213.1", 2022-09-09T22:59:51.133 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.134 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-33:4b05608c-7197-4e9b-bd29-7989607fa277.4354.33", 2022-09-09T22:59:51.134 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.527104670Z", 2022-09-09T22:59:51.134 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.134 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.135 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xQnGVMAoouaLAz4qR87BBiu", 2022-09-09T22:59:51.135 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.135 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.135 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.136 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xQnGVMAoouaLAz4qR87BBiu", 2022-09-09T22:59:51.136 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:51.136 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.136 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.137 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.137 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.137 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.138 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.138 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.138 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763988.533014_214.1", 2022-09-09T22:59:51.138 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.139 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-33:4b05608c-7197-4e9b-bd29-7989607fa277.4354.33", 2022-09-09T22:59:51.139 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:08.533014771Z", 2022-09-09T22:59:51.139 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.140 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.140 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xQnGVMAoouaLAz4qR87BBiu", 2022-09-09T22:59:51.140 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.140 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.141 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.141 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xQnGVMAoouaLAz4qR87BBiu", 2022-09-09T22:59:51.141 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:51.142 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.142 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.142 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.142 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.143 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.143 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.143 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.143 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763989.558518_215.1", 2022-09-09T22:59:51.144 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.144 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-34:4b05608c-7197-4e9b-bd29-7989607fa277.4354.34", 2022-09-09T22:59:51.144 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:09.558518140Z", 2022-09-09T22:59:51.144 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.145 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.145 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_4_jQEFJZa-xniCNap3y5x4k", 2022-09-09T22:59:51.145 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.145 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.145 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.146 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_4_jQEFJZa-xniCNap3y5x4k", 2022-09-09T22:59:51.146 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:51.146 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.146 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.147 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.147 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.147 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.147 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.148 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.148 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763989.567553_216.1", 2022-09-09T22:59:51.148 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.149 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-34:4b05608c-7197-4e9b-bd29-7989607fa277.4354.34", 2022-09-09T22:59:51.149 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:09.567553622Z", 2022-09-09T22:59:51.149 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.149 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.150 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_4_jQEFJZa-xniCNap3y5x4k", 2022-09-09T22:59:51.150 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.150 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.150 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.151 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_4_jQEFJZa-xniCNap3y5x4k", 2022-09-09T22:59:51.151 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:51.151 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.151 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.152 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.152 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.152 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.152 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.153 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.153 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763990.604998_217.1", 2022-09-09T22:59:51.153 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.153 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-35:4b05608c-7197-4e9b-bd29-7989607fa277.4354.35", 2022-09-09T22:59:51.154 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:10.604998041Z", 2022-09-09T22:59:51.154 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.154 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.154 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xarcxlcdFlngGuAvxUBGTlW", 2022-09-09T22:59:51.155 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.155 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.155 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.155 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xarcxlcdFlngGuAvxUBGTlW", 2022-09-09T22:59:51.156 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:51.156 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.156 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.156 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.157 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.157 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.157 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.157 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.158 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763990.613292_218.1", 2022-09-09T22:59:51.158 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.158 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-35:4b05608c-7197-4e9b-bd29-7989607fa277.4354.35", 2022-09-09T22:59:51.158 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:10.613292272Z", 2022-09-09T22:59:51.159 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.159 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.159 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xarcxlcdFlngGuAvxUBGTlW", 2022-09-09T22:59:51.159 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.160 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.160 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.160 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xarcxlcdFlngGuAvxUBGTlW", 2022-09-09T22:59:51.160 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 2 2022-09-09T22:59:51.161 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.161 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.161 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.161 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.162 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.162 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.162 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.162 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763990.663150_219.1", 2022-09-09T22:59:51.163 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.163 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36", 2022-09-09T22:59:51.163 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:10.663150053Z", 2022-09-09T22:59:51.163 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.164 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.164 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.164 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.164 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.165 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.165 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_jPN1kAi0u5MPj33UDl-gVII", 2022-09-09T22:59:51.165 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.165 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.166 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.166 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.166 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.166 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.167 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.167 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.167 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763990.667756_220.1", 2022-09-09T22:59:51.167 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.168 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36", 2022-09-09T22:59:51.168 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:10.667756273Z", 2022-09-09T22:59:51.168 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.168 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.169 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.169 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.169 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.169 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.170 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_jPN1kAi0u5MPj33UDl-gVII", 2022-09-09T22:59:51.170 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.170 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.170 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.171 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.171 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.171 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.171 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.172 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.172 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763990.669560_221.1", 2022-09-09T22:59:51.172 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:51.172 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-36", 2022-09-09T22:59:51.173 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:10.669560632Z", 2022-09-09T22:59:51.173 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.173 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.173 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.174 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.174 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.174 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.174 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_gu46bZkwwhtar51KzoH8j_M", 2022-09-09T22:59:51.175 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.175 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.175 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.175 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.176 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.176 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.176 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.176 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.177 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763990.673725_222.1", 2022-09-09T22:59:51.177 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:51.177 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-36", 2022-09-09T22:59:51.177 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:10.673725202Z", 2022-09-09T22:59:51.178 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.178 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.178 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.178 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.179 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.179 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.179 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_gu46bZkwwhtar51KzoH8j_M", 2022-09-09T22:59:51.179 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.180 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.180 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.180 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.180 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.181 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.181 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.181 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.181 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763990.710819_223.1", 2022-09-09T22:59:51.182 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.182 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37", 2022-09-09T22:59:51.182 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:10.710819128Z", 2022-09-09T22:59:51.182 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.183 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.183 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.183 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.183 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.184 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.184 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_FX49dxEb4uM_d2ZifiXBHNo", 2022-09-09T22:59:51.184 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.184 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.185 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.185 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.185 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.185 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.186 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.186 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.186 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763990.715764_224.1", 2022-09-09T22:59:51.186 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.187 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37", 2022-09-09T22:59:51.187 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:10.715764014Z", 2022-09-09T22:59:51.187 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.187 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.188 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.188 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.188 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.188 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.189 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_FX49dxEb4uM_d2ZifiXBHNo", 2022-09-09T22:59:51.189 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.189 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.189 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.190 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.190 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.190 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.190 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.191 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.191 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763990.717500_225.1", 2022-09-09T22:59:51.191 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:51.192 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-37", 2022-09-09T22:59:51.192 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:10.717500107Z", 2022-09-09T22:59:51.192 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.192 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.193 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.193 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.193 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.193 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.194 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_hJzznYY2Hded3epYJ_abHVf", 2022-09-09T22:59:51.194 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.194 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.194 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.195 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.195 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.195 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.196 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.196 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.196 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763990.722195_226.1", 2022-09-09T22:59:51.196 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:51.197 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-37", 2022-09-09T22:59:51.197 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:10.722195904Z", 2022-09-09T22:59:51.197 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.197 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.198 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.198 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.198 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.198 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.199 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_hJzznYY2Hded3epYJ_abHVf", 2022-09-09T22:59:51.199 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.199 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.199 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.200 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.200 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.200 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.200 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.201 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.201 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763990.784611_227.1", 2022-09-09T22:59:51.201 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.201 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38", 2022-09-09T22:59:51.202 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:10.784611285Z", 2022-09-09T22:59:51.202 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.202 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.203 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.203 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.203 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.203 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.203 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_qDxaPPFFwg6ZX6ss6qndeGO", 2022-09-09T22:59:51.204 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.204 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.204 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.205 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.205 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.205 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.205 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.206 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.206 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763990.788678_228.1", 2022-09-09T22:59:51.206 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.206 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38", 2022-09-09T22:59:51.207 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:10.788678416Z", 2022-09-09T22:59:51.207 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.207 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.207 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.208 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.208 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.208 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.208 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_qDxaPPFFwg6ZX6ss6qndeGO", 2022-09-09T22:59:51.209 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.209 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.209 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.209 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.210 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.210 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.210 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.210 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.211 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763990.792062_229.1", 2022-09-09T22:59:51.211 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:51.211 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-38", 2022-09-09T22:59:51.211 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:10.792062116Z", 2022-09-09T22:59:51.212 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.212 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.212 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.212 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.213 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.213 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.213 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_gCNfZZlq_uw7Nsy58KndY3K", 2022-09-09T22:59:51.213 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.214 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.214 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.214 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.214 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.215 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.215 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.215 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.215 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662763990.796002_230.1", 2022-09-09T22:59:51.216 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:51.216 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-38", 2022-09-09T22:59:51.216 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:53:10.796002253Z", 2022-09-09T22:59:51.216 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.217 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.217 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.217 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.217 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.218 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.218 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_gCNfZZlq_uw7Nsy58KndY3K", 2022-09-09T22:59:51.218 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.218 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.219 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.219 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.219 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.219 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.220 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.220 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.220 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764240.029920_231.1", 2022-09-09T22:59:51.220 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.221 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-39:4b05608c-7197-4e9b-bd29-7989607fa277.4354.39", 2022-09-09T22:59:51.221 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:57:20.029920453Z", 2022-09-09T22:59:51.221 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.221 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.222 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.222 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.222 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.222 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.223 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_yHfFFXCLbn_I9-CIE0fRcSE", 2022-09-09T22:59:51.223 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.223 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.223 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.224 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.224 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.224 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.224 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.225 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.225 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764240.034539_232.1", 2022-09-09T22:59:51.225 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.225 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-39:4b05608c-7197-4e9b-bd29-7989607fa277.4354.39", 2022-09-09T22:59:51.226 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:57:20.034539448Z", 2022-09-09T22:59:51.226 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.226 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.226 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.227 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.227 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.227 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.227 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_yHfFFXCLbn_I9-CIE0fRcSE", 2022-09-09T22:59:51.228 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.228 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.228 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.228 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.229 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.229 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.229 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.229 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.230 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764240.036306_233.1", 2022-09-09T22:59:51.230 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:51.230 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-39", 2022-09-09T22:59:51.230 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:57:20.036306500Z", 2022-09-09T22:59:51.231 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.231 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.231 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.231 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.232 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.232 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.232 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_kuQlL_FG34kY8wCt0puQ02d", 2022-09-09T22:59:51.232 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.233 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.233 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.233 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.233 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.234 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.234 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.234 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.235 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764240.040726_234.1", 2022-09-09T22:59:51.235 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:51.235 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-39", 2022-09-09T22:59:51.235 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:57:20.040726112Z", 2022-09-09T22:59:51.236 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.236 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.236 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.236 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.237 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.237 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.237 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_kuQlL_FG34kY8wCt0puQ02d", 2022-09-09T22:59:51.237 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.238 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.238 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.238 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.238 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.239 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.239 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.239 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.239 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764240.078400_235.1", 2022-09-09T22:59:51.240 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.240 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-40:4b05608c-7197-4e9b-bd29-7989607fa277.4354.40", 2022-09-09T22:59:51.240 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:57:20.078400843Z", 2022-09-09T22:59:51.240 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.241 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.241 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.241 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.241 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.242 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.242 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_vfgdN60g8MjxPCCEgs2R_vv", 2022-09-09T22:59:51.242 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.242 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.243 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.243 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.243 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.243 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.244 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.244 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.244 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764240.084545_236.1", 2022-09-09T22:59:51.244 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.245 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-40:4b05608c-7197-4e9b-bd29-7989607fa277.4354.40", 2022-09-09T22:59:51.245 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:57:20.084545560Z", 2022-09-09T22:59:51.245 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.245 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.246 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.246 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.246 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.246 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.247 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_vfgdN60g8MjxPCCEgs2R_vv", 2022-09-09T22:59:51.247 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.247 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.247 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.248 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.248 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.248 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.248 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.249 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.249 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764240.086165_237.1", 2022-09-09T22:59:51.249 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:51.249 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-40", 2022-09-09T22:59:51.250 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:57:20.086165638Z", 2022-09-09T22:59:51.250 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.250 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.250 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.251 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.251 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.251 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.251 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_TJv9-9zJFLq6wQcqNFr3ADa", 2022-09-09T22:59:51.252 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.252 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.252 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.252 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.253 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.253 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.253 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.253 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.254 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764240.090984_238.1", 2022-09-09T22:59:51.254 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:51.254 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-40", 2022-09-09T22:59:51.254 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:57:20.090984173Z", 2022-09-09T22:59:51.255 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.255 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.255 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.255 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.256 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.256 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.256 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_TJv9-9zJFLq6wQcqNFr3ADa", 2022-09-09T22:59:51.256 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.257 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.257 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.257 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.257 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.258 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.258 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.258 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.258 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764240.151909_239.1", 2022-09-09T22:59:51.259 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.259 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-41:4b05608c-7197-4e9b-bd29-7989607fa277.4354.41", 2022-09-09T22:59:51.259 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:57:20.151909503Z", 2022-09-09T22:59:51.259 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.260 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.260 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.260 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.260 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.261 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.261 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_lh1D7Ui8P7X3ZyWeXN_Rbr6", 2022-09-09T22:59:51.261 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.261 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.262 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.262 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.262 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.262 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.263 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.263 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.263 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764240.157917_240.1", 2022-09-09T22:59:51.263 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.264 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-41:4b05608c-7197-4e9b-bd29-7989607fa277.4354.41", 2022-09-09T22:59:51.264 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:57:20.157917182Z", 2022-09-09T22:59:51.264 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.264 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.265 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.265 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.265 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.265 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.266 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_lh1D7Ui8P7X3ZyWeXN_Rbr6", 2022-09-09T22:59:51.266 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.266 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.266 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.267 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.267 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.267 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.267 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.268 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.268 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764240.159693_241.1", 2022-09-09T22:59:51.268 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:51.268 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-41", 2022-09-09T22:59:51.269 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:57:20.159693443Z", 2022-09-09T22:59:51.269 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.269 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.269 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.270 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.270 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.270 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.270 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xxvbuFhSEtP4oKwQrN2eVjC", 2022-09-09T22:59:51.271 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.271 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.271 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.271 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.272 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.272 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.272 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.272 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.273 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764240.168869_243.1", 2022-09-09T22:59:51.273 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:51.273 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-41", 2022-09-09T22:59:51.273 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:57:20.168869275Z", 2022-09-09T22:59:51.274 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.274 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.274 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.274 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.275 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.275 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.275 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_xxvbuFhSEtP4oKwQrN2eVjC", 2022-09-09T22:59:51.275 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.276 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.276 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.276 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.276 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.277 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.277 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.277 INFO:teuthology.orchestra.run.smithi080.stdout:] 2022-09-09T22:59:51.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '2efdc3a1-023b-4e78-aaa2-0e069a034dac', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:51.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '2efdc3a1-023b-4e78-aaa2-0e069a034dac', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:51.280 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 2efdc3a1-023b-4e78-aaa2-0e069a034dac --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:59:51.327 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:51.327 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:51.384 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T22:59:51.385 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:51.385 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764312.475928_2677.1", 2022-09-09T22:59:51.385 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.385 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-42:cc1fc6ed-f06f-45bc-961e-adec328c8651.5441.1", 2022-09-09T22:59:51.386 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:58:32.475928643Z", 2022-09-09T22:59:51.386 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:51.386 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:51.387 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:51.387 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:51.387 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.387 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:51.388 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_7GEIyhMt_0Gavk9BOsX95y0", 2022-09-09T22:59:51.388 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:51.388 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.388 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:51.389 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:51.389 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.389 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.389 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.390 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:51.390 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764312.481280_2678.1", 2022-09-09T22:59:51.390 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.390 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-42:cc1fc6ed-f06f-45bc-961e-adec328c8651.5441.1", 2022-09-09T22:59:51.391 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:58:32.481280696Z", 2022-09-09T22:59:51.391 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:51.391 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:51.391 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:51.392 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:51.392 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.393 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:51.393 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_7GEIyhMt_0Gavk9BOsX95y0", 2022-09-09T22:59:51.393 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:51.394 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.394 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:51.395 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:51.395 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.396 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.396 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.396 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:51.396 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764312.483221_2679.1", 2022-09-09T22:59:51.397 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:51.397 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-42", 2022-09-09T22:59:51.397 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:58:32.483221265Z", 2022-09-09T22:59:51.398 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:51.398 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:51.398 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:51.399 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:51.399 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.399 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:51.399 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_csXbmbjZv-zG2On1SB_1khE", 2022-09-09T22:59:51.400 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:51.400 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.400 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:51.400 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:51.401 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.401 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.401 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.401 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:51.402 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764312.488633_2680.1", 2022-09-09T22:59:51.402 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:51.402 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-42", 2022-09-09T22:59:51.403 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:58:32.488633258Z", 2022-09-09T22:59:51.403 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:51.403 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:51.404 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:51.404 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:51.404 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.405 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:51.405 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_csXbmbjZv-zG2On1SB_1khE", 2022-09-09T22:59:51.405 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:51.406 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.406 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:51.407 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:51.407 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.407 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.407 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.408 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:51.408 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764312.551198_2681.1", 2022-09-09T22:59:51.408 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.409 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-43:cc1fc6ed-f06f-45bc-961e-adec328c8651.5441.2", 2022-09-09T22:59:51.409 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:58:32.551198619Z", 2022-09-09T22:59:51.409 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:51.409 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:51.410 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:51.410 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:51.410 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.411 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:51.411 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "__N0lo786k2N4p_ZN-AJ-YcE", 2022-09-09T22:59:51.412 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:51.412 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.412 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:51.412 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:51.413 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.413 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.413 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.413 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:51.414 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764312.556831_2682.1", 2022-09-09T22:59:51.414 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.414 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-43:cc1fc6ed-f06f-45bc-961e-adec328c8651.5441.2", 2022-09-09T22:59:51.414 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:58:32.556831732Z", 2022-09-09T22:59:51.415 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:51.415 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:51.415 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:51.416 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:51.416 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.416 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:51.417 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "__N0lo786k2N4p_ZN-AJ-YcE", 2022-09-09T22:59:51.417 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:51.417 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.418 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:51.418 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:51.418 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.419 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.419 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.419 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:51.419 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764312.558864_2683.1", 2022-09-09T22:59:51.420 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:51.420 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-43", 2022-09-09T22:59:51.420 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:58:32.558864696Z", 2022-09-09T22:59:51.420 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:51.421 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:51.421 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:51.421 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:51.421 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.422 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:51.422 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_NZG8_L_F2NENzinE8p8YSnA", 2022-09-09T22:59:51.423 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:51.423 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.423 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:51.423 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:51.424 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.424 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.424 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.424 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:51.425 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764312.564852_2684.1", 2022-09-09T22:59:51.425 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:51.425 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-43", 2022-09-09T22:59:51.425 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:58:32.564852554Z", 2022-09-09T22:59:51.426 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:51.426 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:51.426 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:51.426 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:51.427 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.427 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:51.427 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_NZG8_L_F2NENzinE8p8YSnA", 2022-09-09T22:59:51.427 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:51.428 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.428 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:51.430 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:51.431 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.431 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.431 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.432 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T22:59:51.432 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '2efdc3a1-023b-4e78-aaa2-0e069a034dac', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:51.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '2efdc3a1-023b-4e78-aaa2-0e069a034dac', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:51.433 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 2efdc3a1-023b-4e78-aaa2-0e069a034dac --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:59:51.480 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:51.480 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:51.557 INFO:teuthology.orchestra.run.smithi080.stdout:[ 2022-09-09T22:59:51.557 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.558 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764312.475928_2677.1", 2022-09-09T22:59:51.558 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.558 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-42:cc1fc6ed-f06f-45bc-961e-adec328c8651.5441.1", 2022-09-09T22:59:51.558 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:58:32.475928643Z", 2022-09-09T22:59:51.559 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.559 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.559 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.560 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.560 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.560 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.560 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_7GEIyhMt_0Gavk9BOsX95y0", 2022-09-09T22:59:51.561 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.561 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.561 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.561 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.562 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.562 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.562 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.563 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.563 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764312.481280_2678.1", 2022-09-09T22:59:51.563 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.564 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-42:cc1fc6ed-f06f-45bc-961e-adec328c8651.5441.1", 2022-09-09T22:59:51.564 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:58:32.481280696Z", 2022-09-09T22:59:51.564 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.564 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.565 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.565 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.565 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.565 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.566 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_7GEIyhMt_0Gavk9BOsX95y0", 2022-09-09T22:59:51.566 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.566 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.566 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.567 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.567 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.567 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.567 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.568 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.568 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764312.483221_2679.1", 2022-09-09T22:59:51.568 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:51.568 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-42", 2022-09-09T22:59:51.569 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:58:32.483221265Z", 2022-09-09T22:59:51.569 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.569 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.569 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.570 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.570 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.570 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.570 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_csXbmbjZv-zG2On1SB_1khE", 2022-09-09T22:59:51.571 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.571 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.571 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.572 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.572 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.572 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.572 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.573 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.573 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764312.488633_2680.1", 2022-09-09T22:59:51.573 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:51.573 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-42", 2022-09-09T22:59:51.574 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:58:32.488633258Z", 2022-09-09T22:59:51.574 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.574 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.574 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.575 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.575 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.575 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.575 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_csXbmbjZv-zG2On1SB_1khE", 2022-09-09T22:59:51.575 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.576 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.576 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.576 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.576 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.577 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.577 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.577 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.577 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764312.551198_2681.1", 2022-09-09T22:59:51.578 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.578 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-43:cc1fc6ed-f06f-45bc-961e-adec328c8651.5441.2", 2022-09-09T22:59:51.578 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:58:32.551198619Z", 2022-09-09T22:59:51.578 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.579 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.579 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.579 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.579 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.580 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.580 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "__N0lo786k2N4p_ZN-AJ-YcE", 2022-09-09T22:59:51.580 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.580 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.581 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.581 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.581 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.581 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.582 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.582 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.582 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764312.556831_2682.1", 2022-09-09T22:59:51.582 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.583 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-43:cc1fc6ed-f06f-45bc-961e-adec328c8651.5441.2", 2022-09-09T22:59:51.583 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:58:32.556831732Z", 2022-09-09T22:59:51.583 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.583 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.584 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.584 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.584 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.584 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.584 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "__N0lo786k2N4p_ZN-AJ-YcE", 2022-09-09T22:59:51.585 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.585 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.585 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.585 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.586 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.586 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.586 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.586 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.587 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764312.558864_2683.1", 2022-09-09T22:59:51.587 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:51.587 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-43", 2022-09-09T22:59:51.587 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:58:32.558864696Z", 2022-09-09T22:59:51.588 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.588 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.589 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.589 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.589 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.589 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.590 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_NZG8_L_F2NENzinE8p8YSnA", 2022-09-09T22:59:51.590 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.590 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.590 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.591 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.591 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.591 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.591 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.591 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.592 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764312.564852_2684.1", 2022-09-09T22:59:51.592 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:51.592 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-43", 2022-09-09T22:59:51.592 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:58:32.564852554Z", 2022-09-09T22:59:51.593 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.593 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.593 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.593 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.594 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.594 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.594 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_NZG8_L_F2NENzinE8p8YSnA", 2022-09-09T22:59:51.594 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.595 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.595 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.596 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.597 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.597 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.597 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.597 INFO:teuthology.orchestra.run.smithi080.stdout:] 2022-09-09T22:59:51.598 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '2efdc3a1-023b-4e78-aaa2-0e069a034dac', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:51.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '2efdc3a1-023b-4e78-aaa2-0e069a034dac', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:51.599 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 2efdc3a1-023b-4e78-aaa2-0e069a034dac --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:59:51.648 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:51.648 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:51.703 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T22:59:51.704 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:51.704 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764312.475928_2677.1", 2022-09-09T22:59:51.704 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.704 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-42:cc1fc6ed-f06f-45bc-961e-adec328c8651.5441.1", 2022-09-09T22:59:51.705 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:58:32.475928643Z", 2022-09-09T22:59:51.705 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:51.705 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:51.705 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:51.706 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:51.706 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.706 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:51.706 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_7GEIyhMt_0Gavk9BOsX95y0", 2022-09-09T22:59:51.707 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:51.707 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.707 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:51.708 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:51.708 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.708 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.709 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.709 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:51.709 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764312.481280_2678.1", 2022-09-09T22:59:51.710 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.710 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-42:cc1fc6ed-f06f-45bc-961e-adec328c8651.5441.1", 2022-09-09T22:59:51.710 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:58:32.481280696Z", 2022-09-09T22:59:51.711 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:51.711 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:51.711 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:51.712 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:51.712 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.712 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:51.712 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_7GEIyhMt_0Gavk9BOsX95y0", 2022-09-09T22:59:51.713 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:51.713 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.713 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:51.713 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:51.713 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.714 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.714 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.714 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:51.714 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764312.483221_2679.1", 2022-09-09T22:59:51.715 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:51.715 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-42", 2022-09-09T22:59:51.715 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:58:32.483221265Z", 2022-09-09T22:59:51.715 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:51.716 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:51.716 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:51.716 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:51.717 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.717 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:51.717 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_csXbmbjZv-zG2On1SB_1khE", 2022-09-09T22:59:51.717 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:51.718 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.718 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:51.719 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:51.719 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.720 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.720 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.720 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:51.720 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764312.488633_2680.1", 2022-09-09T22:59:51.721 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:51.721 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-42", 2022-09-09T22:59:51.721 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:58:32.488633258Z", 2022-09-09T22:59:51.722 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:51.722 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:51.722 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:51.722 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:51.723 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.723 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:51.723 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_csXbmbjZv-zG2On1SB_1khE", 2022-09-09T22:59:51.723 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:51.724 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.724 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:51.724 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:51.724 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.725 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.725 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.725 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:51.725 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764312.551198_2681.1", 2022-09-09T22:59:51.726 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.726 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-43:cc1fc6ed-f06f-45bc-961e-adec328c8651.5441.2", 2022-09-09T22:59:51.726 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:58:32.551198619Z", 2022-09-09T22:59:51.726 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:51.726 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:51.727 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:51.727 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:51.727 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.727 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:51.728 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "__N0lo786k2N4p_ZN-AJ-YcE", 2022-09-09T22:59:51.728 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:51.728 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.729 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:51.729 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:51.729 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.730 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.730 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.730 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:51.730 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764312.556831_2682.1", 2022-09-09T22:59:51.731 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.731 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-43:cc1fc6ed-f06f-45bc-961e-adec328c8651.5441.2", 2022-09-09T22:59:51.731 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:58:32.556831732Z", 2022-09-09T22:59:51.731 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:51.732 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:51.732 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:51.732 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:51.732 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.733 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:51.733 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "__N0lo786k2N4p_ZN-AJ-YcE", 2022-09-09T22:59:51.733 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:51.733 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.734 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:51.734 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:51.734 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.734 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.735 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.735 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:51.735 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764312.558864_2683.1", 2022-09-09T22:59:51.735 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:51.736 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-43", 2022-09-09T22:59:51.736 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:58:32.558864696Z", 2022-09-09T22:59:51.736 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:51.737 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:51.737 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:51.737 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:51.737 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.738 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:51.738 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_NZG8_L_F2NENzinE8p8YSnA", 2022-09-09T22:59:51.738 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:51.739 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.739 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:51.739 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:51.740 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.740 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.740 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.740 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:51.741 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764312.564852_2684.1", 2022-09-09T22:59:51.741 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:51.741 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-43", 2022-09-09T22:59:51.741 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:58:32.564852554Z", 2022-09-09T22:59:51.742 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:51.742 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:51.742 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:51.742 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:51.743 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.743 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:51.743 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_NZG8_L_F2NENzinE8p8YSnA", 2022-09-09T22:59:51.743 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:51.744 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:51.744 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:51.745 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:51.745 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.746 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.746 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:51.746 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T22:59:51.747 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '2efdc3a1-023b-4e78-aaa2-0e069a034dac', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:51.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '2efdc3a1-023b-4e78-aaa2-0e069a034dac', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:51.748 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 2efdc3a1-023b-4e78-aaa2-0e069a034dac --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:59:51.794 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:51.795 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:51.863 INFO:teuthology.orchestra.run.smithi080.stdout:[ 2022-09-09T22:59:51.863 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.863 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764312.475928_2677.1", 2022-09-09T22:59:51.864 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.864 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-42:cc1fc6ed-f06f-45bc-961e-adec328c8651.5441.1", 2022-09-09T22:59:51.864 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:58:32.475928643Z", 2022-09-09T22:59:51.864 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.865 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.865 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.865 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.866 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.866 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.866 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_7GEIyhMt_0Gavk9BOsX95y0", 2022-09-09T22:59:51.867 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.867 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.867 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.867 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.868 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.868 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.868 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.868 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.869 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764312.481280_2678.1", 2022-09-09T22:59:51.869 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.869 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-42:cc1fc6ed-f06f-45bc-961e-adec328c8651.5441.1", 2022-09-09T22:59:51.870 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:58:32.481280696Z", 2022-09-09T22:59:51.870 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.870 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.871 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.871 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.871 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.871 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.872 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_7GEIyhMt_0Gavk9BOsX95y0", 2022-09-09T22:59:51.872 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.872 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.872 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.873 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.873 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.873 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.874 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.874 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.874 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764312.483221_2679.1", 2022-09-09T22:59:51.874 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:51.875 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-42", 2022-09-09T22:59:51.875 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:58:32.483221265Z", 2022-09-09T22:59:51.875 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.876 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.876 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.876 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.876 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.877 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.877 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_csXbmbjZv-zG2On1SB_1khE", 2022-09-09T22:59:51.877 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.877 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.878 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.878 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.878 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.878 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.879 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.879 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.879 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764312.488633_2680.1", 2022-09-09T22:59:51.879 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:51.880 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-42", 2022-09-09T22:59:51.880 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:58:32.488633258Z", 2022-09-09T22:59:51.880 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.880 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.881 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.881 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.881 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.881 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.882 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_csXbmbjZv-zG2On1SB_1khE", 2022-09-09T22:59:51.882 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.882 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.882 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.883 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.883 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.883 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.883 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.884 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.884 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764312.551198_2681.1", 2022-09-09T22:59:51.884 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.884 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-43:cc1fc6ed-f06f-45bc-961e-adec328c8651.5441.2", 2022-09-09T22:59:51.885 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:58:32.551198619Z", 2022-09-09T22:59:51.885 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.885 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.885 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.886 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.886 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.886 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.886 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "__N0lo786k2N4p_ZN-AJ-YcE", 2022-09-09T22:59:51.887 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.887 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.887 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.888 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.888 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.888 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.888 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.888 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.889 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764312.556831_2682.1", 2022-09-09T22:59:51.889 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:51.889 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-43:cc1fc6ed-f06f-45bc-961e-adec328c8651.5441.2", 2022-09-09T22:59:51.890 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:58:32.556831732Z", 2022-09-09T22:59:51.890 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.890 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.890 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.891 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.891 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.891 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.891 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "__N0lo786k2N4p_ZN-AJ-YcE", 2022-09-09T22:59:51.892 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.892 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.892 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.892 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.893 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.893 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.893 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.893 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.894 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764312.558864_2683.1", 2022-09-09T22:59:51.894 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:51.894 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-43", 2022-09-09T22:59:51.894 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:58:32.558864696Z", 2022-09-09T22:59:51.895 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.895 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.895 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.895 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.896 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.896 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.896 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_NZG8_L_F2NENzinE8p8YSnA", 2022-09-09T22:59:51.896 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.897 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.897 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.897 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:51.897 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.898 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.898 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.898 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:51.898 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764312.564852_2684.1", 2022-09-09T22:59:51.899 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:51.899 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-43", 2022-09-09T22:59:51.899 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:58:32.564852554Z", 2022-09-09T22:59:51.899 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:51.900 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:51.900 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:51.900 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:51.900 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.901 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:51.901 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_NZG8_L_F2NENzinE8p8YSnA", 2022-09-09T22:59:51.901 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:51.901 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:51.902 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:51.903 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:51.903 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.904 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.904 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:51.904 INFO:teuthology.orchestra.run.smithi080.stdout:] 2022-09-09T22:59:51.905 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'fd4bc158-06fd-4840-8a27-33fabda67b99', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:51.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'fd4bc158-06fd-4840-8a27-33fabda67b99', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:51.906 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period fd4bc158-06fd-4840-8a27-33fabda67b99 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:59:51.954 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:51.955 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:52.011 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T22:59:52.012 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:52.012 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764373.312961_353.1", 2022-09-09T22:59:52.012 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:52.012 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-44:4b05608c-7197-4e9b-bd29-7989607fa277.5119.1", 2022-09-09T22:59:52.013 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:59:33.312961736Z", 2022-09-09T22:59:52.013 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:52.013 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:52.014 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:52.014 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:52.014 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.014 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:52.015 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_SdQ0ofLGohecZhrwUlqU7Uf", 2022-09-09T22:59:52.015 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:52.015 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.015 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:52.016 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:52.016 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.016 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.017 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.017 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:52.017 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764373.317673_354.1", 2022-09-09T22:59:52.018 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:52.018 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-44:4b05608c-7197-4e9b-bd29-7989607fa277.5119.1", 2022-09-09T22:59:52.018 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:59:33.317673000Z", 2022-09-09T22:59:52.019 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:52.019 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:52.019 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:52.019 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:52.020 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.020 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:52.020 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_SdQ0ofLGohecZhrwUlqU7Uf", 2022-09-09T22:59:52.020 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:52.021 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.021 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:52.022 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:52.022 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.022 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.022 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.023 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:52.023 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764373.319365_355.1", 2022-09-09T22:59:52.023 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:52.023 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-44", 2022-09-09T22:59:52.024 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:59:33.319365444Z", 2022-09-09T22:59:52.024 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:52.024 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:52.024 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:52.025 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:52.025 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.025 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:52.025 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_3B_bJqMjcngjF6K-YOD8j1_", 2022-09-09T22:59:52.026 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:52.026 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.026 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:52.026 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:52.027 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.027 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.027 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.027 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:52.028 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764373.323396_356.1", 2022-09-09T22:59:52.028 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:52.028 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-44", 2022-09-09T22:59:52.029 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:59:33.323396896Z", 2022-09-09T22:59:52.029 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:52.029 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:52.029 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:52.030 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:52.030 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.030 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:52.030 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_3B_bJqMjcngjF6K-YOD8j1_", 2022-09-09T22:59:52.031 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:52.031 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.031 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:52.032 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:52.032 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.032 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.033 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.033 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:52.033 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764373.365027_357.1", 2022-09-09T22:59:52.033 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:52.034 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-45:4b05608c-7197-4e9b-bd29-7989607fa277.5119.2", 2022-09-09T22:59:52.034 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:59:33.365027182Z", 2022-09-09T22:59:52.034 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:52.035 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:52.035 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:52.035 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:52.035 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.036 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:52.036 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_RI6BWlEPVe9ZvHaNzeMnR27", 2022-09-09T22:59:52.036 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:52.037 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.037 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:52.037 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:52.038 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.038 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.038 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.039 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:52.039 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764373.371107_358.1", 2022-09-09T22:59:52.039 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:52.039 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-45:4b05608c-7197-4e9b-bd29-7989607fa277.5119.2", 2022-09-09T22:59:52.040 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:59:33.371107006Z", 2022-09-09T22:59:52.040 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:52.040 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:52.040 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:52.041 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:52.041 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.041 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:52.042 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_RI6BWlEPVe9ZvHaNzeMnR27", 2022-09-09T22:59:52.042 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:52.042 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.042 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:52.043 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:52.043 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.043 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.043 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.044 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:52.044 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764373.372810_359.1", 2022-09-09T22:59:52.044 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:52.044 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-45", 2022-09-09T22:59:52.045 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:59:33.372810657Z", 2022-09-09T22:59:52.045 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:52.045 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:52.046 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:52.046 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:52.046 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.046 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:52.047 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_LPIzPm1LLXtT87shzO7qIYQ", 2022-09-09T22:59:52.047 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:52.047 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.047 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:52.048 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:52.048 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.048 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.048 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.049 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:52.049 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764373.377065_360.1", 2022-09-09T22:59:52.049 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:52.049 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-45", 2022-09-09T22:59:52.050 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:59:33.377065599Z", 2022-09-09T22:59:52.050 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:52.051 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:52.051 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:52.051 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:52.051 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.052 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:52.052 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_LPIzPm1LLXtT87shzO7qIYQ", 2022-09-09T22:59:52.052 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:52.052 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.053 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:52.054 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:52.054 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.055 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.055 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.055 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T22:59:52.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'fd4bc158-06fd-4840-8a27-33fabda67b99', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:52.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'fd4bc158-06fd-4840-8a27-33fabda67b99', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:52.057 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period fd4bc158-06fd-4840-8a27-33fabda67b99 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:59:52.102 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:52.102 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:52.168 INFO:teuthology.orchestra.run.smithi080.stdout:[ 2022-09-09T22:59:52.169 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:52.169 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764373.312961_353.1", 2022-09-09T22:59:52.169 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:52.170 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-44:4b05608c-7197-4e9b-bd29-7989607fa277.5119.1", 2022-09-09T22:59:52.170 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:33.312961736Z", 2022-09-09T22:59:52.170 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:52.170 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:52.171 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:52.171 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:52.171 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.171 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:52.172 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_SdQ0ofLGohecZhrwUlqU7Uf", 2022-09-09T22:59:52.172 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:52.172 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.172 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:52.173 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:52.173 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.173 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.173 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.174 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:52.174 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764373.317673_354.1", 2022-09-09T22:59:52.174 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:52.175 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-44:4b05608c-7197-4e9b-bd29-7989607fa277.5119.1", 2022-09-09T22:59:52.175 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:33.317673000Z", 2022-09-09T22:59:52.175 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:52.175 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:52.176 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:52.176 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:52.176 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.176 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:52.177 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_SdQ0ofLGohecZhrwUlqU7Uf", 2022-09-09T22:59:52.177 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:52.177 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.177 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:52.178 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:52.178 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.178 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.179 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.179 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:52.179 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764373.319365_355.1", 2022-09-09T22:59:52.180 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:52.180 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-44", 2022-09-09T22:59:52.180 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:33.319365444Z", 2022-09-09T22:59:52.180 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:52.181 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:52.181 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:52.181 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:52.181 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.182 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:52.182 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_3B_bJqMjcngjF6K-YOD8j1_", 2022-09-09T22:59:52.182 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:52.182 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.183 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:52.183 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:52.183 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.183 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.184 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.184 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:52.184 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764373.323396_356.1", 2022-09-09T22:59:52.184 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:52.185 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-44", 2022-09-09T22:59:52.185 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:33.323396896Z", 2022-09-09T22:59:52.185 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:52.185 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:52.186 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:52.186 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:52.186 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.187 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:52.187 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_3B_bJqMjcngjF6K-YOD8j1_", 2022-09-09T22:59:52.187 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:52.187 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.188 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:52.188 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:52.188 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.189 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.189 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.190 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:52.190 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764373.365027_357.1", 2022-09-09T22:59:52.190 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:52.190 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-45:4b05608c-7197-4e9b-bd29-7989607fa277.5119.2", 2022-09-09T22:59:52.191 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:33.365027182Z", 2022-09-09T22:59:52.191 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:52.191 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:52.191 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:52.192 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:52.192 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.192 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:52.192 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_RI6BWlEPVe9ZvHaNzeMnR27", 2022-09-09T22:59:52.193 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:52.193 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.193 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:52.193 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:52.194 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.194 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.195 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.195 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:52.196 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764373.371107_358.1", 2022-09-09T22:59:52.196 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:52.197 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-45:4b05608c-7197-4e9b-bd29-7989607fa277.5119.2", 2022-09-09T22:59:52.197 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:33.371107006Z", 2022-09-09T22:59:52.198 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:52.198 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:52.199 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:52.199 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:52.200 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.200 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:52.201 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_RI6BWlEPVe9ZvHaNzeMnR27", 2022-09-09T22:59:52.201 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:52.202 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.202 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:52.203 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:52.203 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.204 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.204 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.204 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:52.205 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764373.372810_359.1", 2022-09-09T22:59:52.205 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:52.206 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-45", 2022-09-09T22:59:52.206 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:33.372810657Z", 2022-09-09T22:59:52.206 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:52.207 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:52.207 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:52.207 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:52.207 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.208 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:52.208 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_LPIzPm1LLXtT87shzO7qIYQ", 2022-09-09T22:59:52.208 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:52.208 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.209 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:52.209 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:52.209 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.210 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.210 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.210 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:52.210 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764373.377065_360.1", 2022-09-09T22:59:52.211 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:52.211 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-45", 2022-09-09T22:59:52.211 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:33.377065599Z", 2022-09-09T22:59:52.211 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:52.212 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:52.212 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:52.212 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:52.213 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.213 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:52.213 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_LPIzPm1LLXtT87shzO7qIYQ", 2022-09-09T22:59:52.213 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:52.214 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.214 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:52.216 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:52.216 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.216 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.217 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.217 INFO:teuthology.orchestra.run.smithi080.stdout:] 2022-09-09T22:59:52.217 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'fd4bc158-06fd-4840-8a27-33fabda67b99', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:52.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'fd4bc158-06fd-4840-8a27-33fabda67b99', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:52.219 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period fd4bc158-06fd-4840-8a27-33fabda67b99 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:59:52.263 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:52.264 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:52.321 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T22:59:52.322 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:52.322 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764373.312961_353.1", 2022-09-09T22:59:52.322 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:52.323 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-44:4b05608c-7197-4e9b-bd29-7989607fa277.5119.1", 2022-09-09T22:59:52.323 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:59:33.312961736Z", 2022-09-09T22:59:52.323 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:52.323 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:52.324 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:52.324 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:52.324 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.324 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:52.325 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_SdQ0ofLGohecZhrwUlqU7Uf", 2022-09-09T22:59:52.325 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:52.325 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.325 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:52.326 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:52.326 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.326 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.327 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.327 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:52.327 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764373.317673_354.1", 2022-09-09T22:59:52.328 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:52.329 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-44:4b05608c-7197-4e9b-bd29-7989607fa277.5119.1", 2022-09-09T22:59:52.329 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:59:33.317673000Z", 2022-09-09T22:59:52.329 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:52.329 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:52.330 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:52.330 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:52.330 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.331 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:52.331 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_SdQ0ofLGohecZhrwUlqU7Uf", 2022-09-09T22:59:52.331 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:52.331 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.332 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:52.332 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:52.332 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.332 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.333 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.333 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:52.333 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764373.319365_355.1", 2022-09-09T22:59:52.334 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:52.334 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-44", 2022-09-09T22:59:52.334 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:59:33.319365444Z", 2022-09-09T22:59:52.334 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:52.335 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:52.335 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:52.335 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:52.336 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.336 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:52.336 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_3B_bJqMjcngjF6K-YOD8j1_", 2022-09-09T22:59:52.337 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:52.337 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.338 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:52.338 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:52.338 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.339 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.339 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.339 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:52.339 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764373.323396_356.1", 2022-09-09T22:59:52.340 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:52.340 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-44", 2022-09-09T22:59:52.340 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:59:33.323396896Z", 2022-09-09T22:59:52.341 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:52.341 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:52.341 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:52.341 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:52.342 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.342 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:52.342 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_3B_bJqMjcngjF6K-YOD8j1_", 2022-09-09T22:59:52.343 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:52.343 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.343 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:52.343 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:52.344 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.344 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.344 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.345 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:52.345 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764373.365027_357.1", 2022-09-09T22:59:52.345 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:52.346 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-45:4b05608c-7197-4e9b-bd29-7989607fa277.5119.2", 2022-09-09T22:59:52.346 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:59:33.365027182Z", 2022-09-09T22:59:52.346 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:52.347 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:52.347 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:52.347 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:52.347 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.348 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:52.348 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_RI6BWlEPVe9ZvHaNzeMnR27", 2022-09-09T22:59:52.349 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:52.349 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.349 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:52.350 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:52.350 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.350 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.350 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.351 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:52.351 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764373.371107_358.1", 2022-09-09T22:59:52.351 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket.instance", 2022-09-09T22:59:52.351 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-45:4b05608c-7197-4e9b-bd29-7989607fa277.5119.2", 2022-09-09T22:59:52.352 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:59:33.371107006Z", 2022-09-09T22:59:52.352 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:52.352 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:52.352 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:52.353 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:52.353 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.353 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:52.354 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_RI6BWlEPVe9ZvHaNzeMnR27", 2022-09-09T22:59:52.354 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:52.354 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.355 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:52.355 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:52.356 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.356 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.356 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.356 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:52.357 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764373.372810_359.1", 2022-09-09T22:59:52.357 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:52.357 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-45", 2022-09-09T22:59:52.357 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:59:33.372810657Z", 2022-09-09T22:59:52.358 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:52.358 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:52.358 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:52.358 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:52.359 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.359 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:52.360 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_LPIzPm1LLXtT87shzO7qIYQ", 2022-09-09T22:59:52.360 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:52.360 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.360 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:52.361 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "write" 2022-09-09T22:59:52.361 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.361 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.361 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.362 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:52.362 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "1_1662764373.377065_360.1", 2022-09-09T22:59:52.362 INFO:teuthology.orchestra.run.smithi005.stdout: "section": "bucket", 2022-09-09T22:59:52.362 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "jkcgbj-45", 2022-09-09T22:59:52.363 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:59:33.377065599Z", 2022-09-09T22:59:52.363 INFO:teuthology.orchestra.run.smithi005.stdout: "data": { 2022-09-09T22:59:52.363 INFO:teuthology.orchestra.run.smithi005.stdout: "read_version": { 2022-09-09T22:59:52.364 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "", 2022-09-09T22:59:52.364 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 0 2022-09-09T22:59:52.364 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.365 INFO:teuthology.orchestra.run.smithi005.stdout: "write_version": { 2022-09-09T22:59:52.366 INFO:teuthology.orchestra.run.smithi005.stdout: "tag": "_LPIzPm1LLXtT87shzO7qIYQ", 2022-09-09T22:59:52.366 INFO:teuthology.orchestra.run.smithi005.stdout: "ver": 1 2022-09-09T22:59:52.366 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:52.366 INFO:teuthology.orchestra.run.smithi005.stdout: "status": { 2022-09-09T22:59:52.368 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "complete" 2022-09-09T22:59:52.368 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.368 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.369 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:52.369 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T22:59:52.369 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'fd4bc158-06fd-4840-8a27-33fabda67b99', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:52.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'fd4bc158-06fd-4840-8a27-33fabda67b99', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:52.371 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period fd4bc158-06fd-4840-8a27-33fabda67b99 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:59:52.417 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:52.418 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:52.489 INFO:teuthology.orchestra.run.smithi080.stdout:[ 2022-09-09T22:59:52.489 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:52.490 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764373.312961_353.1", 2022-09-09T22:59:52.490 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:52.490 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-44:4b05608c-7197-4e9b-bd29-7989607fa277.5119.1", 2022-09-09T22:59:52.491 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:33.312961736Z", 2022-09-09T22:59:52.491 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:52.491 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:52.492 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:52.493 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:52.493 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.494 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:52.494 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_SdQ0ofLGohecZhrwUlqU7Uf", 2022-09-09T22:59:52.495 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:52.495 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.496 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:52.496 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:52.497 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.497 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.498 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.498 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:52.499 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764373.317673_354.1", 2022-09-09T22:59:52.500 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:52.501 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-44:4b05608c-7197-4e9b-bd29-7989607fa277.5119.1", 2022-09-09T22:59:52.501 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:33.317673000Z", 2022-09-09T22:59:52.502 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:52.502 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:52.503 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:52.503 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:52.504 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.504 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:52.504 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_SdQ0ofLGohecZhrwUlqU7Uf", 2022-09-09T22:59:52.505 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:52.505 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.506 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:52.506 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:52.507 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.507 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.508 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.508 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:52.509 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764373.319365_355.1", 2022-09-09T22:59:52.509 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:52.510 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-44", 2022-09-09T22:59:52.510 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:33.319365444Z", 2022-09-09T22:59:52.511 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:52.511 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:52.512 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:52.512 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:52.513 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.513 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:52.514 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_3B_bJqMjcngjF6K-YOD8j1_", 2022-09-09T22:59:52.514 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:52.515 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.515 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:52.516 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:52.516 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.516 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.517 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.517 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:52.517 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764373.323396_356.1", 2022-09-09T22:59:52.517 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:52.518 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-44", 2022-09-09T22:59:52.518 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:33.323396896Z", 2022-09-09T22:59:52.518 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:52.518 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:52.519 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:52.519 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:52.519 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.519 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:52.520 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_3B_bJqMjcngjF6K-YOD8j1_", 2022-09-09T22:59:52.520 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:52.520 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.521 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:52.521 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:52.521 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.521 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.522 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.522 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:52.522 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764373.365027_357.1", 2022-09-09T22:59:52.522 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:52.523 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-45:4b05608c-7197-4e9b-bd29-7989607fa277.5119.2", 2022-09-09T22:59:52.523 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:33.365027182Z", 2022-09-09T22:59:52.523 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:52.523 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:52.524 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:52.524 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:52.524 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.525 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:52.525 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_RI6BWlEPVe9ZvHaNzeMnR27", 2022-09-09T22:59:52.525 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:52.525 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.525 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:52.526 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:52.526 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.526 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.526 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.527 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:52.527 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764373.371107_358.1", 2022-09-09T22:59:52.527 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket.instance", 2022-09-09T22:59:52.527 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-45:4b05608c-7197-4e9b-bd29-7989607fa277.5119.2", 2022-09-09T22:59:52.528 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:33.371107006Z", 2022-09-09T22:59:52.528 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:52.528 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:52.529 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:52.529 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:52.529 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.530 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:52.530 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_RI6BWlEPVe9ZvHaNzeMnR27", 2022-09-09T22:59:52.530 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:52.531 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.531 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:52.531 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:52.531 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.532 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.532 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.532 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:52.532 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764373.372810_359.1", 2022-09-09T22:59:52.533 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:52.533 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-45", 2022-09-09T22:59:52.533 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:33.372810657Z", 2022-09-09T22:59:52.533 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:52.533 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:52.534 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:52.534 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:52.534 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.535 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:52.535 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_LPIzPm1LLXtT87shzO7qIYQ", 2022-09-09T22:59:52.535 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:52.536 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.536 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:52.536 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "write" 2022-09-09T22:59:52.536 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.537 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.537 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.537 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:52.537 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "1_1662764373.377065_360.1", 2022-09-09T22:59:52.538 INFO:teuthology.orchestra.run.smithi080.stdout: "section": "bucket", 2022-09-09T22:59:52.538 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "jkcgbj-45", 2022-09-09T22:59:52.538 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:33.377065599Z", 2022-09-09T22:59:52.539 INFO:teuthology.orchestra.run.smithi080.stdout: "data": { 2022-09-09T22:59:52.539 INFO:teuthology.orchestra.run.smithi080.stdout: "read_version": { 2022-09-09T22:59:52.539 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "", 2022-09-09T22:59:52.539 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 0 2022-09-09T22:59:52.540 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.540 INFO:teuthology.orchestra.run.smithi080.stdout: "write_version": { 2022-09-09T22:59:52.540 INFO:teuthology.orchestra.run.smithi080.stdout: "tag": "_LPIzPm1LLXtT87shzO7qIYQ", 2022-09-09T22:59:52.541 INFO:teuthology.orchestra.run.smithi080.stdout: "ver": 1 2022-09-09T22:59:52.541 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:52.541 INFO:teuthology.orchestra.run.smithi080.stdout: "status": { 2022-09-09T22:59:52.542 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "complete" 2022-09-09T22:59:52.543 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.543 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.543 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:52.544 INFO:teuthology.orchestra.run.smithi080.stdout:] 2022-09-09T22:59:52.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:59:52.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:59:52.545 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-09T22:59:52.589 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:52.590 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:52.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:59:52.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:59:52.715 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-09T22:59:52.757 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:52.758 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:52.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:59:52.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:59:52.909 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-09T22:59:52.952 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:52.953 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:53.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:59:53.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T22:59:53.051 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-09T22:59:53.093 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:53.093 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:53.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '503b1d61-4513-48c8-a308-17b76219610c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:53.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '503b1d61-4513-48c8-a308-17b76219610c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:53.208 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 503b1d61-4513-48c8-a308-17b76219610c --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:59:53.254 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:53.254 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:53.338 INFO:teuthology.orchestra.run.smithi005.stdout:[] 2022-09-09T22:59:53.341 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '503b1d61-4513-48c8-a308-17b76219610c', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:53.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '503b1d61-4513-48c8-a308-17b76219610c', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:53.343 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 503b1d61-4513-48c8-a308-17b76219610c --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:59:53.388 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:53.389 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:53.486 INFO:teuthology.orchestra.run.smithi080.stdout:[] 2022-09-09T22:59:53.488 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '503b1d61-4513-48c8-a308-17b76219610c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:53.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '503b1d61-4513-48c8-a308-17b76219610c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:53.490 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 503b1d61-4513-48c8-a308-17b76219610c --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:59:53.545 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:53.545 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:53.611 INFO:teuthology.orchestra.run.smithi005.stdout:[] 2022-09-09T22:59:53.614 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '503b1d61-4513-48c8-a308-17b76219610c', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:53.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '503b1d61-4513-48c8-a308-17b76219610c', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:53.615 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 503b1d61-4513-48c8-a308-17b76219610c --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:59:53.663 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:53.663 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:53.737 INFO:teuthology.orchestra.run.smithi080.stdout:[] 2022-09-09T22:59:53.739 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '2efdc3a1-023b-4e78-aaa2-0e069a034dac', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:53.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '2efdc3a1-023b-4e78-aaa2-0e069a034dac', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:53.741 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 2efdc3a1-023b-4e78-aaa2-0e069a034dac --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:59:53.782 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:53.782 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:53.860 INFO:teuthology.orchestra.run.smithi005.stdout:[] 2022-09-09T22:59:53.863 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '2efdc3a1-023b-4e78-aaa2-0e069a034dac', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:53.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '2efdc3a1-023b-4e78-aaa2-0e069a034dac', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:53.865 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 2efdc3a1-023b-4e78-aaa2-0e069a034dac --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:59:53.910 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:53.911 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:54.010 INFO:teuthology.orchestra.run.smithi080.stdout:[] 2022-09-09T22:59:54.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '2efdc3a1-023b-4e78-aaa2-0e069a034dac', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:54.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '2efdc3a1-023b-4e78-aaa2-0e069a034dac', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:54.013 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 2efdc3a1-023b-4e78-aaa2-0e069a034dac --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:59:54.060 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:54.060 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:54.148 INFO:teuthology.orchestra.run.smithi005.stdout:[] 2022-09-09T22:59:54.148 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '2efdc3a1-023b-4e78-aaa2-0e069a034dac', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:54.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '2efdc3a1-023b-4e78-aaa2-0e069a034dac', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:54.150 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 2efdc3a1-023b-4e78-aaa2-0e069a034dac --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:59:54.198 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:54.199 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:54.295 INFO:teuthology.orchestra.run.smithi080.stdout:[] 2022-09-09T22:59:54.298 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'fd4bc158-06fd-4840-8a27-33fabda67b99', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:54.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'fd4bc158-06fd-4840-8a27-33fabda67b99', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:54.300 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period fd4bc158-06fd-4840-8a27-33fabda67b99 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:59:54.349 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:54.350 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:54.433 INFO:teuthology.orchestra.run.smithi005.stdout:[] 2022-09-09T22:59:54.437 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'fd4bc158-06fd-4840-8a27-33fabda67b99', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:54.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'fd4bc158-06fd-4840-8a27-33fabda67b99', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:54.438 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period fd4bc158-06fd-4840-8a27-33fabda67b99 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:59:54.485 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:54.485 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:54.567 INFO:teuthology.orchestra.run.smithi080.stdout:[] 2022-09-09T22:59:54.571 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'fd4bc158-06fd-4840-8a27-33fabda67b99', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:54.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'fd4bc158-06fd-4840-8a27-33fabda67b99', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:54.572 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period fd4bc158-06fd-4840-8a27-33fabda67b99 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:59:54.619 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:54.620 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:54.700 INFO:teuthology.orchestra.run.smithi005.stdout:[] 2022-09-09T22:59:54.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'fd4bc158-06fd-4840-8a27-33fabda67b99', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:54.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'fd4bc158-06fd-4840-8a27-33fabda67b99', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:54.706 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period fd4bc158-06fd-4840-8a27-33fabda67b99 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:59:54.748 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:54.749 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:54.837 INFO:teuthology.orchestra.run.smithi080.stdout:[] 2022-09-09T22:59:54.840 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... ok 2022-09-09T22:59:54.843 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jkcgbj-46 2022-09-09T22:59:54.844 DEBUG:boto:path=/jkcgbj-46/ 2022-09-09T22:59:54.845 DEBUG:boto:auth_path=/jkcgbj-46/ 2022-09-09T22:59:54.845 DEBUG:boto:Method: PUT 2022-09-09T22:59:54.846 DEBUG:boto:Path: /jkcgbj-46/ 2022-09-09T22:59:54.847 DEBUG:boto:Data: 2022-09-09T22:59:54.848 DEBUG:boto:Headers: {} 2022-09-09T22:59:54.848 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:54.849 DEBUG:boto:Port: 8000 2022-09-09T22:59:54.849 DEBUG:boto:Params: {} 2022-09-09T22:59:54.850 DEBUG:boto:Token: None 2022-09-09T22:59:54.851 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:59:54 GMT /jkcgbj-46/ 2022-09-09T22:59:54.852 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OSxLw4UdOtgzOQiTrVbL8xZi96k= 2022-09-09T22:59:54.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:54 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OSxLw4UdOtgzOQiTrVbL8xZi96k=', 'Content-Length': '0'} 2022-09-09T22:59:54.885 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005b5fceacf075a954-00631bc56a-13fd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:54.885 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jkcgbj-47 2022-09-09T22:59:54.886 DEBUG:boto:path=/jkcgbj-47/ 2022-09-09T22:59:54.887 DEBUG:boto:auth_path=/jkcgbj-47/ 2022-09-09T22:59:54.887 DEBUG:boto:Method: PUT 2022-09-09T22:59:54.888 DEBUG:boto:Path: /jkcgbj-47/ 2022-09-09T22:59:54.888 DEBUG:boto:Data: 2022-09-09T22:59:54.889 DEBUG:boto:Headers: {} 2022-09-09T22:59:54.889 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:54.890 DEBUG:boto:Port: 8000 2022-09-09T22:59:54.890 DEBUG:boto:Params: {} 2022-09-09T22:59:54.891 DEBUG:boto:Token: None 2022-09-09T22:59:54.891 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:59:54 GMT /jkcgbj-47/ 2022-09-09T22:59:54.892 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:nbRg4gayR0M9s46eVM/COii3jCg= 2022-09-09T22:59:54.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:54 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:nbRg4gayR0M9s46eVM/COii3jCg=', 'Content-Length': '0'} 2022-09-09T22:59:54.951 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000450d2066ad611e8d-00631bc56a-1549-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:54.952 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jkcgbj-48 2022-09-09T22:59:54.953 DEBUG:boto:path=/jkcgbj-48/ 2022-09-09T22:59:54.954 DEBUG:boto:auth_path=/jkcgbj-48/ 2022-09-09T22:59:54.955 DEBUG:boto:Method: PUT 2022-09-09T22:59:54.956 DEBUG:boto:Path: /jkcgbj-48/ 2022-09-09T22:59:54.957 DEBUG:boto:Data: 2022-09-09T22:59:54.958 DEBUG:boto:Headers: {} 2022-09-09T22:59:54.958 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:54.959 DEBUG:boto:Port: 8001 2022-09-09T22:59:54.960 DEBUG:boto:Params: {} 2022-09-09T22:59:54.961 DEBUG:boto:Token: None 2022-09-09T22:59:54.961 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 22:59:54 GMT /jkcgbj-48/ 2022-09-09T22:59:54.962 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:1xtAmminYAslb2VL7YCY+W7yC0U= 2022-09-09T22:59:54.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:54 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:1xtAmminYAslb2VL7YCY+W7yC0U=', 'Content-Length': '0'} 2022-09-09T22:59:55.012 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000076ddf07eee3a43e1-00631bc56a-1403-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:55.013 DEBUG:boto:path=/jkcgbj-46/ 2022-09-09T22:59:55.014 DEBUG:boto:auth_path=/jkcgbj-46/ 2022-09-09T22:59:55.014 DEBUG:boto:Method: HEAD 2022-09-09T22:59:55.015 DEBUG:boto:Path: /jkcgbj-46/ 2022-09-09T22:59:55.015 DEBUG:boto:Data: 2022-09-09T22:59:55.016 DEBUG:boto:Headers: {} 2022-09-09T22:59:55.016 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:55.017 DEBUG:boto:Port: 8000 2022-09-09T22:59:55.017 DEBUG:boto:Params: {} 2022-09-09T22:59:55.018 DEBUG:boto:Token: None 2022-09-09T22:59:55.018 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:55 GMT /jkcgbj-46/ 2022-09-09T22:59:55.019 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9lDTvFP7MM/lEMdJH928IBuDMHI= 2022-09-09T22:59:55.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:55 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9lDTvFP7MM/lEMdJH928IBuDMHI=', 'Content-Length': '0'} 2022-09-09T22:59:55.024 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da84a77a0c0f47e5-00631bc56b-13fd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:55.025 DEBUG:boto:path=/jkcgbj-46/key 2022-09-09T22:59:55.025 DEBUG:boto:auth_path=/jkcgbj-46/key 2022-09-09T22:59:55.026 DEBUG:boto:Method: PUT 2022-09-09T22:59:55.026 DEBUG:boto:Path: /jkcgbj-46/key 2022-09-09T22:59:55.027 DEBUG:boto:Data: 2022-09-09T22:59:55.027 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2022-09-09T22:59:55.028 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T22:59:55.029 DEBUG:boto:Port: 8000 2022-09-09T22:59:55.030 DEBUG:boto:Params: {} 2022-09-09T22:59:55.031 DEBUG:boto:Token: None 2022-09-09T22:59:55.031 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Fri, 09 Sep 2022 22:59:55 GMT /jkcgbj-46/key 2022-09-09T22:59:55.032 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:ow9Ulk+EhPntmpxADTR5ON6yRDs= 2022-09-09T22:59:55.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:59:55 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:ow9Ulk+EhPntmpxADTR5ON6yRDs='} 2022-09-09T22:59:55.078 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d565f028595f9096-00631bc56b-13fd-test-zone1'), ('Date', 'Fri, 09 Sep 2022 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:55.079 DEBUG:boto:path=/jkcgbj-47/ 2022-09-09T22:59:55.080 DEBUG:boto:auth_path=/jkcgbj-47/ 2022-09-09T22:59:55.080 DEBUG:boto:Method: HEAD 2022-09-09T22:59:55.081 DEBUG:boto:Path: /jkcgbj-47/ 2022-09-09T22:59:55.081 DEBUG:boto:Data: 2022-09-09T22:59:55.082 DEBUG:boto:Headers: {} 2022-09-09T22:59:55.082 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:55.083 DEBUG:boto:Port: 8000 2022-09-09T22:59:55.084 DEBUG:boto:Params: {} 2022-09-09T22:59:55.084 DEBUG:boto:Token: None 2022-09-09T22:59:55.085 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:55 GMT /jkcgbj-47/ 2022-09-09T22:59:55.085 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:n3iX5EhWm2LdSt9l0Ca4HoSzZWo= 2022-09-09T22:59:55.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:55 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:n3iX5EhWm2LdSt9l0Ca4HoSzZWo=', 'Content-Length': '0'} 2022-09-09T22:59:55.092 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000087c2370fa0532fc6-00631bc56b-1549-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:55.093 DEBUG:boto:path=/jkcgbj-47/key 2022-09-09T22:59:55.093 DEBUG:boto:auth_path=/jkcgbj-47/key 2022-09-09T22:59:55.094 DEBUG:boto:Method: PUT 2022-09-09T22:59:55.094 DEBUG:boto:Path: /jkcgbj-47/key 2022-09-09T22:59:55.095 DEBUG:boto:Data: 2022-09-09T22:59:55.095 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2022-09-09T22:59:55.096 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T22:59:55.096 DEBUG:boto:Port: 8000 2022-09-09T22:59:55.097 DEBUG:boto:Params: {} 2022-09-09T22:59:55.097 DEBUG:boto:Token: None 2022-09-09T22:59:55.098 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Fri, 09 Sep 2022 22:59:55 GMT /jkcgbj-47/key 2022-09-09T22:59:55.098 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:sCVDDSYaO7abFC40d6sbDvC3LyY= 2022-09-09T22:59:55.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:59:55 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:sCVDDSYaO7abFC40d6sbDvC3LyY='} 2022-09-09T22:59:55.146 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000044e7896c66414a7d-00631bc56b-1549-test-zone2'), ('Date', 'Fri, 09 Sep 2022 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:55.147 DEBUG:boto:path=/jkcgbj-48/ 2022-09-09T22:59:55.148 DEBUG:boto:auth_path=/jkcgbj-48/ 2022-09-09T22:59:55.148 DEBUG:boto:Method: HEAD 2022-09-09T22:59:55.149 DEBUG:boto:Path: /jkcgbj-48/ 2022-09-09T22:59:55.149 DEBUG:boto:Data: 2022-09-09T22:59:55.150 DEBUG:boto:Headers: {} 2022-09-09T22:59:55.150 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:55.151 DEBUG:boto:Port: 8001 2022-09-09T22:59:55.151 DEBUG:boto:Params: {} 2022-09-09T22:59:55.152 DEBUG:boto:Token: None 2022-09-09T22:59:55.152 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 22:59:55 GMT /jkcgbj-48/ 2022-09-09T22:59:55.153 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0lmxbGNw+U+JkzrBDUeidKYE1M0= 2022-09-09T22:59:55.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 22:59:55 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0lmxbGNw+U+JkzrBDUeidKYE1M0=', 'Content-Length': '0'} 2022-09-09T22:59:55.158 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd99a123ebe0d013-00631bc56b-1403-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:55.160 DEBUG:boto:path=/jkcgbj-48/key 2022-09-09T22:59:55.161 DEBUG:boto:auth_path=/jkcgbj-48/key 2022-09-09T22:59:55.162 DEBUG:boto:Method: PUT 2022-09-09T22:59:55.163 DEBUG:boto:Path: /jkcgbj-48/key 2022-09-09T22:59:55.164 DEBUG:boto:Data: 2022-09-09T22:59:55.165 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2022-09-09T22:59:55.166 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T22:59:55.166 DEBUG:boto:Port: 8001 2022-09-09T22:59:55.167 DEBUG:boto:Params: {} 2022-09-09T22:59:55.168 DEBUG:boto:Token: None 2022-09-09T22:59:55.168 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Fri, 09 Sep 2022 22:59:55 GMT /jkcgbj-48/key 2022-09-09T22:59:55.169 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:V+ESg+fcpDHDEQVqcnS2Oqe7pJ0= 2022-09-09T22:59:55.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 22:59:55 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:V+ESg+fcpDHDEQVqcnS2Oqe7pJ0='} 2022-09-09T22:59:55.182 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b678473e4b509e2-00631bc56b-1403-test-zone3'), ('Date', 'Fri, 09 Sep 2022 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T22:59:55.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:55.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:55.184 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:59:55.229 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:55.229 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:55.287 INFO:teuthology.orchestra.run.smithi005.stderr:No --period given, using current period=fd4bc158-06fd-4840-8a27-33fabda67b99 2022-09-09T22:59:55.315 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T22:59:55.315 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:55.316 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:59:55.316 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:59:55.316 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:55.316 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:55.317 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:59:55.317 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:59:55.317 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:55.318 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:55.318 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T22:59:55.318 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T22:59:55.318 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T22:59:55.319 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T22:59:55.319 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662764394.995549_383.1", 2022-09-09T22:59:55.319 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:59:54.995549Z" 2022-09-09T22:59:55.319 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T22:59:55.320 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T22:59:55.321 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '', 3: '1_1662764394.995549_383.1'} 2022-09-09T22:59:55.322 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-09T22:59:55.322 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:55.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T22:59:55.323 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T22:59:55.372 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T22:59:55.372 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T22:59:55.435 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:59:55.432+0000 7f75f5636dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:59:55.437 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T22:59:55.432+0000 7f75f5636dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T22:59:55.466 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T22:59:55.466 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T22:59:55.466 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T22:59:55.467 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T22:59:55.467 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T22:59:55.467 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T22:59:55.467 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T22:59:55.468 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:55.468 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T22:59:55.469 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:55.469 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T22:59:55.469 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:59:55.469 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:59:55.470 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T22:59:55.470 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:59:55.470 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T22:59:55.470 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:59:55.471 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T22:59:55.471 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:59:55.471 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:55.471 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:55.472 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:55.472 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T22:59:55.472 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:59:55.472 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:59:55.473 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T22:59:55.473 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:59:55.473 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:59:55.474 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:59:55.474 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T22:59:55.474 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:59:55.475 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:55.475 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:55.475 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:55.476 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T22:59:55.476 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:59:55.476 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:59:55.476 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T22:59:55.477 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:59:55.477 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:59:55.477 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:59:55.478 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T22:59:55.478 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T22:59:55.479 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:55.479 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:55.479 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T22:59:55.479 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T22:59:55.480 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T22:59:55.481 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T22:59:55.481 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764394.924013_376.1", 2022-09-09T22:59:55.481 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T22:59:55.482 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T22:59:55.482 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T22:59:55.482 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:54.924013Z", 2022-09-09T22:59:55.483 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T22:59:55.483 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:55.483 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T22:59:55.484 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T22:59:55.484 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:55.485 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T22:59:55.485 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T22:59:55.485 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T22:59:55.485 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T22:59:55.486 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T22:59:55Z" 2022-09-09T22:59:55.486 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T22:59:55.487 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764394.924013_376.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:54.924013Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T22:59:55Z" } 2022-09-09T22:59:55.488 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764394.924013_376.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:54.924013Z', 'realm_epoch': 4}}] 2022-09-09T22:59:55.489 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764394.995549_383.1'} 2022-09-09T22:59:55.489 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764394.924013_376.1'} 2022-09-09T22:59:55.490 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764394.995549_383.1 target=1_1662764394.924013_376.1 2022-09-09T23:00:00.492 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:00.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:00.494 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:00:00.543 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:00.543 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:00.606 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:00:00.604+0000 7fc522fdadc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:00:00.608 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:00:00.604+0000 7fc522fdadc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:00:00.630 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:00:00.631 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:00:00.631 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:00:00.631 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:00:00.631 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:00:00.632 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:00:00.632 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:00:00.632 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:00.632 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:00:00.633 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:00.633 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:00:00.633 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:00:00.633 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:00:00.634 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:00:00.634 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:00:00.634 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:00:00.634 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:00:00.635 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:00:00.635 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:00:00.635 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:00.635 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:00.636 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:00.636 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:00:00.636 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:00:00.636 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:00:00.637 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:00:00.637 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:00:00.637 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:00:00.637 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:00:00.638 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:00:00.638 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:00:00.638 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:00.638 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:00.639 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:00.639 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:00:00.639 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:00:00.639 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:00:00.640 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:00:00.640 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:00:00.640 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:00:00.640 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:00:00.641 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:00:00.641 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:00:00.641 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:00.641 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:00.641 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:00.642 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:00:00.642 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:00:00.642 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:00:00.642 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764394.995549_383.1", 2022-09-09T23:00:00.643 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:00:00.643 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:00:00.643 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:00:00.643 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:54.995549Z", 2022-09-09T23:00:00.644 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:00:00.644 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:00.644 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:00.644 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:00:00.645 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:00.645 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:00:00.645 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:00:00.645 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:00:00.646 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:00.646 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:00:00Z" 2022-09-09T23:00:00.646 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:00:00.647 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764394.995549_383.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:54.995549Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:00:00Z" } 2022-09-09T23:00:00.648 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764394.995549_383.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:54.995549Z', 'realm_epoch': 4}}] 2022-09-09T23:00:00.649 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764394.995549_383.1'} 2022-09-09T23:00:00.649 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764394.995549_383.1'} 2022-09-09T23:00:00.650 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-09T23:00:00.650 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-09T23:00:00.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:00.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:00.652 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:00:00.700 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:00.700 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:00.750 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T23:00:00.748+0000 7f8125307dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:00:00.752 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T23:00:00.752+0000 7f8125307dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:00:00.778 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T23:00:00.778 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T23:00:00.779 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T23:00:00.779 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T23:00:00.779 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T23:00:00.779 INFO:teuthology.orchestra.run.smithi005.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:00:00.780 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 4 2022-09-09T23:00:00.780 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:00.781 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T23:00:00.781 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:00.782 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T23:00:00.782 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:00.782 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T23:00:00.783 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:00:00.784 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:00.784 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 1, 2022-09-09T23:00:00.784 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:00.785 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:00:00.785 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T23:00:00.785 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:00.785 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:00.786 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:00.786 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T23:00:00.786 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:00.787 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T23:00:00.787 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T23:00:00.787 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:00.787 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:00.788 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:00.788 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T23:00:00.788 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T23:00:00.788 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:00.789 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:00.789 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:00.790 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T23:00:00.790 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:00.790 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T23:00:00.790 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:00:00.791 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:00.791 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:00.791 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:00.791 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:00:00.792 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T23:00:00.792 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:00.792 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:00.792 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:00.793 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T23:00:00.793 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:00.793 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T23:00:00.793 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662764394.995549_383.1", 2022-09-09T23:00:00.794 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:00.794 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:00.794 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:00.795 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:59:54.995549Z", 2022-09-09T23:00:00.795 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 4 2022-09-09T23:00:00.795 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:00.795 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:00.796 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:00:00.796 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:00.796 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T23:00:00.796 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 1, 2022-09-09T23:00:00.797 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 1 2022-09-09T23:00:00.797 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:00.797 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T23:00:00Z" 2022-09-09T23:00:00.797 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T23:00:00.799 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764394.995549_383.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:54.995549Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:00:00Z" } 2022-09-09T23:00:00.799 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764394.995549_383.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:54.995549Z', 'realm_epoch': 4}}] 2022-09-09T23:00:00.800 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764394.995549_383.1'} 2022-09-09T23:00:00.801 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764394.995549_383.1'} 2022-09-09T23:00:00.801 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-09T23:00:00.801 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-09T23:00:00.802 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:00.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:00.803 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:00:00.849 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:00.850 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:00.913 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:00:00.908+0000 7f71089f1dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:00:00.916 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:00:00.912+0000 7f71089f1dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:00:00.940 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:00:00.941 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:00:00.941 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:00:00.942 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:00:00.942 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:00:00.943 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:00:00.943 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:00:00.944 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:00.944 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:00:00.945 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:00.945 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:00:00.946 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:00:00.946 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:00:00.947 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:00:00.947 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:00:00.947 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:00:00.948 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:00:00.948 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:00:00.949 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:00:00.950 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:00.950 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:00.950 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:00.951 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:00:00.951 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:00:00.951 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:00:00.952 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:00:00.952 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:00:00.952 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:00:00.952 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:00:00.953 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:00:00.953 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:00:00.953 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:00.954 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:00.954 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:00.955 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:00:00.955 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:00:00.955 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:00:00.956 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:00:00.956 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:00:00.956 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:00:00.956 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:00:00.957 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:00:00.957 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:00:00.958 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:00.958 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:00.959 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:00.959 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:00:00.960 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:00:00.960 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:00:00.960 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764394.995549_383.1", 2022-09-09T23:00:00.961 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:00:00.961 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:00:00.961 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:00:00.962 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:54.995549Z", 2022-09-09T23:00:00.962 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:00:00.962 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:00.963 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:00.963 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:00:00.963 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:00.963 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:00:00.964 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:00:00.964 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:00:00.964 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:00.964 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:00:00Z" 2022-09-09T23:00:00.965 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:00:00.966 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764394.995549_383.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:54.995549Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:00:00Z" } 2022-09-09T23:00:00.967 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764394.995549_383.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:54.995549Z', 'realm_epoch': 4}}] 2022-09-09T23:00:00.967 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764394.995549_383.1'} 2022-09-09T23:00:00.968 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764394.995549_383.1'} 2022-09-09T23:00:00.968 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-09T23:00:00.968 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone2 2022-09-09T23:00:00.969 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:00.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:00.971 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:00:01.019 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:01.019 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:01.117 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T23:00:01.118 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:01.118 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-09T23:00:01.118 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:59:53.537383Z" 2022-09-09T23:00:01.118 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:01.119 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:01.119 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000002768", 2022-09-09T23:00:01.119 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:38:39.082329Z" 2022-09-09T23:00:01.119 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:01.120 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:01.120 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000004319", 2022-09-09T23:00:01.120 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:59:55.048046Z" 2022-09-09T23:00:01.120 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:01.121 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:01.121 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000005870", 2022-09-09T23:00:01.121 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:40:26.882485Z" 2022-09-09T23:00:01.121 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:01.122 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T23:00:01.123 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002768', 2: '00000000000000000000:00000000000000004319', 3: '00000000000000000000:00000000000000005870'} 2022-09-09T23:00:01.124 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2022-09-09T23:00:01.124 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:01.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:01.126 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:00:01.173 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:01.174 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:01.275 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:00:01.276 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:00:01.276 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:00:01.276 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:00:01.276 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:00:01.277 INFO:teuthology.orchestra.run.smithi080.stdout: "instance_id": 315074542659551751 2022-09-09T23:00:01.277 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:01.277 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:00:01.277 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:01.278 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:00:01.278 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:00:01.278 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:00:01.278 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000000654", 2022-09-09T23:00:01.279 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:00:01.279 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:00:01.279 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:00:01.279 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:57:20.235834Z" 2022-09-09T23:00:01.280 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:01.280 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:01.280 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:01.280 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:00:01.281 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:00:01.281 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:00:01.281 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000002768", 2022-09-09T23:00:01.281 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:00:01.282 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:00:01.282 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:00:01.283 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:38:39.082329Z" 2022-09-09T23:00:01.283 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:01.284 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:01.284 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:01.284 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:00:01.285 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:00:01.285 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:00:01.285 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000004178", 2022-09-09T23:00:01.285 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:00:01.286 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:00:01.286 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:00:01.286 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:50:42.920582Z" 2022-09-09T23:00:01.287 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:01.287 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:01.287 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:01.287 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:00:01.288 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:00:01.288 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:00:01.288 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000005870", 2022-09-09T23:00:01.288 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:00:01.289 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:00:01.289 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:00:01.289 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:40:26.882485Z" 2022-09-09T23:00:01.289 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:01.290 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:01.290 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:00:01.290 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:01.290 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:00:01.291 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 0, 2022-09-09T23:00:01.291 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 0 2022-09-09T23:00:01.291 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:01.291 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:00:01Z" 2022-09-09T23:00:01.292 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:00:01.293 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 315074542659551751 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000654", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:57:20.235834Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002768", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:38:39.082329Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004178", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:50:42.920582Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005870", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:40:26.882485Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-09T23:00:01Z" } 2022-09-09T23:00:01.294 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000654', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:57:20.235834Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002768', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:38:39.082329Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004178', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:50:42.920582Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005870', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:40:26.882485Z'}}] 2022-09-09T23:00:01.295 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002768', 2: '00000000000000000000:00000000000000004319', 3: '00000000000000000000:00000000000000005870'} 2022-09-09T23:00:01.295 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000654', 1: '00000000000000000000:00000000000000002768', 2: '00000000000000000000:00000000000000004178', 3: '00000000000000000000:00000000000000005870'} 2022-09-09T23:00:01.296 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000000796 target=00000000000000000000:00000000000000000654, shard=2 master=00000000000000000000:00000000000000004319 target=00000000000000000000:00000000000000004178 2022-09-09T23:00:06.296 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:06.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:06.298 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:00:06.344 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:06.345 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:06.432 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:00:06.432 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:00:06.433 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:00:06.433 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:00:06.433 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:00:06.434 INFO:teuthology.orchestra.run.smithi080.stdout: "instance_id": 315074542659551751 2022-09-09T23:00:06.434 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:06.435 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:00:06.435 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:06.435 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:00:06.436 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:00:06.436 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:00:06.436 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-09T23:00:06.437 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:00:06.437 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:00:06.438 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:00:06.438 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:53.537383Z" 2022-09-09T23:00:06.439 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:06.439 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:06.440 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:06.441 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:00:06.441 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:00:06.441 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:00:06.442 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000002768", 2022-09-09T23:00:06.442 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:00:06.443 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:00:06.443 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:00:06.444 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:38:39.082329Z" 2022-09-09T23:00:06.444 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:06.445 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:06.445 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:06.446 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:00:06.446 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:00:06.447 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:00:06.447 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000004319", 2022-09-09T23:00:06.448 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:00:06.448 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:00:06.448 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:00:06.449 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:55.048046Z" 2022-09-09T23:00:06.449 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:06.450 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:06.450 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:06.451 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:00:06.451 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:00:06.452 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:00:06.452 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000005870", 2022-09-09T23:00:06.452 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:00:06.453 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:00:06.453 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:00:06.454 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:40:26.882485Z" 2022-09-09T23:00:06.454 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:06.455 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:06.455 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:00:06.456 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:06.456 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:00:06.457 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 0, 2022-09-09T23:00:06.457 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 0 2022-09-09T23:00:06.458 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:06.458 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:00:06Z" 2022-09-09T23:00:06.459 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:00:06.461 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 315074542659551751 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:53.537383Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002768", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:38:39.082329Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:55.048046Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005870", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:40:26.882485Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-09T23:00:06Z" } 2022-09-09T23:00:06.462 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:53.537383Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002768', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:38:39.082329Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:55.048046Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005870', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:40:26.882485Z'}}] 2022-09-09T23:00:06.463 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002768', 2: '00000000000000000000:00000000000000004319', 3: '00000000000000000000:00000000000000005870'} 2022-09-09T23:00:06.464 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002768', 2: '00000000000000000000:00000000000000004319', 3: '00000000000000000000:00000000000000005870'} 2022-09-09T23:00:06.465 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2022-09-09T23:00:06.465 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone3 2022-09-09T23:00:06.470 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:06.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:06.471 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:00:06.518 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:06.518 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:06.619 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T23:00:06.619 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:06.620 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-09T23:00:06.620 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:59:53.537383Z" 2022-09-09T23:00:06.621 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:06.621 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:06.622 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000002768", 2022-09-09T23:00:06.623 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:38:39.082329Z" 2022-09-09T23:00:06.623 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:06.623 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:06.623 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000004459", 2022-09-09T23:00:06.624 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T23:00:04.557222Z" 2022-09-09T23:00:06.624 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:06.624 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:06.624 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000005870", 2022-09-09T23:00:06.625 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:40:26.882485Z" 2022-09-09T23:00:06.625 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:06.625 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T23:00:06.627 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002768', 2: '00000000000000000000:00000000000000004459', 3: '00000000000000000000:00000000000000005870'} 2022-09-09T23:00:06.627 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2022-09-09T23:00:06.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:06.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:06.629 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:00:06.677 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:06.678 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:06.761 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T23:00:06.761 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T23:00:06.761 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T23:00:06.762 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T23:00:06.762 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T23:00:06.762 INFO:teuthology.orchestra.run.smithi005.stdout: "instance_id": 7350307779150546029 2022-09-09T23:00:06.762 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:06.763 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T23:00:06.763 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:06.763 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T23:00:06.763 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:06.764 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:06.764 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-09T23:00:06.764 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:06.764 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:06.765 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:06.765 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:59:53.537383Z" 2022-09-09T23:00:06.765 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:06.765 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:06.766 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:06.766 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T23:00:06.766 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:06.766 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:06.767 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000002768", 2022-09-09T23:00:06.767 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:06.767 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:06.767 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:06.768 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:38:39.082329Z" 2022-09-09T23:00:06.768 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:06.768 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:06.768 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:06.769 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T23:00:06.769 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:06.769 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:06.769 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000004178", 2022-09-09T23:00:06.770 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:06.770 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:06.770 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:06.770 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:42.920582Z" 2022-09-09T23:00:06.771 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:06.771 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:06.771 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:06.771 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T23:00:06.772 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:06.772 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:06.772 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000005870", 2022-09-09T23:00:06.773 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:06.773 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:06.773 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:06.773 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:40:26.882485Z" 2022-09-09T23:00:06.774 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:06.774 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:06.774 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:00:06.774 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:06.775 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T23:00:06.775 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 0, 2022-09-09T23:00:06.775 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 0 2022-09-09T23:00:06.775 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:06.776 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T23:00:06Z" 2022-09-09T23:00:06.776 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T23:00:06.777 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7350307779150546029 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:53.537383Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002768", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:38:39.082329Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004178", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:50:42.920582Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005870", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:40:26.882485Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-09T23:00:06Z" } 2022-09-09T23:00:06.778 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:53.537383Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002768', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:38:39.082329Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004178', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:50:42.920582Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005870', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:40:26.882485Z'}}] 2022-09-09T23:00:06.779 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002768', 2: '00000000000000000000:00000000000000004459', 3: '00000000000000000000:00000000000000005870'} 2022-09-09T23:00:06.779 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002768', 2: '00000000000000000000:00000000000000004178', 3: '00000000000000000000:00000000000000005870'} 2022-09-09T23:00:06.779 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=2 master=00000000000000000000:00000000000000004459 target=00000000000000000000:00000000000000004178 2022-09-09T23:00:11.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:11.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:11.782 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:00:11.834 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:11.834 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:11.917 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T23:00:11.917 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T23:00:11.918 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T23:00:11.918 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T23:00:11.918 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T23:00:11.918 INFO:teuthology.orchestra.run.smithi005.stdout: "instance_id": 7350307779150546029 2022-09-09T23:00:11.919 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:11.919 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T23:00:11.919 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:11.920 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T23:00:11.920 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:11.920 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:11.921 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-09T23:00:11.921 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:11.921 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:11.922 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:11.922 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:59:53.537383Z" 2022-09-09T23:00:11.922 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:11.923 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:11.923 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:11.923 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T23:00:11.924 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:11.924 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:11.924 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000002768", 2022-09-09T23:00:11.924 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:11.925 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:11.925 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:11.925 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:38:39.082329Z" 2022-09-09T23:00:11.925 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:11.926 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:11.926 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:11.927 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T23:00:11.927 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:11.927 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:11.927 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000004178", 2022-09-09T23:00:11.928 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:11.928 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:11.928 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:11.928 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:50:42.920582Z" 2022-09-09T23:00:11.929 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:11.929 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:11.929 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:11.930 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T23:00:11.930 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:11.930 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:11.930 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000005870", 2022-09-09T23:00:11.931 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:11.931 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:11.931 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:11.931 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:40:26.882485Z" 2022-09-09T23:00:11.932 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:11.932 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:11.932 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:00:11.933 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:11.933 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T23:00:11.933 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 0, 2022-09-09T23:00:11.933 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 0 2022-09-09T23:00:11.934 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:11.934 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T23:00:11Z" 2022-09-09T23:00:11.934 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T23:00:11.936 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7350307779150546029 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:53.537383Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002768", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:38:39.082329Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004178", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:50:42.920582Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005870", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:40:26.882485Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-09T23:00:11Z" } 2022-09-09T23:00:11.937 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:53.537383Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002768', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:38:39.082329Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004178', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:50:42.920582Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005870', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:40:26.882485Z'}}] 2022-09-09T23:00:11.937 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002768', 2: '00000000000000000000:00000000000000004459', 3: '00000000000000000000:00000000000000005870'} 2022-09-09T23:00:11.938 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002768', 2: '00000000000000000000:00000000000000004178', 3: '00000000000000000000:00000000000000005870'} 2022-09-09T23:00:11.938 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=2 master=00000000000000000000:00000000000000004459 target=00000000000000000000:00000000000000004178 2022-09-09T23:00:16.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:16.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:16.941 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:00:16.990 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:16.991 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:17.079 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T23:00:17.079 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T23:00:17.080 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T23:00:17.080 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T23:00:17.080 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T23:00:17.080 INFO:teuthology.orchestra.run.smithi005.stdout: "instance_id": 7350307779150546029 2022-09-09T23:00:17.081 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:17.081 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T23:00:17.081 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:17.081 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T23:00:17.082 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:17.082 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:17.082 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-09T23:00:17.082 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:17.083 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:17.083 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:17.083 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:59:53.537383Z" 2022-09-09T23:00:17.083 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:17.084 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:17.084 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:17.084 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T23:00:17.084 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:17.085 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:17.085 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000002768", 2022-09-09T23:00:17.085 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:17.086 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:17.086 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:17.086 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:38:39.082329Z" 2022-09-09T23:00:17.086 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:17.087 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:17.087 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:17.087 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T23:00:17.087 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:17.088 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:17.088 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000004599", 2022-09-09T23:00:17.088 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:17.088 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:17.089 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:17.089 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T23:00:13.517424Z" 2022-09-09T23:00:17.089 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:17.089 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:17.090 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:17.090 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T23:00:17.090 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:17.090 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:17.091 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000005870", 2022-09-09T23:00:17.091 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:17.091 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:17.091 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:17.092 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:40:26.882485Z" 2022-09-09T23:00:17.092 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:17.092 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:17.092 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:00:17.093 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:17.093 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T23:00:17.093 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 0, 2022-09-09T23:00:17.093 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 0 2022-09-09T23:00:17.094 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:17.094 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T23:00:17Z" 2022-09-09T23:00:17.094 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T23:00:17.095 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7350307779150546029 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:53.537383Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002768", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:38:39.082329Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004599", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:00:13.517424Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005870", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:40:26.882485Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-09T23:00:17Z" } 2022-09-09T23:00:17.096 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:53.537383Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002768', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:38:39.082329Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004599', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:00:13.517424Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005870', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:40:26.882485Z'}}] 2022-09-09T23:00:17.097 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002768', 2: '00000000000000000000:00000000000000004459', 3: '00000000000000000000:00000000000000005870'} 2022-09-09T23:00:17.097 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002768', 2: '00000000000000000000:00000000000000004599', 3: '00000000000000000000:00000000000000005870'} 2022-09-09T23:00:17.098 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2022-09-09T23:00:17.098 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone4 2022-09-09T23:00:17.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:17.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:17.100 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:00:17.146 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:17.146 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:17.241 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T23:00:17.242 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:17.242 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-09T23:00:17.243 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:59:53.537383Z" 2022-09-09T23:00:17.243 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:17.244 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:17.244 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000002768", 2022-09-09T23:00:17.245 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:38:39.082329Z" 2022-09-09T23:00:17.245 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:17.246 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:17.246 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000004599", 2022-09-09T23:00:17.247 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T23:00:13.517424Z" 2022-09-09T23:00:17.247 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:17.248 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:17.248 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000005870", 2022-09-09T23:00:17.249 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:40:26.882485Z" 2022-09-09T23:00:17.249 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:17.250 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T23:00:17.252 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002768', 2: '00000000000000000000:00000000000000004599', 3: '00000000000000000000:00000000000000005870'} 2022-09-09T23:00:17.252 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2022-09-09T23:00:17.253 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:17.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:17.254 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:00:17.303 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:17.304 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:17.392 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:00:17.393 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:00:17.393 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:00:17.393 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:00:17.394 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:00:17.394 INFO:teuthology.orchestra.run.smithi080.stdout: "instance_id": 4950540593819681557 2022-09-09T23:00:17.394 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:17.395 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:00:17.395 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:17.395 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:00:17.395 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:00:17.396 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:00:17.396 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-09T23:00:17.396 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:00:17.396 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:00:17.397 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:00:17.397 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:53.537383Z" 2022-09-09T23:00:17.397 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:17.398 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:17.398 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:17.398 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:00:17.398 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:00:17.399 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:00:17.399 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000002768", 2022-09-09T23:00:17.399 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:00:17.399 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:00:17.400 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:00:17.400 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:38:39.082329Z" 2022-09-09T23:00:17.400 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:17.400 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:17.401 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:17.401 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:00:17.401 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:00:17.401 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:00:17.402 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000004599", 2022-09-09T23:00:17.402 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:00:17.402 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:00:17.403 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:00:17.403 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:00:13.517424Z" 2022-09-09T23:00:17.403 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:17.404 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:17.404 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:17.404 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:00:17.404 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:00:17.405 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:00:17.405 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000005870", 2022-09-09T23:00:17.405 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:00:17.405 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:00:17.406 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:00:17.406 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:40:26.882485Z" 2022-09-09T23:00:17.406 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:17.406 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:17.407 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:00:17.407 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:17.407 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:00:17.407 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 0, 2022-09-09T23:00:17.408 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 0 2022-09-09T23:00:17.408 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:17.408 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:00:17Z" 2022-09-09T23:00:17.408 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:00:17.410 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 4950540593819681557 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:53.537383Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002768", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:38:39.082329Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004599", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:00:13.517424Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005870", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:40:26.882485Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-09T23:00:17Z" } 2022-09-09T23:00:17.410 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:53.537383Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002768', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:38:39.082329Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004599', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:00:13.517424Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005870', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:40:26.882485Z'}}] 2022-09-09T23:00:17.411 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002768', 2: '00000000000000000000:00000000000000004599', 3: '00000000000000000000:00000000000000005870'} 2022-09-09T23:00:17.412 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002768', 2: '00000000000000000000:00000000000000004599', 3: '00000000000000000000:00000000000000005870'} 2022-09-09T23:00:17.412 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2022-09-09T23:00:17.413 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone1 2022-09-09T23:00:17.413 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:17.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:17.414 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:00:17.459 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:17.459 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:17.565 INFO:teuthology.orchestra.run.smithi080.stdout:[ 2022-09-09T23:00:17.566 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:17.566 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000000654", 2022-09-09T23:00:17.567 INFO:teuthology.orchestra.run.smithi080.stdout: "last_update": "2022-09-09T22:59:53.587319Z" 2022-09-09T23:00:17.567 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:17.567 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:17.567 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000002627", 2022-09-09T23:00:17.568 INFO:teuthology.orchestra.run.smithi080.stdout: "last_update": "2022-09-09T22:38:18.735605Z" 2022-09-09T23:00:17.568 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:17.568 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:17.568 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000006151", 2022-09-09T23:00:17.569 INFO:teuthology.orchestra.run.smithi080.stdout: "last_update": "2022-09-09T23:00:04.567958Z" 2022-09-09T23:00:17.569 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:17.569 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:17.570 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-09T23:00:17.570 INFO:teuthology.orchestra.run.smithi080.stdout: "last_update": "2022-09-09T22:40:26.940143Z" 2022-09-09T23:00:17.570 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:17.570 INFO:teuthology.orchestra.run.smithi080.stdout:] 2022-09-09T23:00:17.574 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000000654', 1: '00000000000000000000:00000000000000002627', 2: '00000000000000000000:00000000000000006151', 3: '00000000000000000000:00000000000000006434'} 2022-09-09T23:00:17.575 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2022-09-09T23:00:17.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:17.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:17.578 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:00:17.629 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:17.630 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:17.712 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T23:00:17.712 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T23:00:17.712 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T23:00:17.713 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T23:00:17.713 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T23:00:17.713 INFO:teuthology.orchestra.run.smithi005.stdout: "instance_id": 5471899075723489543 2022-09-09T23:00:17.714 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:17.714 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T23:00:17.714 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:17.714 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T23:00:17.715 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:17.715 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:17.715 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000000654", 2022-09-09T23:00:17.716 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:17.716 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:17.716 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:17.716 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:59:53.587319Z" 2022-09-09T23:00:17.717 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:17.717 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:17.717 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:17.717 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T23:00:17.718 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:17.718 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:17.718 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000002627", 2022-09-09T23:00:17.719 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:17.719 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:17.719 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:17.719 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:38:18.735605Z" 2022-09-09T23:00:17.720 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:17.720 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:17.720 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:17.720 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T23:00:17.721 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:17.721 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:17.721 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000006011", 2022-09-09T23:00:17.721 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:17.722 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:17.722 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:17.730 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:59:55.116050Z" 2022-09-09T23:00:17.731 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:17.731 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:17.731 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:17.732 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T23:00:17.732 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:17.732 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:17.732 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-09T23:00:17.733 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:17.733 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:17.734 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:17.735 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:40:26.940143Z" 2022-09-09T23:00:17.736 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:17.736 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:17.737 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:00:17.737 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:17.737 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T23:00:17.738 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 0, 2022-09-09T23:00:17.738 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 0 2022-09-09T23:00:17.739 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:17.739 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T23:00:17Z" 2022-09-09T23:00:17.739 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T23:00:17.741 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5471899075723489543 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000654", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:53.587319Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002627", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:38:18.735605Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006011", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:55.116050Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006434", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:40:26.940143Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-09T23:00:17Z" } 2022-09-09T23:00:17.742 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000654', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:53.587319Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002627', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:38:18.735605Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006011', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:55.116050Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006434', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:40:26.940143Z'}}] 2022-09-09T23:00:17.742 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000654', 1: '00000000000000000000:00000000000000002627', 2: '00000000000000000000:00000000000000006151', 3: '00000000000000000000:00000000000000006434'} 2022-09-09T23:00:17.743 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000654', 1: '00000000000000000000:00000000000000002627', 2: '00000000000000000000:00000000000000006011', 3: '00000000000000000000:00000000000000006434'} 2022-09-09T23:00:17.743 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=2 master=00000000000000000000:00000000000000006151 target=00000000000000000000:00000000000000006011 2022-09-09T23:00:22.745 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:22.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:22.748 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:00:22.800 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:22.801 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:22.881 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T23:00:22.882 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T23:00:22.883 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T23:00:22.883 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T23:00:22.883 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T23:00:22.883 INFO:teuthology.orchestra.run.smithi005.stdout: "instance_id": 5471899075723489543 2022-09-09T23:00:22.884 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:22.884 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T23:00:22.884 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:22.885 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T23:00:22.885 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:22.885 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:22.886 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000000654", 2022-09-09T23:00:22.886 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:22.887 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:22.887 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:22.888 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:59:53.587319Z" 2022-09-09T23:00:22.888 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:22.889 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:22.889 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:22.890 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T23:00:22.890 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:22.891 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:22.891 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000002627", 2022-09-09T23:00:22.891 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:22.892 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:22.892 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:22.893 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:38:18.735605Z" 2022-09-09T23:00:22.893 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:22.894 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:22.894 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:22.895 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T23:00:22.895 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:22.896 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:22.896 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000006011", 2022-09-09T23:00:22.897 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:22.897 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:22.898 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:22.898 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:59:55.116050Z" 2022-09-09T23:00:22.899 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:22.899 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:22.900 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:22.900 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T23:00:22.901 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:22.901 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:22.902 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-09T23:00:22.902 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:22.903 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:22.903 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:22.904 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:40:26.940143Z" 2022-09-09T23:00:22.904 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:22.904 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:22.905 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:00:22.905 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:22.906 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T23:00:22.906 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 0, 2022-09-09T23:00:22.907 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 0 2022-09-09T23:00:22.907 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:22.908 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T23:00:22Z" 2022-09-09T23:00:22.909 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T23:00:22.912 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5471899075723489543 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000654", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:53.587319Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002627", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:38:18.735605Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006011", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:55.116050Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006434", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:40:26.940143Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-09T23:00:22Z" } 2022-09-09T23:00:22.913 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000654', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:53.587319Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002627', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:38:18.735605Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006011', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:55.116050Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006434', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:40:26.940143Z'}}] 2022-09-09T23:00:22.914 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000654', 1: '00000000000000000000:00000000000000002627', 2: '00000000000000000000:00000000000000006151', 3: '00000000000000000000:00000000000000006434'} 2022-09-09T23:00:22.915 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000654', 1: '00000000000000000000:00000000000000002627', 2: '00000000000000000000:00000000000000006011', 3: '00000000000000000000:00000000000000006434'} 2022-09-09T23:00:22.916 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=2 master=00000000000000000000:00000000000000006151 target=00000000000000000000:00000000000000006011 2022-09-09T23:00:27.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:27.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:27.919 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:00:27.970 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:27.970 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:28.057 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T23:00:28.058 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T23:00:28.058 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T23:00:28.058 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T23:00:28.059 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T23:00:28.059 INFO:teuthology.orchestra.run.smithi005.stdout: "instance_id": 5471899075723489543 2022-09-09T23:00:28.060 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:28.060 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T23:00:28.060 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:28.061 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T23:00:28.061 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:28.061 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:28.061 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000000654", 2022-09-09T23:00:28.062 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:28.062 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:28.062 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:28.062 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:59:53.587319Z" 2022-09-09T23:00:28.063 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:28.063 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:28.063 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:28.064 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T23:00:28.064 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:28.065 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:28.066 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000002627", 2022-09-09T23:00:28.066 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:28.066 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:28.067 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:28.067 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:38:18.735605Z" 2022-09-09T23:00:28.072 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:28.073 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:28.073 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:28.074 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T23:00:28.074 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:28.075 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:28.076 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000006151", 2022-09-09T23:00:28.076 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:28.077 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:28.078 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:28.078 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T23:00:04.567958Z" 2022-09-09T23:00:28.079 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:28.079 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:28.080 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:28.080 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T23:00:28.081 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:28.081 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:28.081 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-09T23:00:28.082 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:28.082 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:28.083 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:28.083 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:40:26.940143Z" 2022-09-09T23:00:28.084 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:28.084 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:28.085 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:00:28.085 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:28.086 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T23:00:28.086 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 0, 2022-09-09T23:00:28.087 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 0 2022-09-09T23:00:28.087 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:28.088 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T23:00:28Z" 2022-09-09T23:00:28.088 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T23:00:28.090 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5471899075723489543 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000654", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:53.587319Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002627", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:38:18.735605Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006151", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:00:04.567958Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006434", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:40:26.940143Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-09T23:00:28Z" } 2022-09-09T23:00:28.091 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000654', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:53.587319Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002627', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:38:18.735605Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006151', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:00:04.567958Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006434', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:40:26.940143Z'}}] 2022-09-09T23:00:28.092 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000654', 1: '00000000000000000000:00000000000000002627', 2: '00000000000000000000:00000000000000006151', 3: '00000000000000000000:00000000000000006434'} 2022-09-09T23:00:28.093 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000654', 1: '00000000000000000000:00000000000000002627', 2: '00000000000000000000:00000000000000006151', 3: '00000000000000000000:00000000000000006434'} 2022-09-09T23:00:28.094 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2022-09-09T23:00:28.095 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone3 2022-09-09T23:00:28.096 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:28.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:28.100 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:00:28.145 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:28.146 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:28.259 INFO:teuthology.orchestra.run.smithi080.stdout:[ 2022-09-09T23:00:28.260 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:28.261 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-09T23:00:28.261 INFO:teuthology.orchestra.run.smithi080.stdout: "last_update": "2022-09-09T23:00:24.570307Z" 2022-09-09T23:00:28.262 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:28.262 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:28.263 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000002627", 2022-09-09T23:00:28.263 INFO:teuthology.orchestra.run.smithi080.stdout: "last_update": "2022-09-09T22:38:18.735605Z" 2022-09-09T23:00:28.265 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:28.265 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:28.266 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000006291", 2022-09-09T23:00:28.266 INFO:teuthology.orchestra.run.smithi080.stdout: "last_update": "2022-09-09T23:00:24.580881Z" 2022-09-09T23:00:28.266 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:28.267 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:28.267 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-09T23:00:28.267 INFO:teuthology.orchestra.run.smithi080.stdout: "last_update": "2022-09-09T22:40:26.940143Z" 2022-09-09T23:00:28.267 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:28.268 INFO:teuthology.orchestra.run.smithi080.stdout:] 2022-09-09T23:00:28.269 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002627', 2: '00000000000000000000:00000000000000006291', 3: '00000000000000000000:00000000000000006434'} 2022-09-09T23:00:28.269 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2022-09-09T23:00:28.270 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:28.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:28.271 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:00:28.321 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:28.321 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:28.408 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T23:00:28.409 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T23:00:28.409 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T23:00:28.410 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T23:00:28.410 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T23:00:28.411 INFO:teuthology.orchestra.run.smithi005.stdout: "instance_id": 5977856695850153756 2022-09-09T23:00:28.411 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:28.411 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T23:00:28.412 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:28.412 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T23:00:28.413 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:28.414 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:28.414 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000000654", 2022-09-09T23:00:28.415 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:28.415 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:28.416 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:28.416 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:59:53.587319Z" 2022-09-09T23:00:28.416 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:28.417 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:28.417 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:28.418 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T23:00:28.418 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:28.419 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:28.419 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000002627", 2022-09-09T23:00:28.420 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:28.420 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:28.420 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:28.421 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:38:18.735605Z" 2022-09-09T23:00:28.421 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:28.422 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:28.422 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:28.423 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T23:00:28.423 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:28.424 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:28.424 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000006151", 2022-09-09T23:00:28.425 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:28.425 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:28.426 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:28.426 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T23:00:04.567958Z" 2022-09-09T23:00:28.427 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:28.427 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:28.428 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:28.428 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T23:00:28.429 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:28.429 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:28.429 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-09T23:00:28.430 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:28.430 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:28.431 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:28.431 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:40:26.940143Z" 2022-09-09T23:00:28.432 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:28.432 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:28.433 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:00:28.433 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:28.433 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T23:00:28.434 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 0, 2022-09-09T23:00:28.434 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 0 2022-09-09T23:00:28.435 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:28.435 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T23:00:28Z" 2022-09-09T23:00:28.436 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T23:00:28.437 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5977856695850153756 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000654", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:53.587319Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002627", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:38:18.735605Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006151", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:00:04.567958Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006434", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:40:26.940143Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-09T23:00:28Z" } 2022-09-09T23:00:28.438 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000654', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:53.587319Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002627', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:38:18.735605Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006151', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:00:04.567958Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006434', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:40:26.940143Z'}}] 2022-09-09T23:00:28.439 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002627', 2: '00000000000000000000:00000000000000006291', 3: '00000000000000000000:00000000000000006434'} 2022-09-09T23:00:28.439 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000654', 1: '00000000000000000000:00000000000000002627', 2: '00000000000000000000:00000000000000006151', 3: '00000000000000000000:00000000000000006434'} 2022-09-09T23:00:28.440 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone2: shard=0 master=00000000000000000000:00000000000000000796 target=00000000000000000000:00000000000000000654, shard=2 master=00000000000000000000:00000000000000006291 target=00000000000000000000:00000000000000006151 2022-09-09T23:00:33.442 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:33.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:33.444 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:00:33.509 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:33.509 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:33.581 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T23:00:33.582 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T23:00:33.582 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T23:00:33.582 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T23:00:33.582 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T23:00:33.583 INFO:teuthology.orchestra.run.smithi005.stdout: "instance_id": 5977856695850153756 2022-09-09T23:00:33.583 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:33.583 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T23:00:33.584 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:33.585 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T23:00:33.585 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:33.585 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:33.586 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000000654", 2022-09-09T23:00:33.586 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:33.587 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:33.587 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:33.587 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:59:53.587319Z" 2022-09-09T23:00:33.587 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:33.588 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:33.588 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:33.588 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T23:00:33.588 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:33.589 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:33.589 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000002627", 2022-09-09T23:00:33.589 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:33.589 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:33.590 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:33.590 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:38:18.735605Z" 2022-09-09T23:00:33.590 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:33.590 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:33.591 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:33.591 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T23:00:33.591 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:33.591 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:33.592 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000006151", 2022-09-09T23:00:33.592 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:33.592 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:33.592 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:33.593 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T23:00:04.567958Z" 2022-09-09T23:00:33.593 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:33.593 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:33.594 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:33.594 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T23:00:33.594 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:33.594 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:33.595 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-09T23:00:33.595 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:33.595 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:33.595 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:33.596 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:40:26.940143Z" 2022-09-09T23:00:33.596 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:33.597 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:33.597 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:00:33.597 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:33.598 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T23:00:33.598 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 0, 2022-09-09T23:00:33.598 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 0 2022-09-09T23:00:33.599 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:33.599 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T23:00:33Z" 2022-09-09T23:00:33.599 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T23:00:33.600 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5977856695850153756 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000654", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:53.587319Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002627", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:38:18.735605Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006151", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:00:04.567958Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006434", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:40:26.940143Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-09T23:00:33Z" } 2022-09-09T23:00:33.601 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000654', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:53.587319Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002627', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:38:18.735605Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006151', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:00:04.567958Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006434', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:40:26.940143Z'}}] 2022-09-09T23:00:33.602 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002627', 2: '00000000000000000000:00000000000000006291', 3: '00000000000000000000:00000000000000006434'} 2022-09-09T23:00:33.602 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000654', 1: '00000000000000000000:00000000000000002627', 2: '00000000000000000000:00000000000000006151', 3: '00000000000000000000:00000000000000006434'} 2022-09-09T23:00:33.603 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone2: shard=0 master=00000000000000000000:00000000000000000796 target=00000000000000000000:00000000000000000654, shard=2 master=00000000000000000000:00000000000000006291 target=00000000000000000000:00000000000000006151 2022-09-09T23:00:38.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:38.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:38.606 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:00:38.661 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:38.661 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:38.742 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T23:00:38.743 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T23:00:38.743 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T23:00:38.743 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T23:00:38.744 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T23:00:38.744 INFO:teuthology.orchestra.run.smithi005.stdout: "instance_id": 5977856695850153756 2022-09-09T23:00:38.744 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:38.744 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T23:00:38.745 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:38.745 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T23:00:38.745 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:38.745 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:38.746 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-09T23:00:38.746 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:38.747 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:38.747 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:38.747 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T23:00:24.570307Z" 2022-09-09T23:00:38.747 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:38.748 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:38.748 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:38.748 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T23:00:38.748 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:38.749 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:38.749 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000002627", 2022-09-09T23:00:38.749 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:38.749 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:38.750 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:38.750 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:38:18.735605Z" 2022-09-09T23:00:38.750 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:38.750 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:38.751 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:38.751 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T23:00:38.751 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:38.751 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:38.752 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000006291", 2022-09-09T23:00:38.752 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:38.752 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:38.752 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:38.753 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T23:00:24.580881Z" 2022-09-09T23:00:38.753 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:38.753 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:38.753 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:38.754 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T23:00:38.754 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:38.754 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:38.755 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-09T23:00:38.755 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:38.755 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:38.755 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:38.756 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:40:26.940143Z" 2022-09-09T23:00:38.756 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:38.756 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:38.756 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:00:38.757 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:38.757 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T23:00:38.757 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 0, 2022-09-09T23:00:38.757 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 0 2022-09-09T23:00:38.758 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:38.758 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T23:00:38Z" 2022-09-09T23:00:38.758 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T23:00:38.759 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5977856695850153756 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:00:24.570307Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002627", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:38:18.735605Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006291", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:00:24.580881Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006434", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:40:26.940143Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-09T23:00:38Z" } 2022-09-09T23:00:38.760 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:00:24.570307Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002627', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:38:18.735605Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006291', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:00:24.580881Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006434', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:40:26.940143Z'}}] 2022-09-09T23:00:38.761 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002627', 2: '00000000000000000000:00000000000000006291', 3: '00000000000000000000:00000000000000006434'} 2022-09-09T23:00:38.761 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002627', 2: '00000000000000000000:00000000000000006291', 3: '00000000000000000000:00000000000000006434'} 2022-09-09T23:00:38.762 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2022-09-09T23:00:38.762 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone4 2022-09-09T23:00:38.763 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone1 2022-09-09T23:00:38.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:38.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:38.764 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:00:38.816 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:38.816 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:38.903 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T23:00:38.904 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:38.904 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-09T23:00:38.904 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:57:20.177301Z" 2022-09-09T23:00:38.905 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:38.905 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:38.905 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-09T23:00:38.905 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:38:20.103388Z" 2022-09-09T23:00:38.906 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:38.906 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:38.906 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000004740", 2022-09-09T23:00:38.906 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T23:00:13.748880Z" 2022-09-09T23:00:38.907 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:38.907 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:38.907 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000005306", 2022-09-09T23:00:38.908 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:40:26.914030Z" 2022-09-09T23:00:38.908 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:38.908 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T23:00:38.909 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002909', 2: '00000000000000000000:00000000000000004740', 3: '00000000000000000000:00000000000000005306'} 2022-09-09T23:00:38.910 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2022-09-09T23:00:38.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:38.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:38.913 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:00:39.003 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:39.004 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:39.090 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T23:00:39.090 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T23:00:39.091 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T23:00:39.091 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T23:00:39.091 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T23:00:39.092 INFO:teuthology.orchestra.run.smithi005.stdout: "instance_id": 8378826292454742394 2022-09-09T23:00:39.092 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:39.092 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T23:00:39.092 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:39.093 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T23:00:39.093 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:39.093 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:39.093 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-09T23:00:39.094 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:39.094 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:39.094 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:39.094 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:57:20.177301Z" 2022-09-09T23:00:39.095 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:39.095 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:39.095 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:39.096 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T23:00:39.096 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:39.096 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:39.096 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-09T23:00:39.097 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:39.097 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:39.097 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:39.097 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:38:20.103388Z" 2022-09-09T23:00:39.098 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:39.099 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:39.099 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:39.099 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T23:00:39.099 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:39.100 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:39.100 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000004740", 2022-09-09T23:00:39.100 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:39.100 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:39.101 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:39.101 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T23:00:13.748880Z" 2022-09-09T23:00:39.101 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:39.102 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:39.102 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:39.102 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T23:00:39.103 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:00:39.103 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:00:39.104 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000005306", 2022-09-09T23:00:39.104 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:00:39.104 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:00:39.104 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:00:39.105 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:40:26.914030Z" 2022-09-09T23:00:39.105 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:39.105 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:39.105 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:00:39.106 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:39.106 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T23:00:39.107 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 0, 2022-09-09T23:00:39.107 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 0 2022-09-09T23:00:39.107 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:39.108 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T23:00:39Z" 2022-09-09T23:00:39.108 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T23:00:39.109 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 8378826292454742394 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:57:20.177301Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002909", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:38:20.103388Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004740", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:00:13.748880Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:40:26.914030Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-09T23:00:39Z" } 2022-09-09T23:00:39.110 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:57:20.177301Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:38:20.103388Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004740', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:00:13.748880Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:40:26.914030Z'}}] 2022-09-09T23:00:39.110 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002909', 2: '00000000000000000000:00000000000000004740', 3: '00000000000000000000:00000000000000005306'} 2022-09-09T23:00:39.111 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002909', 2: '00000000000000000000:00000000000000004740', 3: '00000000000000000000:00000000000000005306'} 2022-09-09T23:00:39.111 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2022-09-09T23:00:39.112 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone2 2022-09-09T23:00:39.112 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:39.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:39.114 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:00:39.160 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:39.161 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:39.256 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T23:00:39.257 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:39.257 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-09T23:00:39.258 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:57:20.177301Z" 2022-09-09T23:00:39.258 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:39.259 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:39.259 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-09T23:00:39.259 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:38:20.103388Z" 2022-09-09T23:00:39.259 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:39.260 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:39.260 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000004740", 2022-09-09T23:00:39.260 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T23:00:13.748880Z" 2022-09-09T23:00:39.260 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:39.261 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:39.261 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000005306", 2022-09-09T23:00:39.261 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:40:26.914030Z" 2022-09-09T23:00:39.261 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:39.262 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T23:00:39.263 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002909', 2: '00000000000000000000:00000000000000004740', 3: '00000000000000000000:00000000000000005306'} 2022-09-09T23:00:39.264 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2022-09-09T23:00:39.264 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:39.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:39.265 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:00:39.315 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:39.315 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:39.413 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:00:39.414 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:00:39.414 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:00:39.414 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:00:39.415 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:00:39.415 INFO:teuthology.orchestra.run.smithi080.stdout: "instance_id": 11996126732425202593 2022-09-09T23:00:39.415 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:39.415 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:00:39.416 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:39.416 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:00:39.416 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:00:39.416 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:00:39.417 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-09T23:00:39.417 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:00:39.417 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:00:39.418 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:00:39.418 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:57:20.177301Z" 2022-09-09T23:00:39.418 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:39.418 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:39.419 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:39.419 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:00:39.419 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:00:39.419 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:00:39.420 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-09T23:00:39.420 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:00:39.420 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:00:39.420 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:00:39.421 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:38:20.103388Z" 2022-09-09T23:00:39.421 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:39.421 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:39.422 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:39.422 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:00:39.422 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:00:39.422 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:00:39.423 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000004740", 2022-09-09T23:00:39.423 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:00:39.423 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:00:39.424 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:00:39.424 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:00:13.748880Z" 2022-09-09T23:00:39.424 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:39.425 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:39.425 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:39.425 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:00:39.426 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:00:39.426 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:00:39.426 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000005306", 2022-09-09T23:00:39.426 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:00:39.427 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:00:39.427 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:00:39.427 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:40:26.914030Z" 2022-09-09T23:00:39.427 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:39.428 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:39.428 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:00:39.428 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:39.428 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:00:39.429 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 0, 2022-09-09T23:00:39.429 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 0 2022-09-09T23:00:39.429 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:39.429 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:00:39Z" 2022-09-09T23:00:39.430 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:00:39.431 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11996126732425202593 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:57:20.177301Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002909", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:38:20.103388Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004740", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:00:13.748880Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:40:26.914030Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-09T23:00:39Z" } 2022-09-09T23:00:39.432 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:57:20.177301Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:38:20.103388Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004740', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:00:13.748880Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:40:26.914030Z'}}] 2022-09-09T23:00:39.432 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002909', 2: '00000000000000000000:00000000000000004740', 3: '00000000000000000000:00000000000000005306'} 2022-09-09T23:00:39.433 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002909', 2: '00000000000000000000:00000000000000004740', 3: '00000000000000000000:00000000000000005306'} 2022-09-09T23:00:39.433 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2022-09-09T23:00:39.434 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone4 2022-09-09T23:00:39.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:39.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:39.436 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:00:39.481 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:39.481 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:39.581 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T23:00:39.581 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:39.582 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-09T23:00:39.582 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:59:53.537383Z" 2022-09-09T23:00:39.583 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:39.583 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:39.583 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000002768", 2022-09-09T23:00:39.583 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:38:39.082329Z" 2022-09-09T23:00:39.584 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:39.584 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:39.584 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000004599", 2022-09-09T23:00:39.584 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T23:00:13.517424Z" 2022-09-09T23:00:39.585 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:00:39.585 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:00:39.585 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000005870", 2022-09-09T23:00:39.586 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T22:40:26.882485Z" 2022-09-09T23:00:39.586 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:00:39.586 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T23:00:39.587 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:00:39.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:00:39.589 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-09T23:00:39.630 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:39.630 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:39.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:39.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:39.728 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 0 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:00:39.774 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:39.775 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:39.856 INFO:teuthology.orchestra.run.smithi005.stdout:[] 2022-09-09T23:00:39.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:39.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:39.861 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 1 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:00:39.955 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:39.955 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:40.033 INFO:teuthology.orchestra.run.smithi005.stdout:[] 2022-09-09T23:00:40.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:40.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:40.038 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 2 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:00:40.085 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:40.086 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:40.167 INFO:teuthology.orchestra.run.smithi005.stdout:[] 2022-09-09T23:00:40.170 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:40.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:40.172 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 3 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:00:40.222 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:40.223 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:40.308 INFO:teuthology.orchestra.run.smithi005.stdout:[] 2022-09-09T23:00:40.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:40.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:40.312 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:00:40.356 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:40.357 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:40.459 INFO:teuthology.orchestra.run.smithi080.stdout:[ 2022-09-09T23:00:40.460 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:40.460 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-09T23:00:40.461 INFO:teuthology.orchestra.run.smithi080.stdout: "last_update": "2022-09-09T23:00:24.570307Z" 2022-09-09T23:00:40.461 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:40.462 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:40.462 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000002627", 2022-09-09T23:00:40.463 INFO:teuthology.orchestra.run.smithi080.stdout: "last_update": "2022-09-09T22:38:18.735605Z" 2022-09-09T23:00:40.463 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:40.463 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:40.463 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000006291", 2022-09-09T23:00:40.464 INFO:teuthology.orchestra.run.smithi080.stdout: "last_update": "2022-09-09T23:00:24.580881Z" 2022-09-09T23:00:40.464 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:40.464 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:40.464 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-09T23:00:40.465 INFO:teuthology.orchestra.run.smithi080.stdout: "last_update": "2022-09-09T22:40:26.940143Z" 2022-09-09T23:00:40.465 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:40.465 INFO:teuthology.orchestra.run.smithi080.stdout:] 2022-09-09T23:00:40.466 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:00:40.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:00:40.468 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-09T23:00:40.510 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:40.511 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:40.632 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:40.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:00:40.634 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog list --shard-id 0 --max-entries 1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:00:40.678 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:40.678 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:40.768 INFO:teuthology.orchestra.run.smithi080.stdout:[ 2022-09-09T23:00:40.769 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:40.769 INFO:teuthology.orchestra.run.smithi080.stdout: "entity_type": "bucket", 2022-09-09T23:00:40.769 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:10", 2022-09-09T23:00:40.770 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:00:24.569916Z", 2022-09-09T23:00:40.770 INFO:teuthology.orchestra.run.smithi080.stdout: "gen": 0 2022-09-09T23:00:40.770 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:40.771 INFO:teuthology.orchestra.run.smithi080.stdout:] 2022-09-09T23:00:40.774 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... FAIL 2022-09-09T23:00:40.775 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:00:40.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:00:40.777 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-09T23:00:40.859 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:40.860 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:40.909 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:00:40.903+0000 7f9656707dc0 0 failed reading obj info from .rgw.root:zone_info.4b05608c-7197-4e9b-bd29-7989607fa277: (2) No such file or directory 2022-09-09T23:00:40.910 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:00:40.903+0000 7f9656707dc0 0 WARNING: could not read zone params for zone id=4b05608c-7197-4e9b-bd29-7989607fa277 name=test-zone1 2022-09-09T23:00:40.910 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:00:40.907+0000 7f9656707dc0 0 failed reading obj info from .rgw.root:zone_info.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5: (2) No such file or directory 2022-09-09T23:00:40.911 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:00:40.907+0000 7f9656707dc0 0 WARNING: could not read zone params for zone id=79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5 name=test-zone3 2022-09-09T23:00:40.922 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:00:40.923 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T23:00:40.923 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone2", 2022-09-09T23:00:40.923 INFO:teuthology.orchestra.run.smithi080.stdout: "domain_root": "test-zone2.rgw.meta:root", 2022-09-09T23:00:40.924 INFO:teuthology.orchestra.run.smithi080.stdout: "control_pool": "test-zone2.rgw.control", 2022-09-09T23:00:40.924 INFO:teuthology.orchestra.run.smithi080.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2022-09-09T23:00:40.924 INFO:teuthology.orchestra.run.smithi080.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2022-09-09T23:00:40.925 INFO:teuthology.orchestra.run.smithi080.stdout: "log_pool": "test-zone2.rgw.log", 2022-09-09T23:00:40.925 INFO:teuthology.orchestra.run.smithi080.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2022-09-09T23:00:40.925 INFO:teuthology.orchestra.run.smithi080.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2022-09-09T23:00:40.925 INFO:teuthology.orchestra.run.smithi080.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2022-09-09T23:00:40.926 INFO:teuthology.orchestra.run.smithi080.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2022-09-09T23:00:40.926 INFO:teuthology.orchestra.run.smithi080.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2022-09-09T23:00:40.926 INFO:teuthology.orchestra.run.smithi080.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2022-09-09T23:00:40.927 INFO:teuthology.orchestra.run.smithi080.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2022-09-09T23:00:40.930 INFO:teuthology.orchestra.run.smithi080.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2022-09-09T23:00:40.930 INFO:teuthology.orchestra.run.smithi080.stdout: "otp_pool": "test-zone2.rgw.otp", 2022-09-09T23:00:40.931 INFO:teuthology.orchestra.run.smithi080.stdout: "system_key": { 2022-09-09T23:00:40.931 INFO:teuthology.orchestra.run.smithi080.stdout: "access_key": "3U0V7WQYC6MLEXED", 2022-09-09T23:00:40.931 INFO:teuthology.orchestra.run.smithi080.stdout: "secret_key": "jA4iNzYKmfzg78oKBQTy4mNpjERzasVo" 2022-09-09T23:00:40.931 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:40.932 INFO:teuthology.orchestra.run.smithi080.stdout: "placement_pools": [ 2022-09-09T23:00:40.932 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:40.932 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "default-placement", 2022-09-09T23:00:40.932 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:00:40.933 INFO:teuthology.orchestra.run.smithi080.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2022-09-09T23:00:40.933 INFO:teuthology.orchestra.run.smithi080.stdout: "storage_classes": { 2022-09-09T23:00:40.933 INFO:teuthology.orchestra.run.smithi080.stdout: "STANDARD": { 2022-09-09T23:00:40.933 INFO:teuthology.orchestra.run.smithi080.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2022-09-09T23:00:40.934 INFO:teuthology.orchestra.run.smithi080.stdout: "compression_type": "random" 2022-09-09T23:00:40.934 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:40.934 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:40.935 INFO:teuthology.orchestra.run.smithi080.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2022-09-09T23:00:40.935 INFO:teuthology.orchestra.run.smithi080.stdout: "index_type": 0 2022-09-09T23:00:40.935 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:40.935 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:40.936 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:00:40.936 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T23:00:40.936 INFO:teuthology.orchestra.run.smithi080.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2022-09-09T23:00:40.937 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:00:40.938 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:00:40.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:00:40.940 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-09T23:00:40.985 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:00:40.985 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:00:41.078 INFO:teuthology.orchestra.run.smithi080.stderr:Sending period to new master zone 4b05608c-7197-4e9b-bd29-7989607fa277 2022-09-09T23:00:41.140 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:00:41.140 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:00:41.141 INFO:teuthology.orchestra.run.smithi080.stdout: "epoch": 2, 2022-09-09T23:00:41.141 INFO:teuthology.orchestra.run.smithi080.stdout: "predecessor_uuid": "2efdc3a1-023b-4e78-aaa2-0e069a034dac", 2022-09-09T23:00:41.142 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": [ 2022-09-09T23:00:41.142 INFO:teuthology.orchestra.run.smithi080.stdout: "", 2022-09-09T23:00:41.142 INFO:teuthology.orchestra.run.smithi080.stdout: "", 2022-09-09T23:00:41.143 INFO:teuthology.orchestra.run.smithi080.stdout: "", 2022-09-09T23:00:41.143 INFO:teuthology.orchestra.run.smithi080.stdout: "1_1662764312.564852_2684.1" 2022-09-09T23:00:41.143 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:00:41.144 INFO:teuthology.orchestra.run.smithi080.stdout: "period_map": { 2022-09-09T23:00:41.144 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:00:41.144 INFO:teuthology.orchestra.run.smithi080.stdout: "zonegroups": [ 2022-09-09T23:00:41.145 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:41.145 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "test-zonegroup", 2022-09-09T23:00:41.145 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zonegroup", 2022-09-09T23:00:41.146 INFO:teuthology.orchestra.run.smithi080.stdout: "api_name": "", 2022-09-09T23:00:41.146 INFO:teuthology.orchestra.run.smithi080.stdout: "is_master": true, 2022-09-09T23:00:41.146 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T23:00:41.146 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi005.front.sepia.ceph.com:8000" 2022-09-09T23:00:41.147 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:00:41.147 INFO:teuthology.orchestra.run.smithi080.stdout: "hostnames": [], 2022-09-09T23:00:41.147 INFO:teuthology.orchestra.run.smithi080.stdout: "hostnames_s3website": [], 2022-09-09T23:00:41.147 INFO:teuthology.orchestra.run.smithi080.stdout: "master_zone": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:00:41.148 INFO:teuthology.orchestra.run.smithi080.stdout: "zones": [ 2022-09-09T23:00:41.148 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:41.148 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "3ff609e7-37d9-4b47-8369-34e6c99c4c03", 2022-09-09T23:00:41.149 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone4", 2022-09-09T23:00:41.149 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T23:00:41.149 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi080.front.sepia.ceph.com:8001" 2022-09-09T23:00:41.149 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:00:41.150 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": false, 2022-09-09T23:00:41.150 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": true, 2022-09-09T23:00:41.150 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:00:41.151 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": false, 2022-09-09T23:00:41.151 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "pubsub", 2022-09-09T23:00:41.151 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": false, 2022-09-09T23:00:41.151 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [ 2022-09-09T23:00:41.152 INFO:teuthology.orchestra.run.smithi080.stdout: "test-zone1" 2022-09-09T23:00:41.152 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:00:41.152 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "", 2022-09-09T23:00:41.153 INFO:teuthology.orchestra.run.smithi080.stdout: "supported_features": [ 2022-09-09T23:00:41.153 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T23:00:41.153 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:00:41.154 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:41.154 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:41.154 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:00:41.155 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone1", 2022-09-09T23:00:41.155 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T23:00:41.155 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi005.front.sepia.ceph.com:8000" 2022-09-09T23:00:41.156 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:00:41.156 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": false, 2022-09-09T23:00:41.156 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": true, 2022-09-09T23:00:41.156 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:00:41.157 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": false, 2022-09-09T23:00:41.157 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "", 2022-09-09T23:00:41.157 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": true, 2022-09-09T23:00:41.157 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [], 2022-09-09T23:00:41.158 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "", 2022-09-09T23:00:41.158 INFO:teuthology.orchestra.run.smithi080.stdout: "supported_features": [ 2022-09-09T23:00:41.158 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T23:00:41.159 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:00:41.159 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:41.159 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:41.159 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5", 2022-09-09T23:00:41.160 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone3", 2022-09-09T23:00:41.160 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T23:00:41.160 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi005.front.sepia.ceph.com:8001" 2022-09-09T23:00:41.161 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:00:41.161 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": false, 2022-09-09T23:00:41.161 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": true, 2022-09-09T23:00:41.161 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:00:41.162 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": false, 2022-09-09T23:00:41.162 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "", 2022-09-09T23:00:41.162 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": true, 2022-09-09T23:00:41.162 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [], 2022-09-09T23:00:41.163 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "", 2022-09-09T23:00:41.163 INFO:teuthology.orchestra.run.smithi080.stdout: "supported_features": [ 2022-09-09T23:00:41.164 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T23:00:41.164 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:00:41.165 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:41.165 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:41.166 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T23:00:41.166 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone2", 2022-09-09T23:00:41.166 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T23:00:41.167 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi080.front.sepia.ceph.com:8000" 2022-09-09T23:00:41.167 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:00:41.168 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": false, 2022-09-09T23:00:41.168 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": true, 2022-09-09T23:00:41.168 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:00:41.169 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": false, 2022-09-09T23:00:41.169 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "", 2022-09-09T23:00:41.169 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": false, 2022-09-09T23:00:41.170 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [], 2022-09-09T23:00:41.170 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "", 2022-09-09T23:00:41.171 INFO:teuthology.orchestra.run.smithi080.stdout: "supported_features": [ 2022-09-09T23:00:41.171 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T23:00:41.171 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:00:41.172 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:41.172 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:00:41.173 INFO:teuthology.orchestra.run.smithi080.stdout: "placement_targets": [ 2022-09-09T23:00:41.173 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:41.173 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "default-placement", 2022-09-09T23:00:41.175 INFO:teuthology.orchestra.run.smithi080.stdout: "tags": [], 2022-09-09T23:00:41.176 INFO:teuthology.orchestra.run.smithi080.stdout: "storage_classes": [ 2022-09-09T23:00:41.176 INFO:teuthology.orchestra.run.smithi080.stdout: "STANDARD" 2022-09-09T23:00:41.177 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:00:41.177 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:41.178 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:00:41.178 INFO:teuthology.orchestra.run.smithi080.stdout: "default_placement": "default-placement", 2022-09-09T23:00:41.178 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T23:00:41.179 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_policy": { 2022-09-09T23:00:41.179 INFO:teuthology.orchestra.run.smithi080.stdout: "groups": [] 2022-09-09T23:00:41.180 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:41.180 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled_features": [ 2022-09-09T23:00:41.180 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T23:00:41.181 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:00:41.181 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:41.181 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:00:41.181 INFO:teuthology.orchestra.run.smithi080.stdout: "short_zone_ids": [ 2022-09-09T23:00:41.182 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:41.182 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "3ff609e7-37d9-4b47-8369-34e6c99c4c03", 2022-09-09T23:00:41.182 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 2593176765 2022-09-09T23:00:41.183 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:41.183 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:41.183 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:00:41.184 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 1301724575 2022-09-09T23:00:41.184 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:41.184 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:41.185 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5", 2022-09-09T23:00:41.185 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 2787153217 2022-09-09T23:00:41.185 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:41.185 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:00:41.186 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T23:00:41.186 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 1899625128 2022-09-09T23:00:41.187 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:41.187 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:00:41.187 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:41.188 INFO:teuthology.orchestra.run.smithi080.stdout: "master_zonegroup": "test-zonegroup", 2022-09-09T23:00:41.188 INFO:teuthology.orchestra.run.smithi080.stdout: "master_zone": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:00:41.188 INFO:teuthology.orchestra.run.smithi080.stdout: "period_config": { 2022-09-09T23:00:41.189 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_quota": { 2022-09-09T23:00:41.189 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false, 2022-09-09T23:00:41.189 INFO:teuthology.orchestra.run.smithi080.stdout: "check_on_raw": false, 2022-09-09T23:00:41.189 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size": -1, 2022-09-09T23:00:41.190 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size_kb": 0, 2022-09-09T23:00:41.190 INFO:teuthology.orchestra.run.smithi080.stdout: "max_objects": -1 2022-09-09T23:00:41.190 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:41.191 INFO:teuthology.orchestra.run.smithi080.stdout: "user_quota": { 2022-09-09T23:00:41.191 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false, 2022-09-09T23:00:41.191 INFO:teuthology.orchestra.run.smithi080.stdout: "check_on_raw": false, 2022-09-09T23:00:41.191 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size": -1, 2022-09-09T23:00:41.192 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size_kb": 0, 2022-09-09T23:00:41.192 INFO:teuthology.orchestra.run.smithi080.stdout: "max_objects": -1 2022-09-09T23:00:41.192 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:41.192 INFO:teuthology.orchestra.run.smithi080.stdout: "user_ratelimit": { 2022-09-09T23:00:41.193 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_ops": 0, 2022-09-09T23:00:41.193 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_ops": 0, 2022-09-09T23:00:41.193 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_bytes": 0, 2022-09-09T23:00:41.193 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_bytes": 0, 2022-09-09T23:00:41.194 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false 2022-09-09T23:00:41.194 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:41.194 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_ratelimit": { 2022-09-09T23:00:41.194 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_ops": 0, 2022-09-09T23:00:41.195 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_ops": 0, 2022-09-09T23:00:41.195 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_bytes": 0, 2022-09-09T23:00:41.195 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_bytes": 0, 2022-09-09T23:00:41.196 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false 2022-09-09T23:00:41.196 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:41.196 INFO:teuthology.orchestra.run.smithi080.stdout: "anonymous_ratelimit": { 2022-09-09T23:00:41.196 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_ops": 0, 2022-09-09T23:00:41.197 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_ops": 0, 2022-09-09T23:00:41.197 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_bytes": 0, 2022-09-09T23:00:41.197 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_bytes": 0, 2022-09-09T23:00:41.198 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false 2022-09-09T23:00:41.198 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:00:41.199 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:00:41.199 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T23:00:41.199 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_name": "test-realm", 2022-09-09T23:00:41.199 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:00:41.200 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:00:41.200 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2022-09-09T23:01:41.202 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jkcgbj-49 2022-09-09T23:01:41.204 DEBUG:boto:path=/jkcgbj-49/ 2022-09-09T23:01:41.205 DEBUG:boto:auth_path=/jkcgbj-49/ 2022-09-09T23:01:41.206 DEBUG:boto:Method: PUT 2022-09-09T23:01:41.206 DEBUG:boto:Path: /jkcgbj-49/ 2022-09-09T23:01:41.207 DEBUG:boto:Data: 2022-09-09T23:01:41.208 DEBUG:boto:Headers: {} 2022-09-09T23:01:41.208 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T23:01:41.209 DEBUG:boto:Port: 8000 2022-09-09T23:01:41.209 DEBUG:boto:Params: {} 2022-09-09T23:01:41.210 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:01:41.211 DEBUG:boto:Token: None 2022-09-09T23:01:41.212 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:01:41 GMT /jkcgbj-49/ 2022-09-09T23:01:41.213 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:spBEJJWE9ADg/PWCZihSCkNfhjg= 2022-09-09T23:01:41.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:01:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:spBEJJWE9ADg/PWCZihSCkNfhjg=', 'Content-Length': '0'} 2022-09-09T23:01:41.251 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000198863122c1f76b0-00631bc5d5-14f9-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:01:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:01:41.253 DEBUG:boto:path=/jkcgbj-49/testredirect 2022-09-09T23:01:41.254 DEBUG:boto:auth_path=/jkcgbj-49/testredirect 2022-09-09T23:01:41.255 DEBUG:boto:Method: PUT 2022-09-09T23:01:41.256 DEBUG:boto:Path: /jkcgbj-49/testredirect 2022-09-09T23:01:41.256 DEBUG:boto:Data: 2022-09-09T23:01:41.257 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2022-09-09T23:01:41.257 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T23:01:41.258 DEBUG:boto:Port: 8000 2022-09-09T23:01:41.258 DEBUG:boto:Params: {} 2022-09-09T23:01:41.259 DEBUG:boto:Token: None 2022-09-09T23:01:41.259 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Fri, 09 Sep 2022 23:01:41 GMT /jkcgbj-49/testredirect 2022-09-09T23:01:41.260 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:/wxh5DlWDl+J0bGES8+8ZINWQJ0= 2022-09-09T23:01:41.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 23:01:41 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:/wxh5DlWDl+J0bGES8+8ZINWQJ0='} 2022-09-09T23:01:41.277 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000be6e68da8b9530d4-00631bc5d5-14f9-test-zone1'), ('Date', 'Fri, 09 Sep 2022 23:01:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:01:41.278 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:01:41.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:01:41.280 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:01:41.330 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:01:41.331 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:01:41.385 INFO:teuthology.orchestra.run.smithi005.stderr:No --period given, using current period=fd4bc158-06fd-4840-8a27-33fabda67b99 2022-09-09T23:01:41.411 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T23:01:41.411 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:01:41.412 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T23:01:41.412 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T23:01:41.412 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:01:41.413 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:01:41.413 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T23:01:41.413 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T23:01:41.413 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:01:41.414 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:01:41.414 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T23:01:41.414 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T23:01:41.415 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:01:41.415 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:01:41.415 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662764501.249520_420.1", 2022-09-09T23:01:41.416 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T23:01:41.249520Z" 2022-09-09T23:01:41.416 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:01:41.416 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T23:01:41.418 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:01:41.418 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-09T23:01:41.419 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:01:41.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:01:41.420 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:01:41.467 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:01:41.468 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:01:41.540 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:01:41.538+0000 7f1371736dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:01:41.542 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:01:41.542+0000 7f1371736dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:01:41.578 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:01:41.579 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:01:41.579 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:01:41.579 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:01:41.579 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:01:41.580 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:01:41.580 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:01:41.580 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:01:41.581 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:01:41.581 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:01:41.581 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:01:41.581 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:01:41.582 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:01:41.582 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:01:41.582 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:01:41.583 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:01:41.583 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:01:41.583 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:01:41.584 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:01:41.584 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:01:41.584 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:01:41.584 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:01:41.585 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:01:41.585 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:01:41.585 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:01:41.585 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:01:41.586 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:01:41.586 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:01:41.586 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:01:41.586 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:01:41.587 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:01:41.587 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:01:41.587 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:01:41.588 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:01:41.588 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:01:41.588 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:01:41.589 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:01:41.589 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:01:41.589 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:01:41.589 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:01:41.590 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:01:41.590 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:01:41.590 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:01:41.591 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:01:41.591 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:01:41.592 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:01:41.592 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:01:41.592 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:01:41.593 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:01:41.593 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764394.995549_383.1", 2022-09-09T23:01:41.594 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:01:41.594 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:01:41.594 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:01:41.594 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:54.995549Z", 2022-09-09T23:01:41.595 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:01:41.595 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:01:41.595 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:01:41.595 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:01:41.596 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:01:41.596 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:01:41.596 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:01:41.597 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:01:41.597 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:01:41.597 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:01:41Z" 2022-09-09T23:01:41.597 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:01:41.599 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764394.995549_383.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:54.995549Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:01:41Z" } 2022-09-09T23:01:41.599 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764394.995549_383.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:54.995549Z', 'realm_epoch': 4}}] 2022-09-09T23:01:41.600 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:01:41.601 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764394.995549_383.1'} 2022-09-09T23:01:41.601 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764501.249520_420.1 target=1_1662764394.995549_383.1 2022-09-09T23:01:46.603 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:01:46.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:01:46.605 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:01:46.653 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:01:46.654 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:01:46.713 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:01:46.714+0000 7f76f719adc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:01:46.715 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:01:46.714+0000 7f76f719adc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:01:46.744 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:01:46.745 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:01:46.745 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:01:46.745 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:01:46.746 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:01:46.746 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:01:46.746 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:01:46.747 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:01:46.747 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:01:46.747 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:01:46.747 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:01:46.748 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:01:46.748 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:01:46.748 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:01:46.748 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:01:46.749 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:01:46.749 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:01:46.749 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:01:46.750 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:01:46.750 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:01:46.750 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:01:46.750 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:01:46.751 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:01:46.751 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:01:46.751 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:01:46.751 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:01:46.752 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:01:46.752 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:01:46.753 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:01:46.753 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:01:46.754 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:01:46.754 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:01:46.755 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:01:46.755 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:01:46.756 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:01:46.756 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:01:46.757 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:01:46.758 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:01:46.758 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:01:46.759 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:01:46.759 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:01:46.760 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:01:46.760 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:01:46.761 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:01:46.761 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:01:46.762 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:01:46.762 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:01:46.763 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:01:46.763 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:01:46.764 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764394.995549_383.1", 2022-09-09T23:01:46.764 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:01:46.765 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:01:46.765 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:01:46.765 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:54.995549Z", 2022-09-09T23:01:46.766 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:01:46.766 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:01:46.767 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:01:46.767 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:01:46.767 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:01:46.768 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:01:46.768 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:01:46.768 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:01:46.769 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:01:46.769 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:01:46Z" 2022-09-09T23:01:46.769 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:01:46.771 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764394.995549_383.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:54.995549Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:01:46Z" } 2022-09-09T23:01:46.772 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764394.995549_383.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:54.995549Z', 'realm_epoch': 4}}] 2022-09-09T23:01:46.772 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:01:46.773 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764394.995549_383.1'} 2022-09-09T23:01:46.773 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764501.249520_420.1 target=1_1662764394.995549_383.1 2022-09-09T23:01:51.775 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:01:51.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:01:51.777 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:01:51.825 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:01:51.825 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:01:51.889 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:01:51.890+0000 7f77275f6dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:01:51.891 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:01:51.890+0000 7f77275f6dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:01:51.920 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:01:51.920 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:01:51.921 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:01:51.921 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:01:51.922 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:01:51.922 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:01:51.923 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:01:51.923 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:01:51.924 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:01:51.924 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:01:51.925 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:01:51.925 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:01:51.925 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:01:51.926 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:01:51.926 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:01:51.926 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:01:51.926 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:01:51.927 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:01:51.927 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:01:51.927 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:01:51.928 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:01:51.928 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:01:51.928 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:01:51.928 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:01:51.929 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:01:51.929 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:01:51.929 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:01:51.929 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:01:51.930 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:01:51.930 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:01:51.930 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:01:51.931 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:01:51.931 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:01:51.932 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:01:51.932 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:01:51.933 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:01:51.933 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:01:51.934 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:01:51.934 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:01:51.934 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:01:51.935 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:01:51.935 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:01:51.935 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:01:51.936 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:01:51.936 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:01:51.936 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:01:51.936 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:01:51.937 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:01:51.937 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:01:51.937 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764394.995549_383.1", 2022-09-09T23:01:51.938 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:01:51.938 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:01:51.939 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:01:51.939 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:54.995549Z", 2022-09-09T23:01:51.939 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:01:51.939 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:01:51.940 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:01:51.940 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:01:51.941 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:01:51.941 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:01:51.942 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:01:51.942 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:01:51.943 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:01:51.943 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:01:51Z" 2022-09-09T23:01:51.944 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:01:51.945 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764394.995549_383.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:54.995549Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:01:51Z" } 2022-09-09T23:01:51.946 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764394.995549_383.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:54.995549Z', 'realm_epoch': 4}}] 2022-09-09T23:01:51.947 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:01:51.947 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764394.995549_383.1'} 2022-09-09T23:01:51.947 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764501.249520_420.1 target=1_1662764394.995549_383.1 2022-09-09T23:01:56.948 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:01:56.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:01:56.950 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:01:56.997 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:01:56.998 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:01:57.077 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:01:57.074+0000 7f042bb2ddc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:01:57.079 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:01:57.078+0000 7f042bb2ddc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:01:57.114 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:01:57.114 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:01:57.115 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:01:57.115 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:01:57.115 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:01:57.116 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:01:57.116 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:01:57.117 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:01:57.117 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:01:57.118 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:01:57.118 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:01:57.118 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:01:57.119 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:01:57.119 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:01:57.119 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:01:57.120 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:01:57.120 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:01:57.120 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:01:57.120 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:01:57.121 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:01:57.121 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:01:57.122 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:01:57.122 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:01:57.122 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:01:57.122 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:01:57.123 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:01:57.123 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:01:57.123 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:01:57.123 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:01:57.124 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:01:57.124 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:01:57.124 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:01:57.124 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:01:57.125 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:01:57.125 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:01:57.125 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:01:57.125 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:01:57.126 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:01:57.126 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:01:57.126 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:01:57.127 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:01:57.127 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:01:57.127 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:01:57.128 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:01:57.128 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:01:57.128 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:01:57.129 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:01:57.129 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:01:57.130 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:01:57.130 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764394.995549_383.1", 2022-09-09T23:01:57.131 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:01:57.131 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:01:57.131 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:01:57.132 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:54.995549Z", 2022-09-09T23:01:57.132 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:01:57.133 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:01:57.133 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:01:57.133 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:01:57.133 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:01:57.134 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:01:57.134 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:01:57.134 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:01:57.134 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:01:57.135 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:01:57Z" 2022-09-09T23:01:57.135 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:01:57.136 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764394.995549_383.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:54.995549Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:01:57Z" } 2022-09-09T23:01:57.137 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764394.995549_383.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:54.995549Z', 'realm_epoch': 4}}] 2022-09-09T23:01:57.137 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:01:57.138 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764394.995549_383.1'} 2022-09-09T23:01:57.138 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764501.249520_420.1 target=1_1662764394.995549_383.1 2022-09-09T23:02:02.140 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:02:02.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:02:02.141 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:02:02.191 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:02:02.191 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:02:02.256 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:02:02.253+0000 7fe075256dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:02:02.258 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:02:02.257+0000 7fe075256dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:02:02.287 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:02:02.287 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:02:02.288 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:02:02.288 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:02:02.288 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:02:02.289 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:02:02.289 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:02:02.289 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:02.289 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:02:02.290 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:02.290 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:02:02.290 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:02.290 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:02.291 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:02:02.291 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:02.291 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:02:02.291 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:02.292 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:02:02.292 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:02:02.292 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:02.292 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:02.293 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:02.293 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:02:02.293 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:02.293 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:02.294 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:02:02.294 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:02.294 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:02:02.295 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:02.295 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:02:02.295 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:02:02.295 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:02.296 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:02.296 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:02.296 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:02:02.297 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:02.297 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:02.297 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:02:02.297 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:02.298 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:02:02.298 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:02.298 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:02:02.298 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:02:02.299 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:02.299 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:02.299 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:02.299 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:02:02.300 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:02.300 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:02.300 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764394.995549_383.1", 2022-09-09T23:02:02.300 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:02.301 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:02:02.301 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:02.301 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:54.995549Z", 2022-09-09T23:02:02.301 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:02:02.302 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:02.302 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:02.302 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:02:02.302 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:02.303 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:02:02.303 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:02:02.303 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:02:02.303 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:02.304 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:02:02Z" 2022-09-09T23:02:02.304 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:02:02.305 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764394.995549_383.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:54.995549Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:02:02Z" } 2022-09-09T23:02:02.306 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764394.995549_383.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:54.995549Z', 'realm_epoch': 4}}] 2022-09-09T23:02:02.306 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:02:02.307 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764394.995549_383.1'} 2022-09-09T23:02:02.307 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764501.249520_420.1 target=1_1662764394.995549_383.1 2022-09-09T23:02:07.309 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:02:07.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:02:07.311 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:02:07.358 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:02:07.358 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:02:07.415 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:02:07.413+0000 7f5a7fdf8dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:02:07.417 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:02:07.413+0000 7f5a7fdf8dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:02:07.447 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:02:07.448 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:02:07.448 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:02:07.448 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:02:07.449 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:02:07.449 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:02:07.449 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:02:07.450 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:07.450 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:02:07.450 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:07.451 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:02:07.451 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:07.451 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:07.451 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:02:07.452 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:07.452 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:02:07.452 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:07.453 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:02:07.454 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:02:07.454 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:07.455 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:07.455 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:07.456 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:02:07.456 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:07.457 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:07.457 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:02:07.458 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:07.458 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:02:07.458 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:07.459 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:02:07.459 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:02:07.460 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:07.460 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:07.461 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:07.461 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:02:07.462 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:07.462 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:07.463 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:02:07.463 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:07.464 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:02:07.464 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:07.465 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:02:07.465 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:02:07.466 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:07.466 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:07.467 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:07.467 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:02:07.467 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:07.468 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:07.468 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764394.995549_383.1", 2022-09-09T23:02:07.469 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:07.469 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:02:07.469 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:07.470 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:54.995549Z", 2022-09-09T23:02:07.470 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:02:07.471 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:07.471 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:07.472 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:02:07.472 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:07.473 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:02:07.473 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:02:07.474 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:02:07.474 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:07.475 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:02:07Z" 2022-09-09T23:02:07.475 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:02:07.476 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764394.995549_383.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:54.995549Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:02:07Z" } 2022-09-09T23:02:07.477 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764394.995549_383.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:54.995549Z', 'realm_epoch': 4}}] 2022-09-09T23:02:07.477 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:02:07.478 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764394.995549_383.1'} 2022-09-09T23:02:07.479 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764501.249520_420.1 target=1_1662764394.995549_383.1 2022-09-09T23:02:12.480 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:02:12.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:02:12.481 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:02:12.531 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:02:12.531 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:02:12.600 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:02:12.597+0000 7fbcca283dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:02:12.602 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:02:12.601+0000 7fbcca283dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:02:12.630 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:02:12.631 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:02:12.631 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:02:12.632 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:02:12.632 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:02:12.633 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:02:12.633 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:02:12.634 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:12.634 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:02:12.634 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:12.635 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:02:12.635 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:12.636 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:12.636 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:02:12.636 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:12.637 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:02:12.637 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:12.637 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:02:12.638 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:02:12.638 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:12.639 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:12.639 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:12.639 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:02:12.639 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:12.640 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:12.640 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:02:12.641 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:12.641 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:02:12.642 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:12.642 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:02:12.642 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:02:12.643 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:12.643 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:12.643 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:12.643 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:02:12.644 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:12.644 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:12.644 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:02:12.644 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:12.644 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:02:12.645 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:12.645 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:02:12.645 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:02:12.645 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:12.646 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:12.646 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:12.646 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:02:12.646 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:12.647 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:12.647 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764394.995549_383.1", 2022-09-09T23:02:12.647 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:12.647 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:02:12.648 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:12.648 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:54.995549Z", 2022-09-09T23:02:12.648 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:02:12.648 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:12.649 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:12.649 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:02:12.649 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:12.649 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:02:12.650 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:02:12.650 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:02:12.650 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:12.650 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:02:12Z" 2022-09-09T23:02:12.651 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:02:12.652 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764394.995549_383.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:54.995549Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:02:12Z" } 2022-09-09T23:02:12.653 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764394.995549_383.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:54.995549Z', 'realm_epoch': 4}}] 2022-09-09T23:02:12.654 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:02:12.655 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764394.995549_383.1'} 2022-09-09T23:02:12.655 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764501.249520_420.1 target=1_1662764394.995549_383.1 2022-09-09T23:02:17.656 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:02:17.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:02:17.659 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:02:17.710 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:02:17.710 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:02:17.765 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:02:17.761+0000 7f4cbb212dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:02:17.766 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:02:17.765+0000 7f4cbb212dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:02:17.795 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:02:17.796 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:02:17.797 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:02:17.797 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:02:17.798 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:02:17.798 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:02:17.799 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:02:17.799 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:17.799 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:02:17.802 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:17.803 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:02:17.803 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:17.804 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:17.804 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:02:17.805 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:17.805 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:02:17.806 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:17.807 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:02:17.807 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:02:17.808 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:17.808 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:17.809 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:17.809 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:02:17.810 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:17.810 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:17.811 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:02:17.811 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:17.812 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:02:17.812 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:17.812 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:02:17.812 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:02:17.813 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:17.813 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:17.813 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:17.813 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:02:17.814 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:17.814 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:17.814 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:02:17.815 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:17.815 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:02:17.815 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:17.815 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:02:17.816 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:02:17.816 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:17.816 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:17.816 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:17.817 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:02:17.817 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:17.817 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:17.817 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764394.995549_383.1", 2022-09-09T23:02:17.818 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:17.818 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:02:17.818 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:17.819 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:54.995549Z", 2022-09-09T23:02:17.819 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:02:17.819 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:17.819 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:17.820 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:02:17.820 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:17.820 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:02:17.821 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:02:17.821 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:02:17.821 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:17.821 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:02:17Z" 2022-09-09T23:02:17.822 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:02:17.824 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764394.995549_383.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:54.995549Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:02:17Z" } 2022-09-09T23:02:17.825 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764394.995549_383.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:54.995549Z', 'realm_epoch': 4}}] 2022-09-09T23:02:17.826 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:02:17.827 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764394.995549_383.1'} 2022-09-09T23:02:17.828 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764501.249520_420.1 target=1_1662764394.995549_383.1 2022-09-09T23:02:22.829 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:02:22.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:02:22.831 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:02:22.879 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:02:22.880 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:02:22.949 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:02:22.949+0000 7fcf81f86dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:02:22.951 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:02:22.949+0000 7fcf81f86dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:02:22.987 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:02:22.988 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:02:22.988 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:02:22.989 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:02:22.989 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:02:22.989 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:02:22.990 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:02:22.990 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:22.991 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:02:22.991 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:22.992 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:02:22.992 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:22.993 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:22.993 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:02:22.994 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:22.994 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:02:22.995 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:22.995 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:02:22.996 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:02:22.996 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:22.997 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:22.997 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:22.997 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:02:22.998 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:22.998 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:22.998 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:02:22.998 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:22.999 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:02:22.999 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:22.999 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:02:22.999 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:02:23.000 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:23.000 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:23.000 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:23.000 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:02:23.001 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:23.001 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:23.001 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:02:23.001 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:23.002 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:02:23.002 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:23.002 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:02:23.003 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:02:23.003 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:23.003 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:23.003 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:23.004 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:02:23.004 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:23.004 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:23.005 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764394.995549_383.1", 2022-09-09T23:02:23.005 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:23.005 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:02:23.006 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:23.006 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:54.995549Z", 2022-09-09T23:02:23.006 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:02:23.006 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:23.007 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:23.007 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:02:23.007 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:23.008 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:02:23.008 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:02:23.008 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:02:23.009 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:23.009 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:02:22Z" 2022-09-09T23:02:23.009 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:02:23.010 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764394.995549_383.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:54.995549Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:02:22Z" } 2022-09-09T23:02:23.011 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764394.995549_383.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:54.995549Z', 'realm_epoch': 4}}] 2022-09-09T23:02:23.012 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:02:23.012 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764394.995549_383.1'} 2022-09-09T23:02:23.013 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764501.249520_420.1 target=1_1662764394.995549_383.1 2022-09-09T23:02:28.014 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:02:28.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:02:28.016 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:02:28.062 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:02:28.062 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:02:28.117 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:02:28.117+0000 7fb3f5bb7dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:02:28.119 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:02:28.117+0000 7fb3f5bb7dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:02:28.147 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:02:28.148 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:02:28.148 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:02:28.148 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:02:28.149 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:02:28.149 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:02:28.149 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:02:28.149 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:28.150 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:02:28.150 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:28.151 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:02:28.151 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:28.152 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:28.152 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:02:28.152 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:28.152 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:02:28.153 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:28.153 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:02:28.153 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:02:28.153 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:28.154 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:28.154 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:28.154 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:02:28.154 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:28.155 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:28.155 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:02:28.156 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:28.156 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:02:28.156 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:28.156 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:02:28.157 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:02:28.157 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:28.157 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:28.157 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:28.158 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:02:28.158 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:28.158 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:28.159 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:02:28.159 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:28.159 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:02:28.159 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:28.160 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:02:28.160 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:02:28.160 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:28.161 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:28.161 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:28.162 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:02:28.162 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:28.162 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:28.162 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764394.995549_383.1", 2022-09-09T23:02:28.163 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:28.163 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:02:28.163 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:28.164 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:54.995549Z", 2022-09-09T23:02:28.164 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:02:28.164 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:28.164 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:28.165 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:02:28.165 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:28.165 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:02:28.166 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:02:28.166 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:02:28.167 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:28.167 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:02:28Z" 2022-09-09T23:02:28.167 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:02:28.169 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764394.995549_383.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:54.995549Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:02:28Z" } 2022-09-09T23:02:28.169 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764394.995549_383.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:54.995549Z', 'realm_epoch': 4}}] 2022-09-09T23:02:28.170 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:02:28.171 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764394.995549_383.1'} 2022-09-09T23:02:28.171 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764501.249520_420.1 target=1_1662764394.995549_383.1 2022-09-09T23:02:33.172 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:02:33.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:02:33.174 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:02:33.222 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:02:33.223 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:02:33.283 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:02:33.281+0000 7fc608093dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:02:33.284 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:02:33.281+0000 7fc608093dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:02:33.316 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:02:33.316 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:02:33.316 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:02:33.316 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:02:33.317 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:02:33.317 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:02:33.317 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:02:33.317 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:33.318 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:02:33.318 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:33.319 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:02:33.319 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:33.319 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:33.319 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:02:33.320 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:33.320 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:02:33.320 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:33.320 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:02:33.321 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:02:33.321 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:33.321 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:33.322 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:33.323 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:02:33.323 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:33.323 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:33.324 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:02:33.324 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:33.326 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:02:33.326 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:33.326 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:02:33.327 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:02:33.327 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:33.327 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:33.328 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:33.328 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:02:33.328 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:33.328 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:33.329 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:02:33.329 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:33.329 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:02:33.329 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:33.330 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:02:33.330 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:02:33.330 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:33.331 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:33.331 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:33.331 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:02:33.331 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:33.332 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:33.332 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764394.995549_383.1", 2022-09-09T23:02:33.332 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:33.332 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:02:33.333 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:33.333 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:54.995549Z", 2022-09-09T23:02:33.333 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:02:33.333 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:33.334 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:33.334 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:02:33.334 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:33.334 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:02:33.335 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:02:33.335 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:02:33.335 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:33.336 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:02:33Z" 2022-09-09T23:02:33.336 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:02:33.338 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764394.995549_383.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:54.995549Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:02:33Z" } 2022-09-09T23:02:33.339 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764394.995549_383.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:54.995549Z', 'realm_epoch': 4}}] 2022-09-09T23:02:33.340 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:02:33.341 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764394.995549_383.1'} 2022-09-09T23:02:33.342 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764501.249520_420.1 target=1_1662764394.995549_383.1 2022-09-09T23:02:38.344 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:02:38.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:02:38.345 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:02:38.393 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:02:38.394 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:02:38.465 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:02:38.465+0000 7f1d69889dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:02:38.468 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:02:38.465+0000 7f1d69889dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:02:38.503 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:02:38.503 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:02:38.503 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:02:38.504 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:02:38.504 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:02:38.504 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:02:38.504 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:02:38.505 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:38.505 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:02:38.505 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:38.505 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:02:38.505 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:38.506 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:38.506 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:02:38.506 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:38.507 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:02:38.507 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:38.507 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:02:38.507 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:02:38.508 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:38.508 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:38.508 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:38.509 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:02:38.509 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:38.509 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:38.509 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:02:38.510 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:38.510 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:02:38.510 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:38.511 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:02:38.511 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:02:38.511 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:38.512 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:38.512 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:38.512 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:02:38.513 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:38.513 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:38.513 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:02:38.513 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:38.514 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:02:38.514 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:38.514 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:02:38.514 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:02:38.515 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:38.515 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:38.515 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:38.515 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:02:38.516 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:38.516 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:38.516 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764394.995549_383.1", 2022-09-09T23:02:38.516 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:38.517 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:02:38.517 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:38.517 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:54.995549Z", 2022-09-09T23:02:38.517 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:02:38.518 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:38.518 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:38.518 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:02:38.518 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:38.519 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:02:38.519 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:02:38.519 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:02:38.519 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:38.520 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:02:38Z" 2022-09-09T23:02:38.520 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:02:38.521 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764394.995549_383.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:54.995549Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:02:38Z" } 2022-09-09T23:02:38.522 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764394.995549_383.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:54.995549Z', 'realm_epoch': 4}}] 2022-09-09T23:02:38.523 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:02:38.523 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764394.995549_383.1'} 2022-09-09T23:02:38.524 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764501.249520_420.1 target=1_1662764394.995549_383.1 2022-09-09T23:02:43.525 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:02:43.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:02:43.527 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:02:43.574 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:02:43.574 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:02:43.641 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:02:43.640+0000 7f09a7f4ddc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:02:43.643 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:02:43.640+0000 7f09a7f4ddc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:02:43.671 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:02:43.671 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:02:43.672 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:02:43.672 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:02:43.672 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:02:43.672 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:02:43.673 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:02:43.673 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:43.673 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:02:43.674 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:43.674 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:02:43.674 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:43.674 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:43.675 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:02:43.675 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:43.675 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:02:43.675 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:43.676 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:02:43.676 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:02:43.676 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:43.677 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:43.677 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:43.677 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:02:43.677 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:43.678 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:43.678 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:02:43.679 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:43.679 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:02:43.679 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:43.680 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:02:43.680 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:02:43.680 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:43.681 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:43.681 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:43.681 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:02:43.681 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:43.682 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:43.682 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:02:43.682 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:43.682 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:02:43.683 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:43.683 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:02:43.683 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:02:43.683 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:43.684 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:43.684 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:43.684 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:02:43.684 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:43.685 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:43.685 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764501.249520_420.1", 2022-09-09T23:02:43.685 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:43.685 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:02:43.686 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:43.686 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:01:41.249520Z", 2022-09-09T23:02:43.686 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:02:43.686 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:43.687 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:43.687 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:02:43.687 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:43.687 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:02:43.688 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:02:43.688 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:02:43.688 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:43.688 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:02:43Z" 2022-09-09T23:02:43.689 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:02:43.690 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764501.249520_420.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:01:41.249520Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:02:43Z" } 2022-09-09T23:02:43.691 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764501.249520_420.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:01:41.249520Z', 'realm_epoch': 4}}] 2022-09-09T23:02:43.691 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:02:43.692 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:02:43.692 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-09T23:02:43.693 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-09T23:02:43.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:02:43.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:02:43.694 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:02:43.746 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:02:43.746 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:02:43.796 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T23:02:43.796+0000 7fc262925dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:02:43.798 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T23:02:43.796+0000 7fc262925dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:02:43.824 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T23:02:43.824 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T23:02:43.825 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T23:02:43.825 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T23:02:43.825 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T23:02:43.825 INFO:teuthology.orchestra.run.smithi005.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:02:43.826 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 4 2022-09-09T23:02:43.826 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:02:43.826 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T23:02:43.826 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:02:43.827 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T23:02:43.827 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:02:43.827 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T23:02:43.827 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:02:43.828 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:02:43.828 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 1, 2022-09-09T23:02:43.828 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:02:43.828 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:02:43.829 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T23:02:43.829 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:02:43.829 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:02:43.829 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:02:43.830 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T23:02:43.830 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:02:43.830 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T23:02:43.830 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T23:02:43.831 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:02:43.831 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:02:43.831 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:02:43.831 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T23:02:43.832 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T23:02:43.832 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:02:43.832 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:02:43.832 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:02:43.833 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T23:02:43.833 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:02:43.833 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T23:02:43.833 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:02:43.834 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:02:43.834 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:02:43.834 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:02:43.834 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:02:43.835 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T23:02:43.835 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:02:43.835 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:02:43.835 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:02:43.836 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T23:02:43.836 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:02:43.836 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T23:02:43.836 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662764501.249520_420.1", 2022-09-09T23:02:43.837 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:02:43.837 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:02:43.837 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:02:43.837 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T23:01:41.249520Z", 2022-09-09T23:02:43.838 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 4 2022-09-09T23:02:43.838 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:02:43.838 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:02:43.839 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:02:43.839 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:02:43.839 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T23:02:43.839 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 1, 2022-09-09T23:02:43.840 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 1 2022-09-09T23:02:43.840 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:02:43.840 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T23:02:43Z" 2022-09-09T23:02:43.840 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T23:02:43.842 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764501.249520_420.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:01:41.249520Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:02:43Z" } 2022-09-09T23:02:43.842 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764501.249520_420.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:01:41.249520Z', 'realm_epoch': 4}}] 2022-09-09T23:02:43.843 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:02:43.843 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:02:43.844 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-09T23:02:43.844 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-09T23:02:43.845 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:02:43.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:02:43.846 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:02:43.891 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:02:43.892 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:02:43.952 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:02:43.948+0000 7fd1fa562dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:02:43.955 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:02:43.952+0000 7fd1fa562dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:02:43.987 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:02:43.987 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:02:43.987 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:02:43.988 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:02:43.988 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:02:43.988 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:02:43.989 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:02:43.989 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:43.990 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:02:43.990 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:43.990 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:02:43.990 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:43.991 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:43.991 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:02:43.991 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:43.991 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:02:43.992 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:43.992 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:02:43.992 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:02:43.993 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:43.993 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:43.993 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:43.994 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:02:43.994 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:43.994 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:43.995 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:02:43.995 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:43.995 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:02:43.995 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:43.996 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:02:43.997 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:02:43.997 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:43.998 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:43.998 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:43.998 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:02:43.999 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:43.999 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:43.999 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:02:44.000 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:44.000 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:02:44.000 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:44.000 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:02:44.001 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:02:44.001 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:44.001 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:44.002 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:44.002 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:02:44.002 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:44.003 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:02:44.003 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764501.249520_420.1", 2022-09-09T23:02:44.003 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:02:44.003 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:02:44.004 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:02:44.004 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:01:41.249520Z", 2022-09-09T23:02:44.004 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:02:44.005 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:44.005 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:44.005 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:02:44.005 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:44.006 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:02:44.006 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:02:44.006 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:02:44.007 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:44.007 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:02:43Z" 2022-09-09T23:02:44.007 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:02:44.009 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764501.249520_420.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:01:41.249520Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:02:43Z" } 2022-09-09T23:02:44.009 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764501.249520_420.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:01:41.249520Z', 'realm_epoch': 4}}] 2022-09-09T23:02:44.010 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:02:44.010 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:02:44.011 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-09T23:02:44.011 DEBUG:boto:path=/jkcgbj-49/ 2022-09-09T23:02:44.012 DEBUG:boto:auth_path=/jkcgbj-49/ 2022-09-09T23:02:44.013 DEBUG:boto:Method: HEAD 2022-09-09T23:02:44.013 DEBUG:boto:Path: /jkcgbj-49/ 2022-09-09T23:02:44.014 DEBUG:boto:Data: 2022-09-09T23:02:44.014 DEBUG:boto:Headers: {} 2022-09-09T23:02:44.015 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T23:02:44.015 DEBUG:boto:Port: 8000 2022-09-09T23:02:44.016 DEBUG:boto:Params: {} 2022-09-09T23:02:44.017 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:02:44.017 DEBUG:boto:Token: None 2022-09-09T23:02:44.018 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 23:02:44 GMT /jkcgbj-49/ 2022-09-09T23:02:44.019 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:gatCPfIfPWA7aHYS7Q2k/9vM01o= 2022-09-09T23:02:44.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:02:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:gatCPfIfPWA7aHYS7Q2k/9vM01o=', 'Content-Length': '0'} 2022-09-09T23:02:44.028 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f5e860f2fee79df-00631bc614-1607-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:02:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:02:44.029 DEBUG:boto:path=/jkcgbj-49/testredirect 2022-09-09T23:02:44.030 DEBUG:boto:auth_path=/jkcgbj-49/testredirect 2022-09-09T23:02:44.031 DEBUG:boto:Method: HEAD 2022-09-09T23:02:44.031 DEBUG:boto:Path: /jkcgbj-49/testredirect 2022-09-09T23:02:44.032 DEBUG:boto:Data: 2022-09-09T23:02:44.032 DEBUG:boto:Headers: {} 2022-09-09T23:02:44.033 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T23:02:44.033 DEBUG:boto:Port: 8000 2022-09-09T23:02:44.034 DEBUG:boto:Params: {} 2022-09-09T23:02:44.034 DEBUG:boto:Token: None 2022-09-09T23:02:44.035 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 23:02:44 GMT /jkcgbj-49/testredirect 2022-09-09T23:02:44.036 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wI92LddWE6aHfqaHELraLSwU98Q= 2022-09-09T23:02:44.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:02:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wI92LddWE6aHfqaHELraLSwU98Q=', 'Content-Length': '0'} 2022-09-09T23:02:44.039 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000a5132b9d86deea8a-00631bc614-1607-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 23:02:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:02:44.040 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=4b05608c-7197-4e9b-bd29-7989607fa277', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:02:44.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=4b05608c-7197-4e9b-bd29-7989607fa277', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:02:44.041 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone=4b05608c-7197-4e9b-bd29-7989607fa277 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-09T23:02:44.087 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:02:44.087 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:02:44.129 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:02:44.124+0000 7fb9e7ca5dc0 0 failed reading obj info from .rgw.root:zone_info.4b05608c-7197-4e9b-bd29-7989607fa277: (2) No such file or directory 2022-09-09T23:02:44.129 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:02:44.124+0000 7fb9e7ca5dc0 0 WARNING: could not read zone params for zone id=4b05608c-7197-4e9b-bd29-7989607fa277 name=test-zone1 2022-09-09T23:02:44.130 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:02:44.128+0000 7fb9e7ca5dc0 0 failed reading obj info from .rgw.root:zone_info.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5: (2) No such file or directory 2022-09-09T23:02:44.130 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:02:44.128+0000 7fb9e7ca5dc0 0 WARNING: could not read zone params for zone id=79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5 name=test-zone3 2022-09-09T23:02:44.140 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:02:44.140 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T23:02:44.140 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone2", 2022-09-09T23:02:44.141 INFO:teuthology.orchestra.run.smithi080.stdout: "domain_root": "test-zone2.rgw.meta:root", 2022-09-09T23:02:44.141 INFO:teuthology.orchestra.run.smithi080.stdout: "control_pool": "test-zone2.rgw.control", 2022-09-09T23:02:44.142 INFO:teuthology.orchestra.run.smithi080.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2022-09-09T23:02:44.142 INFO:teuthology.orchestra.run.smithi080.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2022-09-09T23:02:44.142 INFO:teuthology.orchestra.run.smithi080.stdout: "log_pool": "test-zone2.rgw.log", 2022-09-09T23:02:44.143 INFO:teuthology.orchestra.run.smithi080.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2022-09-09T23:02:44.143 INFO:teuthology.orchestra.run.smithi080.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2022-09-09T23:02:44.143 INFO:teuthology.orchestra.run.smithi080.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2022-09-09T23:02:44.143 INFO:teuthology.orchestra.run.smithi080.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2022-09-09T23:02:44.144 INFO:teuthology.orchestra.run.smithi080.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2022-09-09T23:02:44.144 INFO:teuthology.orchestra.run.smithi080.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2022-09-09T23:02:44.144 INFO:teuthology.orchestra.run.smithi080.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2022-09-09T23:02:44.144 INFO:teuthology.orchestra.run.smithi080.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2022-09-09T23:02:44.145 INFO:teuthology.orchestra.run.smithi080.stdout: "otp_pool": "test-zone2.rgw.otp", 2022-09-09T23:02:44.145 INFO:teuthology.orchestra.run.smithi080.stdout: "system_key": { 2022-09-09T23:02:44.145 INFO:teuthology.orchestra.run.smithi080.stdout: "access_key": "3U0V7WQYC6MLEXED", 2022-09-09T23:02:44.146 INFO:teuthology.orchestra.run.smithi080.stdout: "secret_key": "jA4iNzYKmfzg78oKBQTy4mNpjERzasVo" 2022-09-09T23:02:44.146 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:44.146 INFO:teuthology.orchestra.run.smithi080.stdout: "placement_pools": [ 2022-09-09T23:02:44.147 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:44.147 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "default-placement", 2022-09-09T23:02:44.147 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:02:44.148 INFO:teuthology.orchestra.run.smithi080.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2022-09-09T23:02:44.148 INFO:teuthology.orchestra.run.smithi080.stdout: "storage_classes": { 2022-09-09T23:02:44.148 INFO:teuthology.orchestra.run.smithi080.stdout: "STANDARD": { 2022-09-09T23:02:44.148 INFO:teuthology.orchestra.run.smithi080.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2022-09-09T23:02:44.149 INFO:teuthology.orchestra.run.smithi080.stdout: "compression_type": "random" 2022-09-09T23:02:44.149 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:44.149 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:44.150 INFO:teuthology.orchestra.run.smithi080.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2022-09-09T23:02:44.150 INFO:teuthology.orchestra.run.smithi080.stdout: "index_type": 0 2022-09-09T23:02:44.150 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:44.150 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:44.151 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:02:44.151 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T23:02:44.151 INFO:teuthology.orchestra.run.smithi080.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2022-09-09T23:02:44.151 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:02:44.153 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:02:44.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:02:44.155 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-09T23:02:44.243 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:02:44.243 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:02:44.349 INFO:teuthology.orchestra.run.smithi080.stderr:Sending period to new master zone 4b05608c-7197-4e9b-bd29-7989607fa277 2022-09-09T23:02:44.418 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:02:44.418 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:02:44.419 INFO:teuthology.orchestra.run.smithi080.stdout: "epoch": 3, 2022-09-09T23:02:44.419 INFO:teuthology.orchestra.run.smithi080.stdout: "predecessor_uuid": "2efdc3a1-023b-4e78-aaa2-0e069a034dac", 2022-09-09T23:02:44.419 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": [ 2022-09-09T23:02:44.420 INFO:teuthology.orchestra.run.smithi080.stdout: "", 2022-09-09T23:02:44.420 INFO:teuthology.orchestra.run.smithi080.stdout: "", 2022-09-09T23:02:44.420 INFO:teuthology.orchestra.run.smithi080.stdout: "", 2022-09-09T23:02:44.420 INFO:teuthology.orchestra.run.smithi080.stdout: "1_1662764312.564852_2684.1" 2022-09-09T23:02:44.421 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:02:44.421 INFO:teuthology.orchestra.run.smithi080.stdout: "period_map": { 2022-09-09T23:02:44.421 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:02:44.421 INFO:teuthology.orchestra.run.smithi080.stdout: "zonegroups": [ 2022-09-09T23:02:44.422 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:44.422 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "test-zonegroup", 2022-09-09T23:02:44.422 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zonegroup", 2022-09-09T23:02:44.422 INFO:teuthology.orchestra.run.smithi080.stdout: "api_name": "", 2022-09-09T23:02:44.423 INFO:teuthology.orchestra.run.smithi080.stdout: "is_master": true, 2022-09-09T23:02:44.423 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T23:02:44.424 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi005.front.sepia.ceph.com:8000" 2022-09-09T23:02:44.424 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:02:44.425 INFO:teuthology.orchestra.run.smithi080.stdout: "hostnames": [], 2022-09-09T23:02:44.426 INFO:teuthology.orchestra.run.smithi080.stdout: "hostnames_s3website": [], 2022-09-09T23:02:44.426 INFO:teuthology.orchestra.run.smithi080.stdout: "master_zone": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:02:44.427 INFO:teuthology.orchestra.run.smithi080.stdout: "zones": [ 2022-09-09T23:02:44.427 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:44.427 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "3ff609e7-37d9-4b47-8369-34e6c99c4c03", 2022-09-09T23:02:44.428 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone4", 2022-09-09T23:02:44.428 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T23:02:44.429 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi080.front.sepia.ceph.com:8001" 2022-09-09T23:02:44.429 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:02:44.430 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": false, 2022-09-09T23:02:44.430 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": true, 2022-09-09T23:02:44.430 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:02:44.431 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": false, 2022-09-09T23:02:44.431 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "pubsub", 2022-09-09T23:02:44.432 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": false, 2022-09-09T23:02:44.432 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [ 2022-09-09T23:02:44.432 INFO:teuthology.orchestra.run.smithi080.stdout: "test-zone1" 2022-09-09T23:02:44.432 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:02:44.433 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "", 2022-09-09T23:02:44.433 INFO:teuthology.orchestra.run.smithi080.stdout: "supported_features": [ 2022-09-09T23:02:44.433 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T23:02:44.434 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:02:44.434 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:44.434 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:44.435 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:02:44.435 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone1", 2022-09-09T23:02:44.435 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T23:02:44.436 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi005.front.sepia.ceph.com:8000" 2022-09-09T23:02:44.436 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:02:44.436 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": false, 2022-09-09T23:02:44.436 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": true, 2022-09-09T23:02:44.437 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:02:44.437 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": false, 2022-09-09T23:02:44.437 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "", 2022-09-09T23:02:44.437 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": true, 2022-09-09T23:02:44.438 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [], 2022-09-09T23:02:44.438 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "", 2022-09-09T23:02:44.438 INFO:teuthology.orchestra.run.smithi080.stdout: "supported_features": [ 2022-09-09T23:02:44.438 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T23:02:44.439 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:02:44.439 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:44.439 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:44.439 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5", 2022-09-09T23:02:44.440 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone3", 2022-09-09T23:02:44.440 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T23:02:44.440 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi005.front.sepia.ceph.com:8001" 2022-09-09T23:02:44.440 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:02:44.441 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": false, 2022-09-09T23:02:44.441 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": true, 2022-09-09T23:02:44.441 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:02:44.441 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": false, 2022-09-09T23:02:44.442 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "", 2022-09-09T23:02:44.442 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": true, 2022-09-09T23:02:44.442 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [], 2022-09-09T23:02:44.442 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "", 2022-09-09T23:02:44.443 INFO:teuthology.orchestra.run.smithi080.stdout: "supported_features": [ 2022-09-09T23:02:44.443 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T23:02:44.444 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:02:44.444 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:44.445 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:44.445 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T23:02:44.446 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone2", 2022-09-09T23:02:44.446 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T23:02:44.447 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi080.front.sepia.ceph.com:8000" 2022-09-09T23:02:44.447 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:02:44.448 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": false, 2022-09-09T23:02:44.448 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": true, 2022-09-09T23:02:44.449 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:02:44.449 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": false, 2022-09-09T23:02:44.449 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "", 2022-09-09T23:02:44.450 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": false, 2022-09-09T23:02:44.450 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [], 2022-09-09T23:02:44.450 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:02:44.451 INFO:teuthology.orchestra.run.smithi080.stdout: "supported_features": [ 2022-09-09T23:02:44.451 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T23:02:44.451 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:02:44.452 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:44.452 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:02:44.452 INFO:teuthology.orchestra.run.smithi080.stdout: "placement_targets": [ 2022-09-09T23:02:44.452 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:44.454 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "default-placement", 2022-09-09T23:02:44.454 INFO:teuthology.orchestra.run.smithi080.stdout: "tags": [], 2022-09-09T23:02:44.455 INFO:teuthology.orchestra.run.smithi080.stdout: "storage_classes": [ 2022-09-09T23:02:44.455 INFO:teuthology.orchestra.run.smithi080.stdout: "STANDARD" 2022-09-09T23:02:44.455 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:02:44.455 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:44.456 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:02:44.456 INFO:teuthology.orchestra.run.smithi080.stdout: "default_placement": "default-placement", 2022-09-09T23:02:44.456 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T23:02:44.457 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_policy": { 2022-09-09T23:02:44.457 INFO:teuthology.orchestra.run.smithi080.stdout: "groups": [] 2022-09-09T23:02:44.457 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:44.458 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled_features": [ 2022-09-09T23:02:44.458 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T23:02:44.458 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:02:44.459 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:44.459 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:02:44.459 INFO:teuthology.orchestra.run.smithi080.stdout: "short_zone_ids": [ 2022-09-09T23:02:44.459 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:44.460 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "3ff609e7-37d9-4b47-8369-34e6c99c4c03", 2022-09-09T23:02:44.460 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 2593176765 2022-09-09T23:02:44.460 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:44.461 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:44.461 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:02:44.461 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 1301724575 2022-09-09T23:02:44.462 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:44.462 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:44.463 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5", 2022-09-09T23:02:44.463 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 2787153217 2022-09-09T23:02:44.463 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:44.464 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:02:44.464 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T23:02:44.465 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 1899625128 2022-09-09T23:02:44.465 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:44.465 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:02:44.466 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:44.466 INFO:teuthology.orchestra.run.smithi080.stdout: "master_zonegroup": "test-zonegroup", 2022-09-09T23:02:44.467 INFO:teuthology.orchestra.run.smithi080.stdout: "master_zone": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:02:44.467 INFO:teuthology.orchestra.run.smithi080.stdout: "period_config": { 2022-09-09T23:02:44.467 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_quota": { 2022-09-09T23:02:44.467 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false, 2022-09-09T23:02:44.468 INFO:teuthology.orchestra.run.smithi080.stdout: "check_on_raw": false, 2022-09-09T23:02:44.468 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size": -1, 2022-09-09T23:02:44.468 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size_kb": 0, 2022-09-09T23:02:44.468 INFO:teuthology.orchestra.run.smithi080.stdout: "max_objects": -1 2022-09-09T23:02:44.469 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:44.469 INFO:teuthology.orchestra.run.smithi080.stdout: "user_quota": { 2022-09-09T23:02:44.469 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false, 2022-09-09T23:02:44.469 INFO:teuthology.orchestra.run.smithi080.stdout: "check_on_raw": false, 2022-09-09T23:02:44.470 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size": -1, 2022-09-09T23:02:44.470 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size_kb": 0, 2022-09-09T23:02:44.470 INFO:teuthology.orchestra.run.smithi080.stdout: "max_objects": -1 2022-09-09T23:02:44.471 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:44.471 INFO:teuthology.orchestra.run.smithi080.stdout: "user_ratelimit": { 2022-09-09T23:02:44.471 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_ops": 0, 2022-09-09T23:02:44.471 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_ops": 0, 2022-09-09T23:02:44.472 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_bytes": 0, 2022-09-09T23:02:44.472 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_bytes": 0, 2022-09-09T23:02:44.472 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false 2022-09-09T23:02:44.472 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:44.473 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_ratelimit": { 2022-09-09T23:02:44.473 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_ops": 0, 2022-09-09T23:02:44.473 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_ops": 0, 2022-09-09T23:02:44.473 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_bytes": 0, 2022-09-09T23:02:44.474 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_bytes": 0, 2022-09-09T23:02:44.474 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false 2022-09-09T23:02:44.474 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:44.474 INFO:teuthology.orchestra.run.smithi080.stdout: "anonymous_ratelimit": { 2022-09-09T23:02:44.475 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_ops": 0, 2022-09-09T23:02:44.475 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_ops": 0, 2022-09-09T23:02:44.475 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_bytes": 0, 2022-09-09T23:02:44.475 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_bytes": 0, 2022-09-09T23:02:44.476 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false 2022-09-09T23:02:44.476 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:02:44.476 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:02:44.476 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T23:02:44.477 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_name": "test-realm", 2022-09-09T23:02:44.477 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:02:44.477 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:02:44.478 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "4b05608c-7197-4e9b-bd29-7989607fa277" 2022-09-09T23:03:44.479 DEBUG:boto:path=/jkcgbj-49/testredirect 2022-09-09T23:03:44.481 DEBUG:boto:auth_path=/jkcgbj-49/testredirect 2022-09-09T23:03:44.482 DEBUG:boto:Method: HEAD 2022-09-09T23:03:44.482 DEBUG:boto:Path: /jkcgbj-49/testredirect 2022-09-09T23:03:44.483 DEBUG:boto:Data: 2022-09-09T23:03:44.483 DEBUG:boto:Headers: {} 2022-09-09T23:03:44.484 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T23:03:44.484 DEBUG:boto:Port: 8000 2022-09-09T23:03:44.485 DEBUG:boto:Params: {} 2022-09-09T23:03:44.486 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:03:44.486 DEBUG:boto:Token: None 2022-09-09T23:03:44.487 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 23:03:44 GMT /jkcgbj-49/testredirect 2022-09-09T23:03:44.488 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:aPlScvoE4YP1uHuj0GDjvzk9vkA= 2022-09-09T23:03:44.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:03:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:aPlScvoE4YP1uHuj0GDjvzk9vkA=', 'Content-Length': '0'} 2022-09-09T23:03:44.498 DEBUG:boto:Response headers: [('Location', 'http://smithi005.front.sepia.ceph.com:8000/jkcgbj-49/testredirect?'), ('x-amz-request-id', 'tx00000ee2993509e6a8d32-00631bc650-166c-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:03:44.499 DEBUG:boto:Redirecting: http://smithi005.front.sepia.ceph.com/jkcgbj-49/testredirect 2022-09-09T23:03:44.499 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': '8000'} 2022-09-09T23:03:44.500 DEBUG:boto:Token: None 2022-09-09T23:03:44.501 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 23:03:44 GMT /jkcgbj-49/testredirect 2022-09-09T23:03:44.502 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:aPlScvoE4YP1uHuj0GDjvzk9vkA= 2022-09-09T23:03:44.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:03:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:aPlScvoE4YP1uHuj0GDjvzk9vkA=', 'Content-Length': '0'} 2022-09-09T23:03:44.513 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 23:01:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000aa3e51151983eae2-00631bc650-150f-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 23:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:03:44.515 DEBUG:boto:path=/jkcgbj-49/testredirect 2022-09-09T23:03:44.515 DEBUG:boto:auth_path=/jkcgbj-49/testredirect 2022-09-09T23:03:44.516 DEBUG:boto:Method: GET 2022-09-09T23:03:44.517 DEBUG:boto:Path: /jkcgbj-49/testredirect 2022-09-09T23:03:44.517 DEBUG:boto:Data: 2022-09-09T23:03:44.518 DEBUG:boto:Headers: {} 2022-09-09T23:03:44.519 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T23:03:44.520 DEBUG:boto:Port: 8000 2022-09-09T23:03:44.520 DEBUG:boto:Params: {} 2022-09-09T23:03:44.521 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:03:44.522 DEBUG:boto:Token: None 2022-09-09T23:03:44.522 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 23:03:44 GMT /jkcgbj-49/testredirect 2022-09-09T23:03:44.523 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8RWA5jq74JCrFYxoQEOKWG2NQ+8= 2022-09-09T23:03:44.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:03:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8RWA5jq74JCrFYxoQEOKWG2NQ+8=', 'Content-Length': '0'} 2022-09-09T23:03:44.527 DEBUG:boto:Response headers: [('Location', 'http://smithi005.front.sepia.ceph.com:8000/jkcgbj-49/testredirect?'), ('x-amz-request-id', 'tx00000b888d49fdc5e20b9-00631bc650-166c-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:03:44.528 DEBUG:boto:Redirecting: http://smithi005.front.sepia.ceph.com/jkcgbj-49/testredirect 2022-09-09T23:03:44.529 DEBUG:boto:Token: None 2022-09-09T23:03:44.530 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 23:03:44 GMT /jkcgbj-49/testredirect 2022-09-09T23:03:44.530 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8RWA5jq74JCrFYxoQEOKWG2NQ+8= 2022-09-09T23:03:44.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:03:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8RWA5jq74JCrFYxoQEOKWG2NQ+8=', 'Content-Length': '0'} 2022-09-09T23:03:44.535 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 23:01:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx000007fa1dacdb7c274b5-00631bc650-150f-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 23:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:03:44.536 DEBUG:boto:path=/jkcgbj-49/testredirect 2022-09-09T23:03:44.537 DEBUG:boto:auth_path=/jkcgbj-49/testredirect 2022-09-09T23:03:44.537 DEBUG:boto:Method: PUT 2022-09-09T23:03:44.538 DEBUG:boto:Path: /jkcgbj-49/testredirect 2022-09-09T23:03:44.538 DEBUG:boto:Data: 2022-09-09T23:03:44.539 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue'} 2022-09-09T23:03:44.540 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T23:03:44.540 DEBUG:boto:Port: 8000 2022-09-09T23:03:44.541 DEBUG:boto:Params: {} 2022-09-09T23:03:44.541 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:03:44.542 DEBUG:boto:Token: None 2022-09-09T23:03:44.543 DEBUG:boto:StringToSign: PUT VpBzljOcorCZvRIkX5Nt3A== application/octet-stream Fri, 09 Sep 2022 23:03:44 GMT /jkcgbj-49/testredirect 2022-09-09T23:03:44.543 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:BMVFfTB4b6/TzWCusaLUyj1osJk= 2022-09-09T23:03:44.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 23:03:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:BMVFfTB4b6/TzWCusaLUyj1osJk='} 2022-09-09T23:03:44.566 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ad42495880b43bfd-00631bc650-150f-test-zone1'), ('Date', 'Fri, 09 Sep 2022 23:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:03:44.567 DEBUG:boto:path=/jkcgbj-49/testredirect 2022-09-09T23:03:44.568 DEBUG:boto:auth_path=/jkcgbj-49/testredirect 2022-09-09T23:03:44.568 DEBUG:boto:Method: GET 2022-09-09T23:03:44.569 DEBUG:boto:Path: /jkcgbj-49/testredirect 2022-09-09T23:03:44.569 DEBUG:boto:Data: 2022-09-09T23:03:44.570 DEBUG:boto:Headers: {} 2022-09-09T23:03:44.570 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T23:03:44.571 DEBUG:boto:Port: 8000 2022-09-09T23:03:44.572 DEBUG:boto:Params: {} 2022-09-09T23:03:44.572 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:03:44.573 DEBUG:boto:Token: None 2022-09-09T23:03:44.574 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 23:03:44 GMT /jkcgbj-49/testredirect 2022-09-09T23:03:44.575 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8RWA5jq74JCrFYxoQEOKWG2NQ+8= 2022-09-09T23:03:44.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:03:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8RWA5jq74JCrFYxoQEOKWG2NQ+8=', 'Content-Length': '0'} 2022-09-09T23:03:44.579 DEBUG:boto:Response headers: [('Location', 'http://smithi005.front.sepia.ceph.com:8000/jkcgbj-49/testredirect?'), ('x-amz-request-id', 'tx00000d9e86851ea938db5-00631bc650-166c-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:03:44.580 DEBUG:boto:Redirecting: http://smithi005.front.sepia.ceph.com/jkcgbj-49/testredirect 2022-09-09T23:03:44.580 DEBUG:boto:Token: None 2022-09-09T23:03:44.581 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 23:03:44 GMT /jkcgbj-49/testredirect 2022-09-09T23:03:44.582 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8RWA5jq74JCrFYxoQEOKWG2NQ+8= 2022-09-09T23:03:44.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:03:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8RWA5jq74JCrFYxoQEOKWG2NQ+8=', 'Content-Length': '0'} 2022-09-09T23:03:44.585 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 23:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('x-amz-request-id', 'tx00000b75a483ab311db51-00631bc650-150f-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 23:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:03:44.587 DEBUG:boto:path=/jkcgbj-49/testredirect 2022-09-09T23:03:44.587 DEBUG:boto:auth_path=/jkcgbj-49/testredirect 2022-09-09T23:03:44.588 DEBUG:boto:Method: PUT 2022-09-09T23:03:44.588 DEBUG:boto:Path: /jkcgbj-49/testredirect 2022-09-09T23:03:44.589 DEBUG:boto:Data: 2022-09-09T23:03:44.589 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue'} 2022-09-09T23:03:44.590 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T23:03:44.591 DEBUG:boto:Port: 8000 2022-09-09T23:03:44.591 DEBUG:boto:Params: {} 2022-09-09T23:03:44.592 DEBUG:boto:Token: None 2022-09-09T23:03:44.592 DEBUG:boto:StringToSign: PUT uk9S5NXZfBvPq4jGr+LM5g== application/octet-stream Fri, 09 Sep 2022 23:03:44 GMT /jkcgbj-49/testredirect 2022-09-09T23:03:44.593 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Jaj6CY8onTnCaUbolpEl4dKvgMw= 2022-09-09T23:03:44.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 23:03:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Jaj6CY8onTnCaUbolpEl4dKvgMw='} 2022-09-09T23:03:44.604 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000cf5bec97faa91b0-00631bc650-150f-test-zone1'), ('Date', 'Fri, 09 Sep 2022 23:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:03:44.604 DEBUG:boto:path=/jkcgbj-49/testredirect 2022-09-09T23:03:44.605 DEBUG:boto:auth_path=/jkcgbj-49/testredirect 2022-09-09T23:03:44.605 DEBUG:boto:Method: GET 2022-09-09T23:03:44.606 DEBUG:boto:Path: /jkcgbj-49/testredirect 2022-09-09T23:03:44.607 DEBUG:boto:Data: 2022-09-09T23:03:44.607 DEBUG:boto:Headers: {} 2022-09-09T23:03:44.608 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T23:03:44.608 DEBUG:boto:Port: 8000 2022-09-09T23:03:44.609 DEBUG:boto:Params: {} 2022-09-09T23:03:44.609 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:03:44.610 DEBUG:boto:Token: None 2022-09-09T23:03:44.610 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 23:03:44 GMT /jkcgbj-49/testredirect 2022-09-09T23:03:44.611 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8RWA5jq74JCrFYxoQEOKWG2NQ+8= 2022-09-09T23:03:44.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:03:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8RWA5jq74JCrFYxoQEOKWG2NQ+8=', 'Content-Length': '0'} 2022-09-09T23:03:44.615 DEBUG:boto:Response headers: [('Location', 'http://smithi005.front.sepia.ceph.com:8000/jkcgbj-49/testredirect?'), ('x-amz-request-id', 'tx0000049b182b1a331edfe-00631bc650-166c-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:03:44.616 DEBUG:boto:Redirecting: http://smithi005.front.sepia.ceph.com/jkcgbj-49/testredirect 2022-09-09T23:03:44.616 DEBUG:boto:Token: None 2022-09-09T23:03:44.617 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 23:03:44 GMT /jkcgbj-49/testredirect 2022-09-09T23:03:44.618 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8RWA5jq74JCrFYxoQEOKWG2NQ+8= 2022-09-09T23:03:44.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:03:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8RWA5jq74JCrFYxoQEOKWG2NQ+8=', 'Content-Length': '0'} 2022-09-09T23:03:44.621 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 23:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('x-amz-request-id', 'tx00000a2a50c8679212666-00631bc650-150f-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 23:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:03:44.622 DEBUG:boto:path=/jkcgbj-49/testredirect 2022-09-09T23:03:44.623 DEBUG:boto:auth_path=/jkcgbj-49/testredirect 2022-09-09T23:03:44.623 DEBUG:boto:Method: PUT 2022-09-09T23:03:44.624 DEBUG:boto:Path: /jkcgbj-49/testredirect 2022-09-09T23:03:44.624 DEBUG:boto:Data: 2022-09-09T23:03:44.625 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue'} 2022-09-09T23:03:44.625 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T23:03:44.626 DEBUG:boto:Port: 8000 2022-09-09T23:03:44.626 DEBUG:boto:Params: {} 2022-09-09T23:03:44.627 DEBUG:boto:Token: None 2022-09-09T23:03:44.627 DEBUG:boto:StringToSign: PUT 6aKmvaJMgaK7Z2rfDZvVeg== application/octet-stream Fri, 09 Sep 2022 23:03:44 GMT /jkcgbj-49/testredirect 2022-09-09T23:03:44.628 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:zzMxyOV97vpk+NLtuweNM0w2msw= 2022-09-09T23:03:44.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 23:03:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:zzMxyOV97vpk+NLtuweNM0w2msw='} 2022-09-09T23:03:44.639 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004034d357176be49a-00631bc650-150f-test-zone1'), ('Date', 'Fri, 09 Sep 2022 23:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:03:44.640 DEBUG:boto:path=/jkcgbj-49/testredirect 2022-09-09T23:03:44.640 DEBUG:boto:auth_path=/jkcgbj-49/testredirect 2022-09-09T23:03:44.641 DEBUG:boto:Method: GET 2022-09-09T23:03:44.641 DEBUG:boto:Path: /jkcgbj-49/testredirect 2022-09-09T23:03:44.642 DEBUG:boto:Data: 2022-09-09T23:03:44.642 DEBUG:boto:Headers: {} 2022-09-09T23:03:44.643 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T23:03:44.643 DEBUG:boto:Port: 8000 2022-09-09T23:03:44.644 DEBUG:boto:Params: {} 2022-09-09T23:03:44.644 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:03:44.645 DEBUG:boto:Token: None 2022-09-09T23:03:44.646 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 23:03:44 GMT /jkcgbj-49/testredirect 2022-09-09T23:03:44.646 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8RWA5jq74JCrFYxoQEOKWG2NQ+8= 2022-09-09T23:03:44.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:03:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8RWA5jq74JCrFYxoQEOKWG2NQ+8=', 'Content-Length': '0'} 2022-09-09T23:03:44.650 DEBUG:boto:Response headers: [('Location', 'http://smithi005.front.sepia.ceph.com:8000/jkcgbj-49/testredirect?'), ('x-amz-request-id', 'tx00000585358f1ae9ced26-00631bc650-166c-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:03:44.651 DEBUG:boto:Redirecting: http://smithi005.front.sepia.ceph.com/jkcgbj-49/testredirect 2022-09-09T23:03:44.652 DEBUG:boto:Token: None 2022-09-09T23:03:44.652 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 23:03:44 GMT /jkcgbj-49/testredirect 2022-09-09T23:03:44.653 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8RWA5jq74JCrFYxoQEOKWG2NQ+8= 2022-09-09T23:03:44.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:03:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8RWA5jq74JCrFYxoQEOKWG2NQ+8=', 'Content-Length': '0'} 2022-09-09T23:03:44.657 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 23:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('x-amz-request-id', 'tx00000269e3f99244a5822-00631bc650-150f-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 23:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:03:44.658 DEBUG:boto:path=/jkcgbj-49/testredirect 2022-09-09T23:03:44.658 DEBUG:boto:auth_path=/jkcgbj-49/testredirect 2022-09-09T23:03:44.659 DEBUG:boto:Method: PUT 2022-09-09T23:03:44.659 DEBUG:boto:Path: /jkcgbj-49/testredirect 2022-09-09T23:03:44.660 DEBUG:boto:Data: 2022-09-09T23:03:44.660 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue'} 2022-09-09T23:03:44.661 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T23:03:44.661 DEBUG:boto:Port: 8000 2022-09-09T23:03:44.662 DEBUG:boto:Params: {} 2022-09-09T23:03:44.662 DEBUG:boto:Token: None 2022-09-09T23:03:44.663 DEBUG:boto:StringToSign: PUT GbNgNQ5MAukCtgY+rsT5Xw== application/octet-stream Fri, 09 Sep 2022 23:03:44 GMT /jkcgbj-49/testredirect 2022-09-09T23:03:44.664 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:zaUwN2Ft3pS04yuv4gtzCRTfgM4= 2022-09-09T23:03:44.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 23:03:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:zaUwN2Ft3pS04yuv4gtzCRTfgM4='} 2022-09-09T23:03:44.710 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f2da8aed88477c64-00631bc650-150f-test-zone1'), ('Date', 'Fri, 09 Sep 2022 23:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:03:44.711 DEBUG:boto:path=/jkcgbj-49/testredirect 2022-09-09T23:03:44.712 DEBUG:boto:auth_path=/jkcgbj-49/testredirect 2022-09-09T23:03:44.712 DEBUG:boto:Method: GET 2022-09-09T23:03:44.713 DEBUG:boto:Path: /jkcgbj-49/testredirect 2022-09-09T23:03:44.713 DEBUG:boto:Data: 2022-09-09T23:03:44.714 DEBUG:boto:Headers: {} 2022-09-09T23:03:44.714 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T23:03:44.715 DEBUG:boto:Port: 8000 2022-09-09T23:03:44.716 DEBUG:boto:Params: {} 2022-09-09T23:03:44.716 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:03:44.717 DEBUG:boto:Token: None 2022-09-09T23:03:44.718 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 23:03:44 GMT /jkcgbj-49/testredirect 2022-09-09T23:03:44.720 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8RWA5jq74JCrFYxoQEOKWG2NQ+8= 2022-09-09T23:03:44.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:03:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8RWA5jq74JCrFYxoQEOKWG2NQ+8=', 'Content-Length': '0'} 2022-09-09T23:03:44.725 DEBUG:boto:Response headers: [('Location', 'http://smithi005.front.sepia.ceph.com:8000/jkcgbj-49/testredirect?'), ('x-amz-request-id', 'tx000001e86738d9fff7202-00631bc650-166c-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:03:44.726 DEBUG:boto:Redirecting: http://smithi005.front.sepia.ceph.com/jkcgbj-49/testredirect 2022-09-09T23:03:44.727 DEBUG:boto:Token: None 2022-09-09T23:03:44.728 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 23:03:44 GMT /jkcgbj-49/testredirect 2022-09-09T23:03:44.729 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:8RWA5jq74JCrFYxoQEOKWG2NQ+8= 2022-09-09T23:03:44.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:03:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:8RWA5jq74JCrFYxoQEOKWG2NQ+8=', 'Content-Length': '0'} 2022-09-09T23:03:44.733 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 09 Sep 2022 23:03:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('x-amz-request-id', 'tx000003bce41c55547d43b-00631bc650-150f-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 09 Sep 2022 23:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:03:44.734 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:03:44.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:03:44.735 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=true --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-09T23:03:44.785 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:03:44.786 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:03:44.830 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:03:44.827+0000 7faabc2addc0 0 failed reading obj info from .rgw.root:zone_info.4b05608c-7197-4e9b-bd29-7989607fa277: (2) No such file or directory 2022-09-09T23:03:44.830 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:03:44.827+0000 7faabc2addc0 0 WARNING: could not read zone params for zone id=4b05608c-7197-4e9b-bd29-7989607fa277 name=test-zone1 2022-09-09T23:03:44.831 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:03:44.827+0000 7faabc2addc0 0 failed reading obj info from .rgw.root:zone_info.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5: (2) No such file or directory 2022-09-09T23:03:44.831 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:03:44.827+0000 7faabc2addc0 0 WARNING: could not read zone params for zone id=79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5 name=test-zone3 2022-09-09T23:03:44.844 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:03:44.844 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T23:03:44.844 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone2", 2022-09-09T23:03:44.845 INFO:teuthology.orchestra.run.smithi080.stdout: "domain_root": "test-zone2.rgw.meta:root", 2022-09-09T23:03:44.845 INFO:teuthology.orchestra.run.smithi080.stdout: "control_pool": "test-zone2.rgw.control", 2022-09-09T23:03:44.845 INFO:teuthology.orchestra.run.smithi080.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2022-09-09T23:03:44.846 INFO:teuthology.orchestra.run.smithi080.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2022-09-09T23:03:44.846 INFO:teuthology.orchestra.run.smithi080.stdout: "log_pool": "test-zone2.rgw.log", 2022-09-09T23:03:44.846 INFO:teuthology.orchestra.run.smithi080.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2022-09-09T23:03:44.846 INFO:teuthology.orchestra.run.smithi080.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2022-09-09T23:03:44.847 INFO:teuthology.orchestra.run.smithi080.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2022-09-09T23:03:44.847 INFO:teuthology.orchestra.run.smithi080.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2022-09-09T23:03:44.847 INFO:teuthology.orchestra.run.smithi080.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2022-09-09T23:03:44.848 INFO:teuthology.orchestra.run.smithi080.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2022-09-09T23:03:44.848 INFO:teuthology.orchestra.run.smithi080.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2022-09-09T23:03:44.848 INFO:teuthology.orchestra.run.smithi080.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2022-09-09T23:03:44.848 INFO:teuthology.orchestra.run.smithi080.stdout: "otp_pool": "test-zone2.rgw.otp", 2022-09-09T23:03:44.849 INFO:teuthology.orchestra.run.smithi080.stdout: "system_key": { 2022-09-09T23:03:44.849 INFO:teuthology.orchestra.run.smithi080.stdout: "access_key": "3U0V7WQYC6MLEXED", 2022-09-09T23:03:44.849 INFO:teuthology.orchestra.run.smithi080.stdout: "secret_key": "jA4iNzYKmfzg78oKBQTy4mNpjERzasVo" 2022-09-09T23:03:44.849 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:03:44.850 INFO:teuthology.orchestra.run.smithi080.stdout: "placement_pools": [ 2022-09-09T23:03:44.850 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:03:44.850 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "default-placement", 2022-09-09T23:03:44.851 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:03:44.851 INFO:teuthology.orchestra.run.smithi080.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2022-09-09T23:03:44.851 INFO:teuthology.orchestra.run.smithi080.stdout: "storage_classes": { 2022-09-09T23:03:44.851 INFO:teuthology.orchestra.run.smithi080.stdout: "STANDARD": { 2022-09-09T23:03:44.852 INFO:teuthology.orchestra.run.smithi080.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2022-09-09T23:03:44.852 INFO:teuthology.orchestra.run.smithi080.stdout: "compression_type": "random" 2022-09-09T23:03:44.852 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:03:44.852 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:03:44.853 INFO:teuthology.orchestra.run.smithi080.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2022-09-09T23:03:44.853 INFO:teuthology.orchestra.run.smithi080.stdout: "index_type": 0 2022-09-09T23:03:44.854 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:03:44.854 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:03:44.855 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:03:44.855 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T23:03:44.855 INFO:teuthology.orchestra.run.smithi080.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2022-09-09T23:03:44.856 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:03:44.857 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:03:44.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:03:44.858 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-09T23:03:44.904 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:03:44.905 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:03:45.003 INFO:teuthology.orchestra.run.smithi080.stderr:Sending period to new master zone 4b05608c-7197-4e9b-bd29-7989607fa277 2022-09-09T23:03:45.063 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:03:45.063 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:03:45.064 INFO:teuthology.orchestra.run.smithi080.stdout: "epoch": 4, 2022-09-09T23:03:45.064 INFO:teuthology.orchestra.run.smithi080.stdout: "predecessor_uuid": "2efdc3a1-023b-4e78-aaa2-0e069a034dac", 2022-09-09T23:03:45.064 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": [ 2022-09-09T23:03:45.064 INFO:teuthology.orchestra.run.smithi080.stdout: "", 2022-09-09T23:03:45.065 INFO:teuthology.orchestra.run.smithi080.stdout: "", 2022-09-09T23:03:45.065 INFO:teuthology.orchestra.run.smithi080.stdout: "", 2022-09-09T23:03:45.065 INFO:teuthology.orchestra.run.smithi080.stdout: "1_1662764312.564852_2684.1" 2022-09-09T23:03:45.065 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:03:45.066 INFO:teuthology.orchestra.run.smithi080.stdout: "period_map": { 2022-09-09T23:03:45.066 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:03:45.066 INFO:teuthology.orchestra.run.smithi080.stdout: "zonegroups": [ 2022-09-09T23:03:45.067 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:03:45.067 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "test-zonegroup", 2022-09-09T23:03:45.067 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zonegroup", 2022-09-09T23:03:45.067 INFO:teuthology.orchestra.run.smithi080.stdout: "api_name": "", 2022-09-09T23:03:45.068 INFO:teuthology.orchestra.run.smithi080.stdout: "is_master": true, 2022-09-09T23:03:45.068 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T23:03:45.068 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi005.front.sepia.ceph.com:8000" 2022-09-09T23:03:45.068 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:03:45.069 INFO:teuthology.orchestra.run.smithi080.stdout: "hostnames": [], 2022-09-09T23:03:45.069 INFO:teuthology.orchestra.run.smithi080.stdout: "hostnames_s3website": [], 2022-09-09T23:03:45.070 INFO:teuthology.orchestra.run.smithi080.stdout: "master_zone": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:03:45.070 INFO:teuthology.orchestra.run.smithi080.stdout: "zones": [ 2022-09-09T23:03:45.071 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:03:45.071 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "3ff609e7-37d9-4b47-8369-34e6c99c4c03", 2022-09-09T23:03:45.072 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone4", 2022-09-09T23:03:45.072 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T23:03:45.072 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi080.front.sepia.ceph.com:8001" 2022-09-09T23:03:45.073 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:03:45.073 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": false, 2022-09-09T23:03:45.074 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": true, 2022-09-09T23:03:45.074 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:03:45.075 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": false, 2022-09-09T23:03:45.075 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "pubsub", 2022-09-09T23:03:45.076 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": false, 2022-09-09T23:03:45.076 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [ 2022-09-09T23:03:45.077 INFO:teuthology.orchestra.run.smithi080.stdout: "test-zone1" 2022-09-09T23:03:45.077 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:03:45.078 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "", 2022-09-09T23:03:45.078 INFO:teuthology.orchestra.run.smithi080.stdout: "supported_features": [ 2022-09-09T23:03:45.079 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T23:03:45.079 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:03:45.080 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:03:45.080 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:03:45.081 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:03:45.081 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone1", 2022-09-09T23:03:45.082 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T23:03:45.082 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi005.front.sepia.ceph.com:8000" 2022-09-09T23:03:45.083 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:03:45.083 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": false, 2022-09-09T23:03:45.084 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": true, 2022-09-09T23:03:45.084 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:03:45.085 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": false, 2022-09-09T23:03:45.085 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "", 2022-09-09T23:03:45.086 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": true, 2022-09-09T23:03:45.087 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [], 2022-09-09T23:03:45.087 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "", 2022-09-09T23:03:45.088 INFO:teuthology.orchestra.run.smithi080.stdout: "supported_features": [ 2022-09-09T23:03:45.088 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T23:03:45.089 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:03:45.089 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:03:45.090 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:03:45.090 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5", 2022-09-09T23:03:45.091 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone3", 2022-09-09T23:03:45.091 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T23:03:45.092 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi005.front.sepia.ceph.com:8001" 2022-09-09T23:03:45.092 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:03:45.093 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": false, 2022-09-09T23:03:45.093 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": true, 2022-09-09T23:03:45.094 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:03:45.094 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": false, 2022-09-09T23:03:45.095 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "", 2022-09-09T23:03:45.095 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": true, 2022-09-09T23:03:45.096 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [], 2022-09-09T23:03:45.096 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "", 2022-09-09T23:03:45.097 INFO:teuthology.orchestra.run.smithi080.stdout: "supported_features": [ 2022-09-09T23:03:45.097 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T23:03:45.098 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:03:45.098 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:03:45.099 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:03:45.099 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T23:03:45.099 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone2", 2022-09-09T23:03:45.100 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T23:03:45.100 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi080.front.sepia.ceph.com:8000" 2022-09-09T23:03:45.101 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:03:45.101 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": false, 2022-09-09T23:03:45.101 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": true, 2022-09-09T23:03:45.102 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:03:45.102 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": false, 2022-09-09T23:03:45.103 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "", 2022-09-09T23:03:45.103 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": true, 2022-09-09T23:03:45.104 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [], 2022-09-09T23:03:45.104 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:03:45.105 INFO:teuthology.orchestra.run.smithi080.stdout: "supported_features": [ 2022-09-09T23:03:45.105 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T23:03:45.106 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:03:45.106 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:03:45.107 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:03:45.107 INFO:teuthology.orchestra.run.smithi080.stdout: "placement_targets": [ 2022-09-09T23:03:45.108 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:03:45.110 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "default-placement", 2022-09-09T23:03:45.110 INFO:teuthology.orchestra.run.smithi080.stdout: "tags": [], 2022-09-09T23:03:45.111 INFO:teuthology.orchestra.run.smithi080.stdout: "storage_classes": [ 2022-09-09T23:03:45.111 INFO:teuthology.orchestra.run.smithi080.stdout: "STANDARD" 2022-09-09T23:03:45.111 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:03:45.112 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:03:45.112 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:03:45.112 INFO:teuthology.orchestra.run.smithi080.stdout: "default_placement": "default-placement", 2022-09-09T23:03:45.113 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T23:03:45.113 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_policy": { 2022-09-09T23:03:45.113 INFO:teuthology.orchestra.run.smithi080.stdout: "groups": [] 2022-09-09T23:03:45.113 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:03:45.113 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled_features": [ 2022-09-09T23:03:45.114 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T23:03:45.114 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:03:45.114 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:03:45.115 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:03:45.115 INFO:teuthology.orchestra.run.smithi080.stdout: "short_zone_ids": [ 2022-09-09T23:03:45.115 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:03:45.115 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "3ff609e7-37d9-4b47-8369-34e6c99c4c03", 2022-09-09T23:03:45.116 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 2593176765 2022-09-09T23:03:45.116 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:03:45.116 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:03:45.116 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:03:45.117 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 1301724575 2022-09-09T23:03:45.117 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:03:45.117 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:03:45.117 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5", 2022-09-09T23:03:45.118 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 2787153217 2022-09-09T23:03:45.118 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:03:45.118 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:03:45.118 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T23:03:45.119 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 1899625128 2022-09-09T23:03:45.119 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:03:45.119 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:03:45.119 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:03:45.120 INFO:teuthology.orchestra.run.smithi080.stdout: "master_zonegroup": "test-zonegroup", 2022-09-09T23:03:45.120 INFO:teuthology.orchestra.run.smithi080.stdout: "master_zone": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:03:45.120 INFO:teuthology.orchestra.run.smithi080.stdout: "period_config": { 2022-09-09T23:03:45.120 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_quota": { 2022-09-09T23:03:45.120 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false, 2022-09-09T23:03:45.121 INFO:teuthology.orchestra.run.smithi080.stdout: "check_on_raw": false, 2022-09-09T23:03:45.121 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size": -1, 2022-09-09T23:03:45.121 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size_kb": 0, 2022-09-09T23:03:45.121 INFO:teuthology.orchestra.run.smithi080.stdout: "max_objects": -1 2022-09-09T23:03:45.122 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:03:45.122 INFO:teuthology.orchestra.run.smithi080.stdout: "user_quota": { 2022-09-09T23:03:45.122 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false, 2022-09-09T23:03:45.122 INFO:teuthology.orchestra.run.smithi080.stdout: "check_on_raw": false, 2022-09-09T23:03:45.123 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size": -1, 2022-09-09T23:03:45.123 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size_kb": 0, 2022-09-09T23:03:45.123 INFO:teuthology.orchestra.run.smithi080.stdout: "max_objects": -1 2022-09-09T23:03:45.123 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:03:45.124 INFO:teuthology.orchestra.run.smithi080.stdout: "user_ratelimit": { 2022-09-09T23:03:45.124 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_ops": 0, 2022-09-09T23:03:45.124 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_ops": 0, 2022-09-09T23:03:45.124 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_bytes": 0, 2022-09-09T23:03:45.125 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_bytes": 0, 2022-09-09T23:03:45.125 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false 2022-09-09T23:03:45.125 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:03:45.125 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_ratelimit": { 2022-09-09T23:03:45.126 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_ops": 0, 2022-09-09T23:03:45.126 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_ops": 0, 2022-09-09T23:03:45.126 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_bytes": 0, 2022-09-09T23:03:45.127 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_bytes": 0, 2022-09-09T23:03:45.128 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false 2022-09-09T23:03:45.128 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:03:45.128 INFO:teuthology.orchestra.run.smithi080.stdout: "anonymous_ratelimit": { 2022-09-09T23:03:45.129 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_ops": 0, 2022-09-09T23:03:45.129 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_ops": 0, 2022-09-09T23:03:45.130 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_bytes": 0, 2022-09-09T23:03:45.131 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_bytes": 0, 2022-09-09T23:03:45.131 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false 2022-09-09T23:03:45.131 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:03:45.132 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:03:45.132 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T23:03:45.133 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_name": "test-realm", 2022-09-09T23:03:45.133 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:03:45.134 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:03:45.136 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to true 2022-09-09T23:04:45.138 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:04:45.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:04:45.141 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone= --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-09T23:04:45.189 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:04:45.189 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:04:45.239 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:04:45.238+0000 7f49fd21bdc0 0 failed reading obj info from .rgw.root:zone_info.4b05608c-7197-4e9b-bd29-7989607fa277: (2) No such file or directory 2022-09-09T23:04:45.240 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:04:45.238+0000 7f49fd21bdc0 0 WARNING: could not read zone params for zone id=4b05608c-7197-4e9b-bd29-7989607fa277 name=test-zone1 2022-09-09T23:04:45.240 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:04:45.238+0000 7f49fd21bdc0 0 failed reading obj info from .rgw.root:zone_info.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5: (2) No such file or directory 2022-09-09T23:04:45.241 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:04:45.238+0000 7f49fd21bdc0 0 WARNING: could not read zone params for zone id=79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5 name=test-zone3 2022-09-09T23:04:45.257 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:04:45.258 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T23:04:45.258 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone2", 2022-09-09T23:04:45.258 INFO:teuthology.orchestra.run.smithi080.stdout: "domain_root": "test-zone2.rgw.meta:root", 2022-09-09T23:04:45.259 INFO:teuthology.orchestra.run.smithi080.stdout: "control_pool": "test-zone2.rgw.control", 2022-09-09T23:04:45.259 INFO:teuthology.orchestra.run.smithi080.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2022-09-09T23:04:45.259 INFO:teuthology.orchestra.run.smithi080.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2022-09-09T23:04:45.259 INFO:teuthology.orchestra.run.smithi080.stdout: "log_pool": "test-zone2.rgw.log", 2022-09-09T23:04:45.260 INFO:teuthology.orchestra.run.smithi080.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2022-09-09T23:04:45.260 INFO:teuthology.orchestra.run.smithi080.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2022-09-09T23:04:45.260 INFO:teuthology.orchestra.run.smithi080.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2022-09-09T23:04:45.260 INFO:teuthology.orchestra.run.smithi080.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2022-09-09T23:04:45.261 INFO:teuthology.orchestra.run.smithi080.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2022-09-09T23:04:45.261 INFO:teuthology.orchestra.run.smithi080.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2022-09-09T23:04:45.261 INFO:teuthology.orchestra.run.smithi080.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2022-09-09T23:04:45.261 INFO:teuthology.orchestra.run.smithi080.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2022-09-09T23:04:45.262 INFO:teuthology.orchestra.run.smithi080.stdout: "otp_pool": "test-zone2.rgw.otp", 2022-09-09T23:04:45.262 INFO:teuthology.orchestra.run.smithi080.stdout: "system_key": { 2022-09-09T23:04:45.262 INFO:teuthology.orchestra.run.smithi080.stdout: "access_key": "3U0V7WQYC6MLEXED", 2022-09-09T23:04:45.262 INFO:teuthology.orchestra.run.smithi080.stdout: "secret_key": "jA4iNzYKmfzg78oKBQTy4mNpjERzasVo" 2022-09-09T23:04:45.263 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:04:45.263 INFO:teuthology.orchestra.run.smithi080.stdout: "placement_pools": [ 2022-09-09T23:04:45.263 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:04:45.264 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "default-placement", 2022-09-09T23:04:45.264 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:04:45.265 INFO:teuthology.orchestra.run.smithi080.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2022-09-09T23:04:45.265 INFO:teuthology.orchestra.run.smithi080.stdout: "storage_classes": { 2022-09-09T23:04:45.265 INFO:teuthology.orchestra.run.smithi080.stdout: "STANDARD": { 2022-09-09T23:04:45.265 INFO:teuthology.orchestra.run.smithi080.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2022-09-09T23:04:45.266 INFO:teuthology.orchestra.run.smithi080.stdout: "compression_type": "random" 2022-09-09T23:04:45.266 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:04:45.266 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:04:45.267 INFO:teuthology.orchestra.run.smithi080.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2022-09-09T23:04:45.267 INFO:teuthology.orchestra.run.smithi080.stdout: "index_type": 0 2022-09-09T23:04:45.267 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:04:45.267 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:04:45.268 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:04:45.268 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T23:04:45.269 INFO:teuthology.orchestra.run.smithi080.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2022-09-09T23:04:45.269 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:04:45.271 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:04:45.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:04:45.273 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-09T23:04:45.319 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:04:45.320 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:04:45.422 INFO:teuthology.orchestra.run.smithi080.stderr:Sending period to new master zone 4b05608c-7197-4e9b-bd29-7989607fa277 2022-09-09T23:04:45.484 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:04:45.484 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:04:45.485 INFO:teuthology.orchestra.run.smithi080.stdout: "epoch": 5, 2022-09-09T23:04:45.485 INFO:teuthology.orchestra.run.smithi080.stdout: "predecessor_uuid": "2efdc3a1-023b-4e78-aaa2-0e069a034dac", 2022-09-09T23:04:45.485 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": [ 2022-09-09T23:04:45.485 INFO:teuthology.orchestra.run.smithi080.stdout: "", 2022-09-09T23:04:45.486 INFO:teuthology.orchestra.run.smithi080.stdout: "", 2022-09-09T23:04:45.486 INFO:teuthology.orchestra.run.smithi080.stdout: "", 2022-09-09T23:04:45.486 INFO:teuthology.orchestra.run.smithi080.stdout: "1_1662764312.564852_2684.1" 2022-09-09T23:04:45.486 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:04:45.487 INFO:teuthology.orchestra.run.smithi080.stdout: "period_map": { 2022-09-09T23:04:45.487 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:04:45.487 INFO:teuthology.orchestra.run.smithi080.stdout: "zonegroups": [ 2022-09-09T23:04:45.487 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:04:45.488 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "test-zonegroup", 2022-09-09T23:04:45.488 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zonegroup", 2022-09-09T23:04:45.488 INFO:teuthology.orchestra.run.smithi080.stdout: "api_name": "", 2022-09-09T23:04:45.488 INFO:teuthology.orchestra.run.smithi080.stdout: "is_master": true, 2022-09-09T23:04:45.489 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T23:04:45.489 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi005.front.sepia.ceph.com:8000" 2022-09-09T23:04:45.489 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:04:45.489 INFO:teuthology.orchestra.run.smithi080.stdout: "hostnames": [], 2022-09-09T23:04:45.490 INFO:teuthology.orchestra.run.smithi080.stdout: "hostnames_s3website": [], 2022-09-09T23:04:45.490 INFO:teuthology.orchestra.run.smithi080.stdout: "master_zone": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:04:45.490 INFO:teuthology.orchestra.run.smithi080.stdout: "zones": [ 2022-09-09T23:04:45.491 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:04:45.491 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "3ff609e7-37d9-4b47-8369-34e6c99c4c03", 2022-09-09T23:04:45.491 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone4", 2022-09-09T23:04:45.491 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T23:04:45.491 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi080.front.sepia.ceph.com:8001" 2022-09-09T23:04:45.492 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:04:45.492 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": false, 2022-09-09T23:04:45.492 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": true, 2022-09-09T23:04:45.492 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:04:45.493 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": false, 2022-09-09T23:04:45.493 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "pubsub", 2022-09-09T23:04:45.493 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": false, 2022-09-09T23:04:45.493 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [ 2022-09-09T23:04:45.494 INFO:teuthology.orchestra.run.smithi080.stdout: "test-zone1" 2022-09-09T23:04:45.494 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:04:45.494 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "", 2022-09-09T23:04:45.494 INFO:teuthology.orchestra.run.smithi080.stdout: "supported_features": [ 2022-09-09T23:04:45.495 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T23:04:45.495 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:04:45.495 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:04:45.495 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:04:45.496 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:04:45.496 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone1", 2022-09-09T23:04:45.496 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T23:04:45.496 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi005.front.sepia.ceph.com:8000" 2022-09-09T23:04:45.497 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:04:45.497 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": false, 2022-09-09T23:04:45.497 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": true, 2022-09-09T23:04:45.497 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:04:45.498 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": false, 2022-09-09T23:04:45.498 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "", 2022-09-09T23:04:45.498 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": true, 2022-09-09T23:04:45.498 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [], 2022-09-09T23:04:45.499 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "", 2022-09-09T23:04:45.499 INFO:teuthology.orchestra.run.smithi080.stdout: "supported_features": [ 2022-09-09T23:04:45.499 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T23:04:45.499 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:04:45.500 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:04:45.500 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:04:45.500 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5", 2022-09-09T23:04:45.500 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone3", 2022-09-09T23:04:45.501 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T23:04:45.501 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi005.front.sepia.ceph.com:8001" 2022-09-09T23:04:45.501 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:04:45.501 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": false, 2022-09-09T23:04:45.502 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": true, 2022-09-09T23:04:45.502 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:04:45.502 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": false, 2022-09-09T23:04:45.502 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "", 2022-09-09T23:04:45.503 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": true, 2022-09-09T23:04:45.503 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [], 2022-09-09T23:04:45.503 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "", 2022-09-09T23:04:45.503 INFO:teuthology.orchestra.run.smithi080.stdout: "supported_features": [ 2022-09-09T23:04:45.504 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T23:04:45.504 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:04:45.504 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:04:45.504 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:04:45.505 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T23:04:45.505 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone2", 2022-09-09T23:04:45.505 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T23:04:45.505 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi080.front.sepia.ceph.com:8000" 2022-09-09T23:04:45.506 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:04:45.506 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": false, 2022-09-09T23:04:45.506 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": true, 2022-09-09T23:04:45.506 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:04:45.507 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": false, 2022-09-09T23:04:45.507 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "", 2022-09-09T23:04:45.507 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": true, 2022-09-09T23:04:45.507 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [], 2022-09-09T23:04:45.508 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "", 2022-09-09T23:04:45.508 INFO:teuthology.orchestra.run.smithi080.stdout: "supported_features": [ 2022-09-09T23:04:45.508 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T23:04:45.508 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:04:45.509 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:04:45.509 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:04:45.509 INFO:teuthology.orchestra.run.smithi080.stdout: "placement_targets": [ 2022-09-09T23:04:45.509 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:04:45.510 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "default-placement", 2022-09-09T23:04:45.510 INFO:teuthology.orchestra.run.smithi080.stdout: "tags": [], 2022-09-09T23:04:45.511 INFO:teuthology.orchestra.run.smithi080.stdout: "storage_classes": [ 2022-09-09T23:04:45.511 INFO:teuthology.orchestra.run.smithi080.stdout: "STANDARD" 2022-09-09T23:04:45.511 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:04:45.511 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:04:45.512 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:04:45.512 INFO:teuthology.orchestra.run.smithi080.stdout: "default_placement": "default-placement", 2022-09-09T23:04:45.512 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T23:04:45.512 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_policy": { 2022-09-09T23:04:45.513 INFO:teuthology.orchestra.run.smithi080.stdout: "groups": [] 2022-09-09T23:04:45.513 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:04:45.513 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled_features": [ 2022-09-09T23:04:45.513 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T23:04:45.514 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:04:45.514 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:04:45.514 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:04:45.514 INFO:teuthology.orchestra.run.smithi080.stdout: "short_zone_ids": [ 2022-09-09T23:04:45.515 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:04:45.515 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "3ff609e7-37d9-4b47-8369-34e6c99c4c03", 2022-09-09T23:04:45.515 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 2593176765 2022-09-09T23:04:45.515 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:04:45.516 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:04:45.516 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:04:45.516 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 1301724575 2022-09-09T23:04:45.516 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:04:45.517 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:04:45.517 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5", 2022-09-09T23:04:45.517 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 2787153217 2022-09-09T23:04:45.518 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:04:45.518 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:04:45.518 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T23:04:45.518 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 1899625128 2022-09-09T23:04:45.519 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:04:45.519 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:04:45.519 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:04:45.519 INFO:teuthology.orchestra.run.smithi080.stdout: "master_zonegroup": "test-zonegroup", 2022-09-09T23:04:45.520 INFO:teuthology.orchestra.run.smithi080.stdout: "master_zone": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:04:45.520 INFO:teuthology.orchestra.run.smithi080.stdout: "period_config": { 2022-09-09T23:04:45.520 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_quota": { 2022-09-09T23:04:45.520 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false, 2022-09-09T23:04:45.521 INFO:teuthology.orchestra.run.smithi080.stdout: "check_on_raw": false, 2022-09-09T23:04:45.521 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size": -1, 2022-09-09T23:04:45.521 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size_kb": 0, 2022-09-09T23:04:45.521 INFO:teuthology.orchestra.run.smithi080.stdout: "max_objects": -1 2022-09-09T23:04:45.522 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:04:45.522 INFO:teuthology.orchestra.run.smithi080.stdout: "user_quota": { 2022-09-09T23:04:45.522 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false, 2022-09-09T23:04:45.522 INFO:teuthology.orchestra.run.smithi080.stdout: "check_on_raw": false, 2022-09-09T23:04:45.523 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size": -1, 2022-09-09T23:04:45.523 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size_kb": 0, 2022-09-09T23:04:45.523 INFO:teuthology.orchestra.run.smithi080.stdout: "max_objects": -1 2022-09-09T23:04:45.523 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:04:45.524 INFO:teuthology.orchestra.run.smithi080.stdout: "user_ratelimit": { 2022-09-09T23:04:45.524 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_ops": 0, 2022-09-09T23:04:45.524 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_ops": 0, 2022-09-09T23:04:45.524 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_bytes": 0, 2022-09-09T23:04:45.525 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_bytes": 0, 2022-09-09T23:04:45.525 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false 2022-09-09T23:04:45.525 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:04:45.525 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_ratelimit": { 2022-09-09T23:04:45.526 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_ops": 0, 2022-09-09T23:04:45.526 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_ops": 0, 2022-09-09T23:04:45.526 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_bytes": 0, 2022-09-09T23:04:45.526 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_bytes": 0, 2022-09-09T23:04:45.527 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false 2022-09-09T23:04:45.527 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:04:45.527 INFO:teuthology.orchestra.run.smithi080.stdout: "anonymous_ratelimit": { 2022-09-09T23:04:45.527 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_ops": 0, 2022-09-09T23:04:45.528 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_ops": 0, 2022-09-09T23:04:45.528 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_bytes": 0, 2022-09-09T23:04:45.528 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_bytes": 0, 2022-09-09T23:04:45.528 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false 2022-09-09T23:04:45.529 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:04:45.529 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:04:45.529 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T23:04:45.529 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_name": "test-realm", 2022-09-09T23:04:45.530 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:04:45.530 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:04:45.531 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "" 2022-09-09T23:05:45.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... ok 2022-09-09T23:05:45.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:05:45.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:05:45.538 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:05:45.586 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:05:45.586 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:05:45.645 INFO:teuthology.orchestra.run.smithi005.stderr:No --period given, using current period=fd4bc158-06fd-4840-8a27-33fabda67b99 2022-09-09T23:05:45.667 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T23:05:45.667 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:45.668 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T23:05:45.668 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T23:05:45.668 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:45.669 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:45.669 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T23:05:45.669 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T23:05:45.670 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:45.670 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:45.671 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T23:05:45.671 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T23:05:45.671 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:45.672 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:45.672 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662764501.249520_420.1", 2022-09-09T23:05:45.673 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T23:01:41.249520Z" 2022-09-09T23:05:45.673 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:05:45.673 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T23:05:45.675 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:05:45.676 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-09T23:05:45.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:05:45.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:05:45.679 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:05:45.725 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:05:45.725 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:05:45.780 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:05:45.777+0000 7fa5c9afedc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:05:45.781 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:05:45.781+0000 7fa5c9afedc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:05:45.810 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:05:45.811 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:05:45.811 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:05:45.812 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:05:45.812 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:05:45.812 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:05:45.813 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:05:45.813 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:45.813 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:05:45.813 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:05:45.814 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:05:45.814 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:05:45.814 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:05:45.814 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:05:45.815 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:05:45.815 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:05:45.815 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:05:45.816 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:05:45.816 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:05:45.816 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:05:45.817 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:45.817 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:05:45.817 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:05:45.817 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:05:45.818 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:05:45.818 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:05:45.818 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:05:45.818 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:05:45.819 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:05:45.819 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:05:45.819 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:05:45.819 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:05:45.820 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:45.820 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:05:45.820 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:05:45.820 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:05:45.821 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:05:45.821 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:05:45.821 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:05:45.821 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:05:45.822 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:05:45.822 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:05:45.822 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:05:45.822 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:05:45.823 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:45.823 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:05:45.823 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:05:45.823 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:05:45.824 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:05:45.824 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764501.249520_420.1", 2022-09-09T23:05:45.824 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:05:45.824 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:05:45.825 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:05:45.825 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:01:41.249520Z", 2022-09-09T23:05:45.825 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:05:45.825 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:05:45.826 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:05:45.826 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:05:45.826 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:45.826 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:05:45.827 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:05:45.827 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:05:45.827 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:45.827 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:05:45Z" 2022-09-09T23:05:45.828 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:05:45.829 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764501.249520_420.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:01:41.249520Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:05:45Z" } 2022-09-09T23:05:45.830 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764501.249520_420.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:01:41.249520Z', 'realm_epoch': 4}}] 2022-09-09T23:05:45.830 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:05:45.831 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:05:45.832 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-09T23:05:45.832 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-09T23:05:45.833 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:05:45.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:05:45.834 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:05:45.881 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:05:45.882 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:05:45.936 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T23:05:45.937+0000 7f10e72c7dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:05:45.939 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T23:05:45.937+0000 7f10e72c7dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:05:45.964 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T23:05:45.964 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T23:05:45.964 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T23:05:45.965 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T23:05:45.965 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T23:05:45.965 INFO:teuthology.orchestra.run.smithi005.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:05:45.965 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 4 2022-09-09T23:05:45.966 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:45.966 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T23:05:45.966 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:45.967 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T23:05:45.967 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:05:45.968 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T23:05:45.968 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:05:45.968 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:05:45.968 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 1, 2022-09-09T23:05:45.969 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:05:45.969 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:05:45.969 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T23:05:45.970 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:05:45.970 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:45.970 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:45.970 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T23:05:45.971 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:05:45.971 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T23:05:45.971 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T23:05:45.971 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:05:45.972 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:05:45.972 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:05:45.972 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T23:05:45.973 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T23:05:45.973 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:05:45.973 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:45.973 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:45.974 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T23:05:45.974 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:05:45.974 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T23:05:45.974 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:05:45.975 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:05:45.975 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:05:45.975 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:05:45.975 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:05:45.976 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T23:05:45.976 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:05:45.976 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:45.977 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:45.977 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T23:05:45.977 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:05:45.977 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T23:05:45.978 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662764501.249520_420.1", 2022-09-09T23:05:45.978 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:05:45.978 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:05:45.978 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:05:45.979 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T23:01:41.249520Z", 2022-09-09T23:05:45.979 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 4 2022-09-09T23:05:45.979 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:05:45.979 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:05:45.980 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:45.980 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:45.980 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T23:05:45.981 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 1, 2022-09-09T23:05:45.981 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 1 2022-09-09T23:05:45.981 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:45.982 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T23:05:45Z" 2022-09-09T23:05:45.982 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T23:05:45.983 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764501.249520_420.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:01:41.249520Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:05:45Z" } 2022-09-09T23:05:45.984 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764501.249520_420.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:01:41.249520Z', 'realm_epoch': 4}}] 2022-09-09T23:05:45.985 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:05:45.985 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:05:45.986 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-09T23:05:45.986 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-09T23:05:45.987 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:05:45.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:05:45.988 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:05:46.036 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:05:46.036 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:05:46.095 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:05:46.093+0000 7f50c7615dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:05:46.097 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:05:46.097+0000 7f50c7615dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:05:46.125 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:05:46.125 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:05:46.125 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:05:46.126 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:05:46.126 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:05:46.126 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:05:46.126 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:05:46.127 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:46.127 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:05:46.127 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:05:46.127 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:05:46.128 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:05:46.128 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:05:46.128 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:05:46.128 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:05:46.129 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:05:46.129 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:05:46.129 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:05:46.130 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:05:46.130 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:05:46.130 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:46.130 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:05:46.131 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:05:46.131 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:05:46.131 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:05:46.131 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:05:46.132 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:05:46.132 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:05:46.132 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:05:46.132 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:05:46.133 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:05:46.133 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:05:46.133 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:46.133 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:05:46.134 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:05:46.134 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:05:46.134 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:05:46.135 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:05:46.135 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:05:46.135 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:05:46.135 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:05:46.136 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:05:46.136 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:05:46.136 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:05:46.136 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:46.137 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:05:46.137 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:05:46.137 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:05:46.137 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:05:46.138 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764501.249520_420.1", 2022-09-09T23:05:46.138 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:05:46.138 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:05:46.138 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:05:46.139 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:01:41.249520Z", 2022-09-09T23:05:46.139 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:05:46.139 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:05:46.140 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:05:46.140 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:05:46.140 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:46.140 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:05:46.141 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:05:46.141 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:05:46.141 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:46.141 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:05:46Z" 2022-09-09T23:05:46.142 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:05:46.143 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764501.249520_420.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:01:41.249520Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:05:46Z" } 2022-09-09T23:05:46.144 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764501.249520_420.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:01:41.249520Z', 'realm_epoch': 4}}] 2022-09-09T23:05:46.144 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:05:46.145 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:05:46.145 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-09T23:05:46.146 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '3U0V7WQYC6MLEXED', '--secret', 'jA4iNzYKmfzg78oKBQTy4mNpjERzasVo', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:05:46.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '3U0V7WQYC6MLEXED', '--secret', 'jA4iNzYKmfzg78oKBQTy4mNpjERzasVo', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:05:46.147 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key 3U0V7WQYC6MLEXED --secret jA4iNzYKmfzg78oKBQTy4mNpjERzasVo --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-09T23:05:46.198 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:05:46.199 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:05:46.255 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:05:46.253+0000 7f1cc1407dc0 0 failed reading obj info from .rgw.root:zone_info.4b05608c-7197-4e9b-bd29-7989607fa277: (2) No such file or directory 2022-09-09T23:05:46.256 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:05:46.253+0000 7f1cc1407dc0 0 WARNING: could not read zone params for zone id=4b05608c-7197-4e9b-bd29-7989607fa277 name=test-zone1 2022-09-09T23:05:46.256 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:05:46.253+0000 7f1cc1407dc0 0 failed reading obj info from .rgw.root:zone_info.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5: (2) No such file or directory 2022-09-09T23:05:46.257 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:05:46.253+0000 7f1cc1407dc0 0 WARNING: could not read zone params for zone id=79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5 name=test-zone3 2022-09-09T23:05:46.265 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:05:46.265 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "f32687a9-8221-4f9d-9725-f38abe378e53", 2022-09-09T23:05:46.265 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "remove", 2022-09-09T23:05:46.266 INFO:teuthology.orchestra.run.smithi080.stdout: "domain_root": "remove.rgw.meta:root", 2022-09-09T23:05:46.266 INFO:teuthology.orchestra.run.smithi080.stdout: "control_pool": "remove.rgw.control", 2022-09-09T23:05:46.266 INFO:teuthology.orchestra.run.smithi080.stdout: "gc_pool": "remove.rgw.log:gc", 2022-09-09T23:05:46.267 INFO:teuthology.orchestra.run.smithi080.stdout: "lc_pool": "remove.rgw.log:lc", 2022-09-09T23:05:46.267 INFO:teuthology.orchestra.run.smithi080.stdout: "log_pool": "remove.rgw.log", 2022-09-09T23:05:46.267 INFO:teuthology.orchestra.run.smithi080.stdout: "intent_log_pool": "remove.rgw.log:intent", 2022-09-09T23:05:46.267 INFO:teuthology.orchestra.run.smithi080.stdout: "usage_log_pool": "remove.rgw.log:usage", 2022-09-09T23:05:46.268 INFO:teuthology.orchestra.run.smithi080.stdout: "roles_pool": "remove.rgw.meta:roles", 2022-09-09T23:05:46.268 INFO:teuthology.orchestra.run.smithi080.stdout: "reshard_pool": "remove.rgw.log:reshard", 2022-09-09T23:05:46.268 INFO:teuthology.orchestra.run.smithi080.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2022-09-09T23:05:46.268 INFO:teuthology.orchestra.run.smithi080.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2022-09-09T23:05:46.269 INFO:teuthology.orchestra.run.smithi080.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2022-09-09T23:05:46.269 INFO:teuthology.orchestra.run.smithi080.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2022-09-09T23:05:46.269 INFO:teuthology.orchestra.run.smithi080.stdout: "otp_pool": "remove.rgw.otp", 2022-09-09T23:05:46.270 INFO:teuthology.orchestra.run.smithi080.stdout: "system_key": { 2022-09-09T23:05:46.270 INFO:teuthology.orchestra.run.smithi080.stdout: "access_key": "3U0V7WQYC6MLEXED", 2022-09-09T23:05:46.270 INFO:teuthology.orchestra.run.smithi080.stdout: "secret_key": "jA4iNzYKmfzg78oKBQTy4mNpjERzasVo" 2022-09-09T23:05:46.270 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:46.271 INFO:teuthology.orchestra.run.smithi080.stdout: "placement_pools": [ 2022-09-09T23:05:46.271 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:05:46.271 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "default-placement", 2022-09-09T23:05:46.271 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:05:46.272 INFO:teuthology.orchestra.run.smithi080.stdout: "index_pool": "remove.rgw.buckets.index", 2022-09-09T23:05:46.272 INFO:teuthology.orchestra.run.smithi080.stdout: "storage_classes": { 2022-09-09T23:05:46.272 INFO:teuthology.orchestra.run.smithi080.stdout: "STANDARD": { 2022-09-09T23:05:46.272 INFO:teuthology.orchestra.run.smithi080.stdout: "data_pool": "remove.rgw.buckets.data" 2022-09-09T23:05:46.273 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:05:46.273 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:46.273 INFO:teuthology.orchestra.run.smithi080.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2022-09-09T23:05:46.273 INFO:teuthology.orchestra.run.smithi080.stdout: "index_type": 0 2022-09-09T23:05:46.274 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:05:46.274 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:05:46.274 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:05:46.274 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T23:05:46.275 INFO:teuthology.orchestra.run.smithi080.stdout: "notif_pool": "remove.rgw.log:notif" 2022-09-09T23:05:46.275 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:05:46.276 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:05:46.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:05:46.277 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-09T23:05:46.322 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:05:46.323 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:05:46.342 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:05:46.341+0000 7f910a1d8dc0 0 period (fd4bc158-06fd-4840-8a27-33fabda67b99 does not have zone f32687a9-8221-4f9d-9725-f38abe378e53 configured 2022-09-09T23:05:52.862 INFO:teuthology.orchestra.run.smithi080.stderr:Sending period to new master zone 4b05608c-7197-4e9b-bd29-7989607fa277 2022-09-09T23:05:52.923 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:05:52.923 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:05:52.924 INFO:teuthology.orchestra.run.smithi080.stdout: "epoch": 6, 2022-09-09T23:05:52.924 INFO:teuthology.orchestra.run.smithi080.stdout: "predecessor_uuid": "2efdc3a1-023b-4e78-aaa2-0e069a034dac", 2022-09-09T23:05:52.924 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": [ 2022-09-09T23:05:52.925 INFO:teuthology.orchestra.run.smithi080.stdout: "", 2022-09-09T23:05:52.925 INFO:teuthology.orchestra.run.smithi080.stdout: "", 2022-09-09T23:05:52.925 INFO:teuthology.orchestra.run.smithi080.stdout: "", 2022-09-09T23:05:52.925 INFO:teuthology.orchestra.run.smithi080.stdout: "1_1662764312.564852_2684.1" 2022-09-09T23:05:52.926 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:05:52.926 INFO:teuthology.orchestra.run.smithi080.stdout: "period_map": { 2022-09-09T23:05:52.926 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:05:52.927 INFO:teuthology.orchestra.run.smithi080.stdout: "zonegroups": [ 2022-09-09T23:05:52.927 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:05:52.927 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "test-zonegroup", 2022-09-09T23:05:52.927 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zonegroup", 2022-09-09T23:05:52.928 INFO:teuthology.orchestra.run.smithi080.stdout: "api_name": "", 2022-09-09T23:05:52.928 INFO:teuthology.orchestra.run.smithi080.stdout: "is_master": true, 2022-09-09T23:05:52.929 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T23:05:52.929 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi005.front.sepia.ceph.com:8000" 2022-09-09T23:05:52.929 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:05:52.929 INFO:teuthology.orchestra.run.smithi080.stdout: "hostnames": [], 2022-09-09T23:05:52.930 INFO:teuthology.orchestra.run.smithi080.stdout: "hostnames_s3website": [], 2022-09-09T23:05:52.930 INFO:teuthology.orchestra.run.smithi080.stdout: "master_zone": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:05:52.930 INFO:teuthology.orchestra.run.smithi080.stdout: "zones": [ 2022-09-09T23:05:52.930 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:05:52.931 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "3ff609e7-37d9-4b47-8369-34e6c99c4c03", 2022-09-09T23:05:52.931 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone4", 2022-09-09T23:05:52.931 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T23:05:52.931 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi080.front.sepia.ceph.com:8001" 2022-09-09T23:05:52.932 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:05:52.932 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": false, 2022-09-09T23:05:52.932 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": true, 2022-09-09T23:05:52.933 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:05:52.933 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": false, 2022-09-09T23:05:52.933 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "pubsub", 2022-09-09T23:05:52.934 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": false, 2022-09-09T23:05:52.934 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [ 2022-09-09T23:05:52.934 INFO:teuthology.orchestra.run.smithi080.stdout: "test-zone1" 2022-09-09T23:05:52.935 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:05:52.935 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "", 2022-09-09T23:05:52.935 INFO:teuthology.orchestra.run.smithi080.stdout: "supported_features": [ 2022-09-09T23:05:52.936 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T23:05:52.936 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:05:52.936 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:52.936 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:05:52.937 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:05:52.937 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone1", 2022-09-09T23:05:52.937 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T23:05:52.937 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi005.front.sepia.ceph.com:8000" 2022-09-09T23:05:52.938 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:05:52.939 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": false, 2022-09-09T23:05:52.940 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": true, 2022-09-09T23:05:52.940 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:05:52.941 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": false, 2022-09-09T23:05:52.941 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "", 2022-09-09T23:05:52.942 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": true, 2022-09-09T23:05:52.942 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [], 2022-09-09T23:05:52.943 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "", 2022-09-09T23:05:52.943 INFO:teuthology.orchestra.run.smithi080.stdout: "supported_features": [ 2022-09-09T23:05:52.943 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T23:05:52.944 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:05:52.944 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:52.945 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:05:52.946 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5", 2022-09-09T23:05:52.946 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone3", 2022-09-09T23:05:52.947 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T23:05:52.947 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi005.front.sepia.ceph.com:8001" 2022-09-09T23:05:52.947 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:05:52.947 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": false, 2022-09-09T23:05:52.948 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": true, 2022-09-09T23:05:52.948 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:05:52.948 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": false, 2022-09-09T23:05:52.948 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "", 2022-09-09T23:05:52.949 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": true, 2022-09-09T23:05:52.949 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [], 2022-09-09T23:05:52.949 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "", 2022-09-09T23:05:52.950 INFO:teuthology.orchestra.run.smithi080.stdout: "supported_features": [ 2022-09-09T23:05:52.950 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T23:05:52.950 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:05:52.950 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:52.951 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:05:52.951 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T23:05:52.951 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "test-zone2", 2022-09-09T23:05:52.951 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [ 2022-09-09T23:05:52.952 INFO:teuthology.orchestra.run.smithi080.stdout: "http://smithi080.front.sepia.ceph.com:8000" 2022-09-09T23:05:52.952 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:05:52.952 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": false, 2022-09-09T23:05:52.952 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": true, 2022-09-09T23:05:52.953 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:05:52.953 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": false, 2022-09-09T23:05:52.953 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "", 2022-09-09T23:05:52.953 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": true, 2022-09-09T23:05:52.954 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [], 2022-09-09T23:05:52.954 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "", 2022-09-09T23:05:52.954 INFO:teuthology.orchestra.run.smithi080.stdout: "supported_features": [ 2022-09-09T23:05:52.954 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T23:05:52.955 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:05:52.955 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:52.955 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:05:52.955 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "f32687a9-8221-4f9d-9725-f38abe378e53", 2022-09-09T23:05:52.956 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "remove", 2022-09-09T23:05:52.957 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [], 2022-09-09T23:05:52.958 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": false, 2022-09-09T23:05:52.958 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": true, 2022-09-09T23:05:52.958 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:05:52.959 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": false, 2022-09-09T23:05:52.959 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "", 2022-09-09T23:05:52.959 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": true, 2022-09-09T23:05:52.960 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [], 2022-09-09T23:05:52.960 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "", 2022-09-09T23:05:52.960 INFO:teuthology.orchestra.run.smithi080.stdout: "supported_features": [ 2022-09-09T23:05:52.960 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T23:05:52.961 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:05:52.961 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:05:52.961 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:05:52.961 INFO:teuthology.orchestra.run.smithi080.stdout: "placement_targets": [ 2022-09-09T23:05:52.962 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:05:52.962 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "default-placement", 2022-09-09T23:05:52.962 INFO:teuthology.orchestra.run.smithi080.stdout: "tags": [], 2022-09-09T23:05:52.962 INFO:teuthology.orchestra.run.smithi080.stdout: "storage_classes": [ 2022-09-09T23:05:52.963 INFO:teuthology.orchestra.run.smithi080.stdout: "STANDARD" 2022-09-09T23:05:52.963 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:05:52.963 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:05:52.964 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:05:52.964 INFO:teuthology.orchestra.run.smithi080.stdout: "default_placement": "default-placement", 2022-09-09T23:05:52.964 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T23:05:52.964 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_policy": { 2022-09-09T23:05:52.965 INFO:teuthology.orchestra.run.smithi080.stdout: "groups": [] 2022-09-09T23:05:52.965 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:52.965 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled_features": [ 2022-09-09T23:05:52.965 INFO:teuthology.orchestra.run.smithi080.stdout: "resharding" 2022-09-09T23:05:52.966 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:05:52.966 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:05:52.966 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2022-09-09T23:05:52.966 INFO:teuthology.orchestra.run.smithi080.stdout: "short_zone_ids": [ 2022-09-09T23:05:52.967 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:05:52.967 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "3ff609e7-37d9-4b47-8369-34e6c99c4c03", 2022-09-09T23:05:52.967 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 2593176765 2022-09-09T23:05:52.967 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:52.968 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:05:52.968 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:05:52.968 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 1301724575 2022-09-09T23:05:52.969 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:52.969 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:05:52.969 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5", 2022-09-09T23:05:52.970 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 2787153217 2022-09-09T23:05:52.970 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:52.970 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:05:52.970 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T23:05:52.971 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 1899625128 2022-09-09T23:05:52.971 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:52.971 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:05:52.971 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "f32687a9-8221-4f9d-9725-f38abe378e53", 2022-09-09T23:05:52.972 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 1889407079 2022-09-09T23:05:52.972 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:05:52.972 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:05:52.973 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:52.973 INFO:teuthology.orchestra.run.smithi080.stdout: "master_zonegroup": "test-zonegroup", 2022-09-09T23:05:52.973 INFO:teuthology.orchestra.run.smithi080.stdout: "master_zone": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:05:52.973 INFO:teuthology.orchestra.run.smithi080.stdout: "period_config": { 2022-09-09T23:05:52.974 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_quota": { 2022-09-09T23:05:52.974 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false, 2022-09-09T23:05:52.974 INFO:teuthology.orchestra.run.smithi080.stdout: "check_on_raw": false, 2022-09-09T23:05:52.974 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size": -1, 2022-09-09T23:05:52.975 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size_kb": 0, 2022-09-09T23:05:52.975 INFO:teuthology.orchestra.run.smithi080.stdout: "max_objects": -1 2022-09-09T23:05:52.975 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:52.975 INFO:teuthology.orchestra.run.smithi080.stdout: "user_quota": { 2022-09-09T23:05:52.976 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false, 2022-09-09T23:05:52.976 INFO:teuthology.orchestra.run.smithi080.stdout: "check_on_raw": false, 2022-09-09T23:05:52.976 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size": -1, 2022-09-09T23:05:52.977 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size_kb": 0, 2022-09-09T23:05:52.977 INFO:teuthology.orchestra.run.smithi080.stdout: "max_objects": -1 2022-09-09T23:05:52.977 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:52.977 INFO:teuthology.orchestra.run.smithi080.stdout: "user_ratelimit": { 2022-09-09T23:05:52.978 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_ops": 0, 2022-09-09T23:05:52.978 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_ops": 0, 2022-09-09T23:05:52.978 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_bytes": 0, 2022-09-09T23:05:52.979 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_bytes": 0, 2022-09-09T23:05:52.979 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false 2022-09-09T23:05:52.979 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:52.979 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_ratelimit": { 2022-09-09T23:05:52.980 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_ops": 0, 2022-09-09T23:05:52.980 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_ops": 0, 2022-09-09T23:05:52.980 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_bytes": 0, 2022-09-09T23:05:52.981 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_bytes": 0, 2022-09-09T23:05:52.981 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false 2022-09-09T23:05:52.981 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:52.981 INFO:teuthology.orchestra.run.smithi080.stdout: "anonymous_ratelimit": { 2022-09-09T23:05:52.982 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_ops": 0, 2022-09-09T23:05:52.982 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_ops": 0, 2022-09-09T23:05:52.982 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_bytes": 0, 2022-09-09T23:05:52.983 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_bytes": 0, 2022-09-09T23:05:52.983 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false 2022-09-09T23:05:52.983 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:05:52.983 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:52.984 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T23:05:52.984 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_name": "test-realm", 2022-09-09T23:05:52.984 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:05:52.984 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:05:52.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:05:52.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:05:52.986 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-09T23:05:53.035 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:05:53.036 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:05:53.059 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T23:05:53.057+0000 7f870eb7edc0 0 failed reading obj info from .rgw.root:zone_info.3ff609e7-37d9-4b47-8369-34e6c99c4c03: (2) No such file or directory 2022-09-09T23:05:53.059 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T23:05:53.057+0000 7f870eb7edc0 0 WARNING: could not read zone params for zone id=3ff609e7-37d9-4b47-8369-34e6c99c4c03 name=test-zone4 2022-09-09T23:05:53.060 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T23:05:53.061+0000 7f870eb7edc0 0 failed reading obj info from .rgw.root:zone_info.cc1fc6ed-f06f-45bc-961e-adec328c8651: (2) No such file or directory 2022-09-09T23:05:53.060 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T23:05:53.061+0000 7f870eb7edc0 0 WARNING: could not read zone params for zone id=cc1fc6ed-f06f-45bc-961e-adec328c8651 name=test-zone2 2022-09-09T23:05:53.066 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T23:05:53.066 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "test-zonegroup", 2022-09-09T23:05:53.067 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zonegroup", 2022-09-09T23:05:53.067 INFO:teuthology.orchestra.run.smithi005.stdout: "api_name": "", 2022-09-09T23:05:53.067 INFO:teuthology.orchestra.run.smithi005.stdout: "is_master": true, 2022-09-09T23:05:53.067 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T23:05:53.068 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi005.front.sepia.ceph.com:8000" 2022-09-09T23:05:53.068 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:53.068 INFO:teuthology.orchestra.run.smithi005.stdout: "hostnames": [], 2022-09-09T23:05:53.068 INFO:teuthology.orchestra.run.smithi005.stdout: "hostnames_s3website": [], 2022-09-09T23:05:53.069 INFO:teuthology.orchestra.run.smithi005.stdout: "master_zone": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:05:53.069 INFO:teuthology.orchestra.run.smithi005.stdout: "zones": [ 2022-09-09T23:05:53.069 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:53.070 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "3ff609e7-37d9-4b47-8369-34e6c99c4c03", 2022-09-09T23:05:53.070 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zone4", 2022-09-09T23:05:53.070 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T23:05:53.071 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi080.front.sepia.ceph.com:8001" 2022-09-09T23:05:53.071 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:53.071 INFO:teuthology.orchestra.run.smithi005.stdout: "log_meta": false, 2022-09-09T23:05:53.071 INFO:teuthology.orchestra.run.smithi005.stdout: "log_data": true, 2022-09-09T23:05:53.072 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:05:53.072 INFO:teuthology.orchestra.run.smithi005.stdout: "read_only": false, 2022-09-09T23:05:53.072 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_type": "pubsub", 2022-09-09T23:05:53.072 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from_all": false, 2022-09-09T23:05:53.073 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from": [ 2022-09-09T23:05:53.073 INFO:teuthology.orchestra.run.smithi005.stdout: "test-zone1" 2022-09-09T23:05:53.073 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:53.073 INFO:teuthology.orchestra.run.smithi005.stdout: "redirect_zone": "", 2022-09-09T23:05:53.074 INFO:teuthology.orchestra.run.smithi005.stdout: "supported_features": [ 2022-09-09T23:05:53.074 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T23:05:53.074 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:53.074 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.075 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:53.075 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:05:53.075 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zone1", 2022-09-09T23:05:53.075 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T23:05:53.076 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi005.front.sepia.ceph.com:8000" 2022-09-09T23:05:53.076 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:53.076 INFO:teuthology.orchestra.run.smithi005.stdout: "log_meta": false, 2022-09-09T23:05:53.076 INFO:teuthology.orchestra.run.smithi005.stdout: "log_data": true, 2022-09-09T23:05:53.077 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:05:53.077 INFO:teuthology.orchestra.run.smithi005.stdout: "read_only": false, 2022-09-09T23:05:53.077 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_type": "", 2022-09-09T23:05:53.077 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from_all": true, 2022-09-09T23:05:53.078 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from": [], 2022-09-09T23:05:53.078 INFO:teuthology.orchestra.run.smithi005.stdout: "redirect_zone": "", 2022-09-09T23:05:53.078 INFO:teuthology.orchestra.run.smithi005.stdout: "supported_features": [ 2022-09-09T23:05:53.078 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T23:05:53.079 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:53.079 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.079 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:53.079 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5", 2022-09-09T23:05:53.079 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zone3", 2022-09-09T23:05:53.080 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T23:05:53.080 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi005.front.sepia.ceph.com:8001" 2022-09-09T23:05:53.080 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:53.081 INFO:teuthology.orchestra.run.smithi005.stdout: "log_meta": false, 2022-09-09T23:05:53.081 INFO:teuthology.orchestra.run.smithi005.stdout: "log_data": true, 2022-09-09T23:05:53.081 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:05:53.082 INFO:teuthology.orchestra.run.smithi005.stdout: "read_only": false, 2022-09-09T23:05:53.082 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_type": "", 2022-09-09T23:05:53.082 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from_all": true, 2022-09-09T23:05:53.083 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from": [], 2022-09-09T23:05:53.083 INFO:teuthology.orchestra.run.smithi005.stdout: "redirect_zone": "", 2022-09-09T23:05:53.083 INFO:teuthology.orchestra.run.smithi005.stdout: "supported_features": [ 2022-09-09T23:05:53.084 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T23:05:53.084 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:53.084 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.084 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:53.084 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T23:05:53.085 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zone2", 2022-09-09T23:05:53.085 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T23:05:53.085 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi080.front.sepia.ceph.com:8000" 2022-09-09T23:05:53.085 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:53.086 INFO:teuthology.orchestra.run.smithi005.stdout: "log_meta": false, 2022-09-09T23:05:53.086 INFO:teuthology.orchestra.run.smithi005.stdout: "log_data": true, 2022-09-09T23:05:53.086 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:05:53.087 INFO:teuthology.orchestra.run.smithi005.stdout: "read_only": false, 2022-09-09T23:05:53.087 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_type": "", 2022-09-09T23:05:53.087 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from_all": true, 2022-09-09T23:05:53.087 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from": [], 2022-09-09T23:05:53.087 INFO:teuthology.orchestra.run.smithi005.stdout: "redirect_zone": "", 2022-09-09T23:05:53.088 INFO:teuthology.orchestra.run.smithi005.stdout: "supported_features": [ 2022-09-09T23:05:53.088 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T23:05:53.088 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:53.088 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:05:53.089 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:53.089 INFO:teuthology.orchestra.run.smithi005.stdout: "placement_targets": [ 2022-09-09T23:05:53.089 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:53.090 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "default-placement", 2022-09-09T23:05:53.090 INFO:teuthology.orchestra.run.smithi005.stdout: "tags": [], 2022-09-09T23:05:53.090 INFO:teuthology.orchestra.run.smithi005.stdout: "storage_classes": [ 2022-09-09T23:05:53.090 INFO:teuthology.orchestra.run.smithi005.stdout: "STANDARD" 2022-09-09T23:05:53.091 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:53.091 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:05:53.091 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:53.091 INFO:teuthology.orchestra.run.smithi005.stdout: "default_placement": "default-placement", 2022-09-09T23:05:53.092 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T23:05:53.092 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_policy": { 2022-09-09T23:05:53.092 INFO:teuthology.orchestra.run.smithi005.stdout: "groups": [] 2022-09-09T23:05:53.093 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.093 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled_features": [ 2022-09-09T23:05:53.093 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T23:05:53.094 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:53.094 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T23:05:53.095 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:05:53.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:05:53.097 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-09T23:05:53.143 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:05:53.144 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:05:53.163 INFO:teuthology.orchestra.run.smithi005.stderr:zone name remove not found in zonegroup test-zonegroup 2022-09-09T23:05:53.171 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-09-09T23:05:53.173 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:05:53.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:05:53.174 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone delete --rgw-zone remove --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-09T23:05:53.214 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:05:53.214 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:05:53.286 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:05:53.284+0000 7ff87a58cdc0 0 failed reading obj info from .rgw.root:zone_info.4b05608c-7197-4e9b-bd29-7989607fa277: (2) No such file or directory 2022-09-09T23:05:53.287 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:05:53.284+0000 7ff87a58cdc0 0 WARNING: could not read zone params for zone id=4b05608c-7197-4e9b-bd29-7989607fa277 name=test-zone1 2022-09-09T23:05:53.287 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:05:53.284+0000 7ff87a58cdc0 0 failed reading obj info from .rgw.root:zone_info.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5: (2) No such file or directory 2022-09-09T23:05:53.287 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:05:53.284+0000 7ff87a58cdc0 0 WARNING: could not read zone params for zone id=79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5 name=test-zone3 2022-09-09T23:05:53.292 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:05:53.288+0000 7ff87a58cdc0 0 zone id f32687a9-8221-4f9d-9725-f38abe378e53 is not a part of zonegroup default 2022-09-09T23:05:53.305 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:05:53.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:05:53.307 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-09T23:05:53.356 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:05:53.357 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:05:53.379 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T23:05:53.377+0000 7f43a318adc0 0 WARNING: can't generate connection for zone f32687a9-8221-4f9d-9725-f38abe378e53 id remove: no endpoints defined 2022-09-09T23:05:53.442 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T23:05:53.443 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:05:53.443 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 7, 2022-09-09T23:05:53.443 INFO:teuthology.orchestra.run.smithi005.stdout: "predecessor_uuid": "2efdc3a1-023b-4e78-aaa2-0e069a034dac", 2022-09-09T23:05:53.443 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": [ 2022-09-09T23:05:53.444 INFO:teuthology.orchestra.run.smithi005.stdout: "", 2022-09-09T23:05:53.444 INFO:teuthology.orchestra.run.smithi005.stdout: "", 2022-09-09T23:05:53.444 INFO:teuthology.orchestra.run.smithi005.stdout: "", 2022-09-09T23:05:53.444 INFO:teuthology.orchestra.run.smithi005.stdout: "1_1662764312.564852_2684.1" 2022-09-09T23:05:53.445 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:53.445 INFO:teuthology.orchestra.run.smithi005.stdout: "period_map": { 2022-09-09T23:05:53.445 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:05:53.445 INFO:teuthology.orchestra.run.smithi005.stdout: "zonegroups": [ 2022-09-09T23:05:53.446 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:53.446 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "test-zonegroup", 2022-09-09T23:05:53.446 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zonegroup", 2022-09-09T23:05:53.447 INFO:teuthology.orchestra.run.smithi005.stdout: "api_name": "", 2022-09-09T23:05:53.447 INFO:teuthology.orchestra.run.smithi005.stdout: "is_master": true, 2022-09-09T23:05:53.447 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T23:05:53.447 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi005.front.sepia.ceph.com:8000" 2022-09-09T23:05:53.448 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:53.448 INFO:teuthology.orchestra.run.smithi005.stdout: "hostnames": [], 2022-09-09T23:05:53.448 INFO:teuthology.orchestra.run.smithi005.stdout: "hostnames_s3website": [], 2022-09-09T23:05:53.448 INFO:teuthology.orchestra.run.smithi005.stdout: "master_zone": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:05:53.449 INFO:teuthology.orchestra.run.smithi005.stdout: "zones": [ 2022-09-09T23:05:53.449 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:53.449 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "3ff609e7-37d9-4b47-8369-34e6c99c4c03", 2022-09-09T23:05:53.449 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zone4", 2022-09-09T23:05:53.450 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T23:05:53.450 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi080.front.sepia.ceph.com:8001" 2022-09-09T23:05:53.450 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:53.450 INFO:teuthology.orchestra.run.smithi005.stdout: "log_meta": false, 2022-09-09T23:05:53.451 INFO:teuthology.orchestra.run.smithi005.stdout: "log_data": true, 2022-09-09T23:05:53.451 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:05:53.451 INFO:teuthology.orchestra.run.smithi005.stdout: "read_only": false, 2022-09-09T23:05:53.451 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_type": "pubsub", 2022-09-09T23:05:53.452 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from_all": false, 2022-09-09T23:05:53.452 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from": [ 2022-09-09T23:05:53.452 INFO:teuthology.orchestra.run.smithi005.stdout: "test-zone1" 2022-09-09T23:05:53.452 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:53.453 INFO:teuthology.orchestra.run.smithi005.stdout: "redirect_zone": "", 2022-09-09T23:05:53.453 INFO:teuthology.orchestra.run.smithi005.stdout: "supported_features": [ 2022-09-09T23:05:53.453 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T23:05:53.453 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:53.454 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.454 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:53.454 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:05:53.454 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zone1", 2022-09-09T23:05:53.455 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T23:05:53.455 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi005.front.sepia.ceph.com:8000" 2022-09-09T23:05:53.455 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:53.455 INFO:teuthology.orchestra.run.smithi005.stdout: "log_meta": false, 2022-09-09T23:05:53.456 INFO:teuthology.orchestra.run.smithi005.stdout: "log_data": true, 2022-09-09T23:05:53.456 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:05:53.456 INFO:teuthology.orchestra.run.smithi005.stdout: "read_only": false, 2022-09-09T23:05:53.456 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_type": "", 2022-09-09T23:05:53.457 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from_all": true, 2022-09-09T23:05:53.457 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from": [], 2022-09-09T23:05:53.457 INFO:teuthology.orchestra.run.smithi005.stdout: "redirect_zone": "", 2022-09-09T23:05:53.457 INFO:teuthology.orchestra.run.smithi005.stdout: "supported_features": [ 2022-09-09T23:05:53.458 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T23:05:53.458 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:53.458 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.458 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:53.459 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5", 2022-09-09T23:05:53.459 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zone3", 2022-09-09T23:05:53.459 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T23:05:53.460 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi005.front.sepia.ceph.com:8001" 2022-09-09T23:05:53.460 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:53.460 INFO:teuthology.orchestra.run.smithi005.stdout: "log_meta": false, 2022-09-09T23:05:53.460 INFO:teuthology.orchestra.run.smithi005.stdout: "log_data": true, 2022-09-09T23:05:53.460 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:05:53.461 INFO:teuthology.orchestra.run.smithi005.stdout: "read_only": false, 2022-09-09T23:05:53.461 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_type": "", 2022-09-09T23:05:53.461 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from_all": true, 2022-09-09T23:05:53.461 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from": [], 2022-09-09T23:05:53.462 INFO:teuthology.orchestra.run.smithi005.stdout: "redirect_zone": "", 2022-09-09T23:05:53.462 INFO:teuthology.orchestra.run.smithi005.stdout: "supported_features": [ 2022-09-09T23:05:53.463 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T23:05:53.463 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:53.463 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.463 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:53.464 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T23:05:53.464 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zone2", 2022-09-09T23:05:53.464 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T23:05:53.464 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi080.front.sepia.ceph.com:8000" 2022-09-09T23:05:53.465 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:53.465 INFO:teuthology.orchestra.run.smithi005.stdout: "log_meta": false, 2022-09-09T23:05:53.465 INFO:teuthology.orchestra.run.smithi005.stdout: "log_data": true, 2022-09-09T23:05:53.465 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:05:53.466 INFO:teuthology.orchestra.run.smithi005.stdout: "read_only": false, 2022-09-09T23:05:53.466 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_type": "", 2022-09-09T23:05:53.466 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from_all": true, 2022-09-09T23:05:53.466 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from": [], 2022-09-09T23:05:53.467 INFO:teuthology.orchestra.run.smithi005.stdout: "redirect_zone": "", 2022-09-09T23:05:53.467 INFO:teuthology.orchestra.run.smithi005.stdout: "supported_features": [ 2022-09-09T23:05:53.467 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T23:05:53.467 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:53.468 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:05:53.468 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:53.468 INFO:teuthology.orchestra.run.smithi005.stdout: "placement_targets": [ 2022-09-09T23:05:53.468 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:53.469 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "default-placement", 2022-09-09T23:05:53.470 INFO:teuthology.orchestra.run.smithi005.stdout: "tags": [], 2022-09-09T23:05:53.470 INFO:teuthology.orchestra.run.smithi005.stdout: "storage_classes": [ 2022-09-09T23:05:53.470 INFO:teuthology.orchestra.run.smithi005.stdout: "STANDARD" 2022-09-09T23:05:53.471 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:53.471 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:05:53.471 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:53.471 INFO:teuthology.orchestra.run.smithi005.stdout: "default_placement": "default-placement", 2022-09-09T23:05:53.472 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T23:05:53.472 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_policy": { 2022-09-09T23:05:53.472 INFO:teuthology.orchestra.run.smithi005.stdout: "groups": [] 2022-09-09T23:05:53.472 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.473 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled_features": [ 2022-09-09T23:05:53.473 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T23:05:53.473 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:53.473 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:05:53.474 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:53.474 INFO:teuthology.orchestra.run.smithi005.stdout: "short_zone_ids": [ 2022-09-09T23:05:53.474 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:53.474 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "3ff609e7-37d9-4b47-8369-34e6c99c4c03", 2022-09-09T23:05:53.475 INFO:teuthology.orchestra.run.smithi005.stdout: "val": 2593176765 2022-09-09T23:05:53.475 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.475 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:53.475 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:05:53.476 INFO:teuthology.orchestra.run.smithi005.stdout: "val": 1301724575 2022-09-09T23:05:53.476 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.476 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:53.476 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5", 2022-09-09T23:05:53.477 INFO:teuthology.orchestra.run.smithi005.stdout: "val": 2787153217 2022-09-09T23:05:53.477 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.477 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:53.477 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T23:05:53.478 INFO:teuthology.orchestra.run.smithi005.stdout: "val": 1899625128 2022-09-09T23:05:53.478 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:05:53.478 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:53.479 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.479 INFO:teuthology.orchestra.run.smithi005.stdout: "master_zonegroup": "test-zonegroup", 2022-09-09T23:05:53.479 INFO:teuthology.orchestra.run.smithi005.stdout: "master_zone": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:05:53.479 INFO:teuthology.orchestra.run.smithi005.stdout: "period_config": { 2022-09-09T23:05:53.480 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_quota": { 2022-09-09T23:05:53.480 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false, 2022-09-09T23:05:53.480 INFO:teuthology.orchestra.run.smithi005.stdout: "check_on_raw": false, 2022-09-09T23:05:53.480 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size": -1, 2022-09-09T23:05:53.481 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size_kb": 0, 2022-09-09T23:05:53.481 INFO:teuthology.orchestra.run.smithi005.stdout: "max_objects": -1 2022-09-09T23:05:53.481 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.481 INFO:teuthology.orchestra.run.smithi005.stdout: "user_quota": { 2022-09-09T23:05:53.482 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false, 2022-09-09T23:05:53.482 INFO:teuthology.orchestra.run.smithi005.stdout: "check_on_raw": false, 2022-09-09T23:05:53.482 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size": -1, 2022-09-09T23:05:53.482 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size_kb": 0, 2022-09-09T23:05:53.483 INFO:teuthology.orchestra.run.smithi005.stdout: "max_objects": -1 2022-09-09T23:05:53.483 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.483 INFO:teuthology.orchestra.run.smithi005.stdout: "user_ratelimit": { 2022-09-09T23:05:53.483 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_ops": 0, 2022-09-09T23:05:53.484 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_ops": 0, 2022-09-09T23:05:53.484 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_bytes": 0, 2022-09-09T23:05:53.484 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_bytes": 0, 2022-09-09T23:05:53.484 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false 2022-09-09T23:05:53.485 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.485 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_ratelimit": { 2022-09-09T23:05:53.485 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_ops": 0, 2022-09-09T23:05:53.485 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_ops": 0, 2022-09-09T23:05:53.486 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_bytes": 0, 2022-09-09T23:05:53.486 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_bytes": 0, 2022-09-09T23:05:53.486 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false 2022-09-09T23:05:53.486 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.486 INFO:teuthology.orchestra.run.smithi005.stdout: "anonymous_ratelimit": { 2022-09-09T23:05:53.487 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_ops": 0, 2022-09-09T23:05:53.487 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_ops": 0, 2022-09-09T23:05:53.487 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_bytes": 0, 2022-09-09T23:05:53.487 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_bytes": 0, 2022-09-09T23:05:53.488 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false 2022-09-09T23:05:53.488 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:05:53.488 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.488 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T23:05:53.489 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_name": "test-realm", 2022-09-09T23:05:53.489 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 4 2022-09-09T23:05:53.489 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T23:05:53.490 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... ok 2022-09-09T23:05:53.491 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:05:53.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:05:53.492 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-09T23:05:53.536 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:05:53.537 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:05:53.583 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T23:05:53.584 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "d5dc7dea-5286-4566-9ec9-fc91ae9afc4d", 2022-09-09T23:05:53.584 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "remove_zg", 2022-09-09T23:05:53.584 INFO:teuthology.orchestra.run.smithi005.stdout: "api_name": "remove_zg", 2022-09-09T23:05:53.584 INFO:teuthology.orchestra.run.smithi005.stdout: "is_master": false, 2022-09-09T23:05:53.585 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [], 2022-09-09T23:05:53.585 INFO:teuthology.orchestra.run.smithi005.stdout: "hostnames": [], 2022-09-09T23:05:53.585 INFO:teuthology.orchestra.run.smithi005.stdout: "hostnames_s3website": [], 2022-09-09T23:05:53.585 INFO:teuthology.orchestra.run.smithi005.stdout: "master_zone": "", 2022-09-09T23:05:53.586 INFO:teuthology.orchestra.run.smithi005.stdout: "zones": [], 2022-09-09T23:05:53.586 INFO:teuthology.orchestra.run.smithi005.stdout: "placement_targets": [], 2022-09-09T23:05:53.586 INFO:teuthology.orchestra.run.smithi005.stdout: "default_placement": "", 2022-09-09T23:05:53.586 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T23:05:53.587 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_policy": { 2022-09-09T23:05:53.587 INFO:teuthology.orchestra.run.smithi005.stdout: "groups": [] 2022-09-09T23:05:53.587 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.587 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled_features": [ 2022-09-09T23:05:53.588 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T23:05:53.588 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:53.588 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T23:05:53.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:05:53.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:05:53.593 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-09T23:05:53.638 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:05:53.638 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:05:53.689 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T23:05:53.689 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "17e11d86-55c9-4d87-9f03-7ad69626d301", 2022-09-09T23:05:53.690 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "remove", 2022-09-09T23:05:53.690 INFO:teuthology.orchestra.run.smithi005.stdout: "domain_root": "remove.rgw.meta:root", 2022-09-09T23:05:53.690 INFO:teuthology.orchestra.run.smithi005.stdout: "control_pool": "remove.rgw.control", 2022-09-09T23:05:53.690 INFO:teuthology.orchestra.run.smithi005.stdout: "gc_pool": "remove.rgw.log:gc", 2022-09-09T23:05:53.691 INFO:teuthology.orchestra.run.smithi005.stdout: "lc_pool": "remove.rgw.log:lc", 2022-09-09T23:05:53.691 INFO:teuthology.orchestra.run.smithi005.stdout: "log_pool": "remove.rgw.log", 2022-09-09T23:05:53.691 INFO:teuthology.orchestra.run.smithi005.stdout: "intent_log_pool": "remove.rgw.log:intent", 2022-09-09T23:05:53.692 INFO:teuthology.orchestra.run.smithi005.stdout: "usage_log_pool": "remove.rgw.log:usage", 2022-09-09T23:05:53.692 INFO:teuthology.orchestra.run.smithi005.stdout: "roles_pool": "remove.rgw.meta:roles", 2022-09-09T23:05:53.692 INFO:teuthology.orchestra.run.smithi005.stdout: "reshard_pool": "remove.rgw.log:reshard", 2022-09-09T23:05:53.692 INFO:teuthology.orchestra.run.smithi005.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2022-09-09T23:05:53.693 INFO:teuthology.orchestra.run.smithi005.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2022-09-09T23:05:53.693 INFO:teuthology.orchestra.run.smithi005.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2022-09-09T23:05:53.693 INFO:teuthology.orchestra.run.smithi005.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2022-09-09T23:05:53.693 INFO:teuthology.orchestra.run.smithi005.stdout: "otp_pool": "remove.rgw.otp", 2022-09-09T23:05:53.694 INFO:teuthology.orchestra.run.smithi005.stdout: "system_key": { 2022-09-09T23:05:53.694 INFO:teuthology.orchestra.run.smithi005.stdout: "access_key": "", 2022-09-09T23:05:53.694 INFO:teuthology.orchestra.run.smithi005.stdout: "secret_key": "" 2022-09-09T23:05:53.694 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.695 INFO:teuthology.orchestra.run.smithi005.stdout: "placement_pools": [ 2022-09-09T23:05:53.695 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:53.695 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "default-placement", 2022-09-09T23:05:53.695 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:05:53.696 INFO:teuthology.orchestra.run.smithi005.stdout: "index_pool": "remove.rgw.buckets.index", 2022-09-09T23:05:53.696 INFO:teuthology.orchestra.run.smithi005.stdout: "storage_classes": { 2022-09-09T23:05:53.696 INFO:teuthology.orchestra.run.smithi005.stdout: "STANDARD": { 2022-09-09T23:05:53.696 INFO:teuthology.orchestra.run.smithi005.stdout: "data_pool": "remove.rgw.buckets.data" 2022-09-09T23:05:53.697 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:05:53.697 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.697 INFO:teuthology.orchestra.run.smithi005.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2022-09-09T23:05:53.697 INFO:teuthology.orchestra.run.smithi005.stdout: "index_type": 0 2022-09-09T23:05:53.698 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:05:53.698 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:05:53.698 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:53.698 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T23:05:53.699 INFO:teuthology.orchestra.run.smithi005.stdout: "notif_pool": "remove.rgw.log:notif" 2022-09-09T23:05:53.699 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T23:05:53.700 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:05:53.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:05:53.701 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-09T23:05:53.750 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:05:53.751 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:05:53.860 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T23:05:53.860 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:05:53.861 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 8, 2022-09-09T23:05:53.861 INFO:teuthology.orchestra.run.smithi005.stdout: "predecessor_uuid": "2efdc3a1-023b-4e78-aaa2-0e069a034dac", 2022-09-09T23:05:53.861 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": [ 2022-09-09T23:05:53.861 INFO:teuthology.orchestra.run.smithi005.stdout: "", 2022-09-09T23:05:53.862 INFO:teuthology.orchestra.run.smithi005.stdout: "", 2022-09-09T23:05:53.862 INFO:teuthology.orchestra.run.smithi005.stdout: "", 2022-09-09T23:05:53.862 INFO:teuthology.orchestra.run.smithi005.stdout: "1_1662764312.564852_2684.1" 2022-09-09T23:05:53.863 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:53.863 INFO:teuthology.orchestra.run.smithi005.stdout: "period_map": { 2022-09-09T23:05:53.863 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:05:53.863 INFO:teuthology.orchestra.run.smithi005.stdout: "zonegroups": [ 2022-09-09T23:05:53.864 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:53.864 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "d5dc7dea-5286-4566-9ec9-fc91ae9afc4d", 2022-09-09T23:05:53.864 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "remove_zg", 2022-09-09T23:05:53.864 INFO:teuthology.orchestra.run.smithi005.stdout: "api_name": "remove_zg", 2022-09-09T23:05:53.865 INFO:teuthology.orchestra.run.smithi005.stdout: "is_master": false, 2022-09-09T23:05:53.865 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [], 2022-09-09T23:05:53.865 INFO:teuthology.orchestra.run.smithi005.stdout: "hostnames": [], 2022-09-09T23:05:53.865 INFO:teuthology.orchestra.run.smithi005.stdout: "hostnames_s3website": [], 2022-09-09T23:05:53.866 INFO:teuthology.orchestra.run.smithi005.stdout: "master_zone": "17e11d86-55c9-4d87-9f03-7ad69626d301", 2022-09-09T23:05:53.866 INFO:teuthology.orchestra.run.smithi005.stdout: "zones": [ 2022-09-09T23:05:53.866 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:53.867 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "17e11d86-55c9-4d87-9f03-7ad69626d301", 2022-09-09T23:05:53.867 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "remove", 2022-09-09T23:05:53.867 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [], 2022-09-09T23:05:53.867 INFO:teuthology.orchestra.run.smithi005.stdout: "log_meta": false, 2022-09-09T23:05:53.868 INFO:teuthology.orchestra.run.smithi005.stdout: "log_data": false, 2022-09-09T23:05:53.868 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:05:53.868 INFO:teuthology.orchestra.run.smithi005.stdout: "read_only": false, 2022-09-09T23:05:53.869 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_type": "", 2022-09-09T23:05:53.869 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from_all": true, 2022-09-09T23:05:53.869 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from": [], 2022-09-09T23:05:53.869 INFO:teuthology.orchestra.run.smithi005.stdout: "redirect_zone": "", 2022-09-09T23:05:53.869 INFO:teuthology.orchestra.run.smithi005.stdout: "supported_features": [ 2022-09-09T23:05:53.870 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T23:05:53.870 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:53.870 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:05:53.871 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:53.871 INFO:teuthology.orchestra.run.smithi005.stdout: "placement_targets": [ 2022-09-09T23:05:53.871 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:53.871 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "default-placement", 2022-09-09T23:05:53.871 INFO:teuthology.orchestra.run.smithi005.stdout: "tags": [], 2022-09-09T23:05:53.872 INFO:teuthology.orchestra.run.smithi005.stdout: "storage_classes": [ 2022-09-09T23:05:53.872 INFO:teuthology.orchestra.run.smithi005.stdout: "STANDARD" 2022-09-09T23:05:53.872 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:53.872 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:05:53.873 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:53.873 INFO:teuthology.orchestra.run.smithi005.stdout: "default_placement": "default-placement", 2022-09-09T23:05:53.873 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T23:05:53.873 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_policy": { 2022-09-09T23:05:53.874 INFO:teuthology.orchestra.run.smithi005.stdout: "groups": [] 2022-09-09T23:05:53.874 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.874 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled_features": [ 2022-09-09T23:05:53.874 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T23:05:53.875 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:53.875 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.878 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:53.878 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "test-zonegroup", 2022-09-09T23:05:53.878 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zonegroup", 2022-09-09T23:05:53.879 INFO:teuthology.orchestra.run.smithi005.stdout: "api_name": "", 2022-09-09T23:05:53.879 INFO:teuthology.orchestra.run.smithi005.stdout: "is_master": true, 2022-09-09T23:05:53.879 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T23:05:53.880 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi005.front.sepia.ceph.com:8000" 2022-09-09T23:05:53.880 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:53.880 INFO:teuthology.orchestra.run.smithi005.stdout: "hostnames": [], 2022-09-09T23:05:53.881 INFO:teuthology.orchestra.run.smithi005.stdout: "hostnames_s3website": [], 2022-09-09T23:05:53.881 INFO:teuthology.orchestra.run.smithi005.stdout: "master_zone": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:05:53.881 INFO:teuthology.orchestra.run.smithi005.stdout: "zones": [ 2022-09-09T23:05:53.881 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:53.882 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "3ff609e7-37d9-4b47-8369-34e6c99c4c03", 2022-09-09T23:05:53.882 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zone4", 2022-09-09T23:05:53.882 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T23:05:53.883 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi080.front.sepia.ceph.com:8001" 2022-09-09T23:05:53.883 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:53.883 INFO:teuthology.orchestra.run.smithi005.stdout: "log_meta": false, 2022-09-09T23:05:53.884 INFO:teuthology.orchestra.run.smithi005.stdout: "log_data": true, 2022-09-09T23:05:53.884 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:05:53.884 INFO:teuthology.orchestra.run.smithi005.stdout: "read_only": false, 2022-09-09T23:05:53.885 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_type": "pubsub", 2022-09-09T23:05:53.885 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from_all": false, 2022-09-09T23:05:53.885 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from": [ 2022-09-09T23:05:53.885 INFO:teuthology.orchestra.run.smithi005.stdout: "test-zone1" 2022-09-09T23:05:53.886 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:53.886 INFO:teuthology.orchestra.run.smithi005.stdout: "redirect_zone": "", 2022-09-09T23:05:53.886 INFO:teuthology.orchestra.run.smithi005.stdout: "supported_features": [ 2022-09-09T23:05:53.887 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T23:05:53.887 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:53.887 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.887 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:53.888 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:05:53.888 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zone1", 2022-09-09T23:05:53.888 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T23:05:53.888 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi005.front.sepia.ceph.com:8000" 2022-09-09T23:05:53.889 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:53.889 INFO:teuthology.orchestra.run.smithi005.stdout: "log_meta": false, 2022-09-09T23:05:53.889 INFO:teuthology.orchestra.run.smithi005.stdout: "log_data": true, 2022-09-09T23:05:53.889 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:05:53.890 INFO:teuthology.orchestra.run.smithi005.stdout: "read_only": false, 2022-09-09T23:05:53.890 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_type": "", 2022-09-09T23:05:53.890 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from_all": true, 2022-09-09T23:05:53.891 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from": [], 2022-09-09T23:05:53.891 INFO:teuthology.orchestra.run.smithi005.stdout: "redirect_zone": "", 2022-09-09T23:05:53.892 INFO:teuthology.orchestra.run.smithi005.stdout: "supported_features": [ 2022-09-09T23:05:53.892 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T23:05:53.892 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:53.894 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.894 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:53.894 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5", 2022-09-09T23:05:53.895 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zone3", 2022-09-09T23:05:53.895 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T23:05:53.896 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi005.front.sepia.ceph.com:8001" 2022-09-09T23:05:53.896 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:53.896 INFO:teuthology.orchestra.run.smithi005.stdout: "log_meta": false, 2022-09-09T23:05:53.897 INFO:teuthology.orchestra.run.smithi005.stdout: "log_data": true, 2022-09-09T23:05:53.897 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:05:53.897 INFO:teuthology.orchestra.run.smithi005.stdout: "read_only": false, 2022-09-09T23:05:53.898 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_type": "", 2022-09-09T23:05:53.898 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from_all": true, 2022-09-09T23:05:53.899 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from": [], 2022-09-09T23:05:53.899 INFO:teuthology.orchestra.run.smithi005.stdout: "redirect_zone": "", 2022-09-09T23:05:53.899 INFO:teuthology.orchestra.run.smithi005.stdout: "supported_features": [ 2022-09-09T23:05:53.899 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T23:05:53.900 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:53.900 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.900 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:53.900 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T23:05:53.901 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zone2", 2022-09-09T23:05:53.901 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T23:05:53.901 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi080.front.sepia.ceph.com:8000" 2022-09-09T23:05:53.901 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:53.902 INFO:teuthology.orchestra.run.smithi005.stdout: "log_meta": false, 2022-09-09T23:05:53.902 INFO:teuthology.orchestra.run.smithi005.stdout: "log_data": true, 2022-09-09T23:05:53.902 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:05:53.902 INFO:teuthology.orchestra.run.smithi005.stdout: "read_only": false, 2022-09-09T23:05:53.903 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_type": "", 2022-09-09T23:05:53.903 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from_all": true, 2022-09-09T23:05:53.903 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from": [], 2022-09-09T23:05:53.903 INFO:teuthology.orchestra.run.smithi005.stdout: "redirect_zone": "", 2022-09-09T23:05:53.904 INFO:teuthology.orchestra.run.smithi005.stdout: "supported_features": [ 2022-09-09T23:05:53.904 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T23:05:53.904 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:53.904 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:05:53.905 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:53.905 INFO:teuthology.orchestra.run.smithi005.stdout: "placement_targets": [ 2022-09-09T23:05:53.905 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:53.905 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "default-placement", 2022-09-09T23:05:53.906 INFO:teuthology.orchestra.run.smithi005.stdout: "tags": [], 2022-09-09T23:05:53.906 INFO:teuthology.orchestra.run.smithi005.stdout: "storage_classes": [ 2022-09-09T23:05:53.906 INFO:teuthology.orchestra.run.smithi005.stdout: "STANDARD" 2022-09-09T23:05:53.906 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:53.907 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:05:53.907 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:53.907 INFO:teuthology.orchestra.run.smithi005.stdout: "default_placement": "default-placement", 2022-09-09T23:05:53.908 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T23:05:53.908 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_policy": { 2022-09-09T23:05:53.908 INFO:teuthology.orchestra.run.smithi005.stdout: "groups": [] 2022-09-09T23:05:53.909 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.909 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled_features": [ 2022-09-09T23:05:53.910 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T23:05:53.910 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:53.910 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:05:53.910 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:53.911 INFO:teuthology.orchestra.run.smithi005.stdout: "short_zone_ids": [ 2022-09-09T23:05:53.911 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:53.911 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "17e11d86-55c9-4d87-9f03-7ad69626d301", 2022-09-09T23:05:53.911 INFO:teuthology.orchestra.run.smithi005.stdout: "val": 2360631156 2022-09-09T23:05:53.912 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.912 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:53.912 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "3ff609e7-37d9-4b47-8369-34e6c99c4c03", 2022-09-09T23:05:53.912 INFO:teuthology.orchestra.run.smithi005.stdout: "val": 2593176765 2022-09-09T23:05:53.913 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.913 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:53.913 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:05:53.913 INFO:teuthology.orchestra.run.smithi005.stdout: "val": 1301724575 2022-09-09T23:05:53.914 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.914 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:53.914 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5", 2022-09-09T23:05:53.914 INFO:teuthology.orchestra.run.smithi005.stdout: "val": 2787153217 2022-09-09T23:05:53.915 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.915 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:53.915 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T23:05:53.915 INFO:teuthology.orchestra.run.smithi005.stdout: "val": 1899625128 2022-09-09T23:05:53.916 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:05:53.916 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:53.916 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.916 INFO:teuthology.orchestra.run.smithi005.stdout: "master_zonegroup": "test-zonegroup", 2022-09-09T23:05:53.917 INFO:teuthology.orchestra.run.smithi005.stdout: "master_zone": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:05:53.917 INFO:teuthology.orchestra.run.smithi005.stdout: "period_config": { 2022-09-09T23:05:53.917 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_quota": { 2022-09-09T23:05:53.917 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false, 2022-09-09T23:05:53.918 INFO:teuthology.orchestra.run.smithi005.stdout: "check_on_raw": false, 2022-09-09T23:05:53.918 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size": -1, 2022-09-09T23:05:53.918 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size_kb": 0, 2022-09-09T23:05:53.918 INFO:teuthology.orchestra.run.smithi005.stdout: "max_objects": -1 2022-09-09T23:05:53.919 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.919 INFO:teuthology.orchestra.run.smithi005.stdout: "user_quota": { 2022-09-09T23:05:53.919 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false, 2022-09-09T23:05:53.920 INFO:teuthology.orchestra.run.smithi005.stdout: "check_on_raw": false, 2022-09-09T23:05:53.920 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size": -1, 2022-09-09T23:05:53.920 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size_kb": 0, 2022-09-09T23:05:53.920 INFO:teuthology.orchestra.run.smithi005.stdout: "max_objects": -1 2022-09-09T23:05:53.921 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.921 INFO:teuthology.orchestra.run.smithi005.stdout: "user_ratelimit": { 2022-09-09T23:05:53.921 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_ops": 0, 2022-09-09T23:05:53.922 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_ops": 0, 2022-09-09T23:05:53.922 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_bytes": 0, 2022-09-09T23:05:53.922 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_bytes": 0, 2022-09-09T23:05:53.922 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false 2022-09-09T23:05:53.923 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.923 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_ratelimit": { 2022-09-09T23:05:53.923 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_ops": 0, 2022-09-09T23:05:53.923 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_ops": 0, 2022-09-09T23:05:53.924 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_bytes": 0, 2022-09-09T23:05:53.924 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_bytes": 0, 2022-09-09T23:05:53.924 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false 2022-09-09T23:05:53.924 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.925 INFO:teuthology.orchestra.run.smithi005.stdout: "anonymous_ratelimit": { 2022-09-09T23:05:53.925 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_ops": 0, 2022-09-09T23:05:53.925 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_ops": 0, 2022-09-09T23:05:53.925 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_bytes": 0, 2022-09-09T23:05:53.926 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_bytes": 0, 2022-09-09T23:05:53.926 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false 2022-09-09T23:05:53.926 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:05:53.927 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:53.927 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T23:05:53.927 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_name": "test-realm", 2022-09-09T23:05:53.927 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 4 2022-09-09T23:05:53.928 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T23:05:53.928 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:05:53.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:05:53.929 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-09T23:05:53.977 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:05:53.978 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:05:54.034 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T23:05:54.033+0000 7f518d502dc0 0 zone id 17e11d86-55c9-4d87-9f03-7ad69626d301 is not a part of zonegroup test-zonegroup 2022-09-09T23:05:54.034 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T23:05:54.033+0000 7f518d502dc0 0 zone id 17e11d86-55c9-4d87-9f03-7ad69626d301 is not a part of zonegroup default 2022-09-09T23:05:54.044 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:05:54.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:05:54.045 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-09T23:05:54.091 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:05:54.092 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:05:54.143 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T23:05:54.145+0000 7ff32d412dc0 0 ERROR: zonegroup remove_zg has a non existent master zone 2022-09-09T23:05:54.150 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-09-09T23:05:54.151 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:05:54.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:05:54.152 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-09T23:05:54.242 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:05:54.243 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:05:54.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:05:54.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:05:54.293 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-09T23:05:54.333 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:05:54.334 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:05:54.429 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T23:05:54.430 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:05:54.430 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 9, 2022-09-09T23:05:54.430 INFO:teuthology.orchestra.run.smithi005.stdout: "predecessor_uuid": "2efdc3a1-023b-4e78-aaa2-0e069a034dac", 2022-09-09T23:05:54.431 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": [ 2022-09-09T23:05:54.431 INFO:teuthology.orchestra.run.smithi005.stdout: "", 2022-09-09T23:05:54.431 INFO:teuthology.orchestra.run.smithi005.stdout: "", 2022-09-09T23:05:54.431 INFO:teuthology.orchestra.run.smithi005.stdout: "", 2022-09-09T23:05:54.432 INFO:teuthology.orchestra.run.smithi005.stdout: "1_1662764312.564852_2684.1" 2022-09-09T23:05:54.432 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:54.432 INFO:teuthology.orchestra.run.smithi005.stdout: "period_map": { 2022-09-09T23:05:54.432 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:05:54.433 INFO:teuthology.orchestra.run.smithi005.stdout: "zonegroups": [ 2022-09-09T23:05:54.433 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:54.433 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "test-zonegroup", 2022-09-09T23:05:54.433 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zonegroup", 2022-09-09T23:05:54.434 INFO:teuthology.orchestra.run.smithi005.stdout: "api_name": "", 2022-09-09T23:05:54.434 INFO:teuthology.orchestra.run.smithi005.stdout: "is_master": true, 2022-09-09T23:05:54.434 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T23:05:54.434 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi005.front.sepia.ceph.com:8000" 2022-09-09T23:05:54.435 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:54.435 INFO:teuthology.orchestra.run.smithi005.stdout: "hostnames": [], 2022-09-09T23:05:54.435 INFO:teuthology.orchestra.run.smithi005.stdout: "hostnames_s3website": [], 2022-09-09T23:05:54.435 INFO:teuthology.orchestra.run.smithi005.stdout: "master_zone": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:05:54.436 INFO:teuthology.orchestra.run.smithi005.stdout: "zones": [ 2022-09-09T23:05:54.436 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:54.436 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "3ff609e7-37d9-4b47-8369-34e6c99c4c03", 2022-09-09T23:05:54.436 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zone4", 2022-09-09T23:05:54.437 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T23:05:54.437 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi080.front.sepia.ceph.com:8001" 2022-09-09T23:05:54.437 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:54.437 INFO:teuthology.orchestra.run.smithi005.stdout: "log_meta": false, 2022-09-09T23:05:54.438 INFO:teuthology.orchestra.run.smithi005.stdout: "log_data": true, 2022-09-09T23:05:54.438 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:05:54.438 INFO:teuthology.orchestra.run.smithi005.stdout: "read_only": false, 2022-09-09T23:05:54.438 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_type": "pubsub", 2022-09-09T23:05:54.439 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from_all": false, 2022-09-09T23:05:54.439 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from": [ 2022-09-09T23:05:54.439 INFO:teuthology.orchestra.run.smithi005.stdout: "test-zone1" 2022-09-09T23:05:54.439 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:54.440 INFO:teuthology.orchestra.run.smithi005.stdout: "redirect_zone": "", 2022-09-09T23:05:54.440 INFO:teuthology.orchestra.run.smithi005.stdout: "supported_features": [ 2022-09-09T23:05:54.440 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T23:05:54.440 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:54.440 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:54.441 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:54.441 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:05:54.441 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zone1", 2022-09-09T23:05:54.441 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T23:05:54.442 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi005.front.sepia.ceph.com:8000" 2022-09-09T23:05:54.442 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:54.442 INFO:teuthology.orchestra.run.smithi005.stdout: "log_meta": false, 2022-09-09T23:05:54.442 INFO:teuthology.orchestra.run.smithi005.stdout: "log_data": true, 2022-09-09T23:05:54.443 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:05:54.443 INFO:teuthology.orchestra.run.smithi005.stdout: "read_only": false, 2022-09-09T23:05:54.443 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_type": "", 2022-09-09T23:05:54.443 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from_all": true, 2022-09-09T23:05:54.444 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from": [], 2022-09-09T23:05:54.444 INFO:teuthology.orchestra.run.smithi005.stdout: "redirect_zone": "", 2022-09-09T23:05:54.444 INFO:teuthology.orchestra.run.smithi005.stdout: "supported_features": [ 2022-09-09T23:05:54.444 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T23:05:54.445 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:54.445 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:54.445 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:54.445 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5", 2022-09-09T23:05:54.446 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zone3", 2022-09-09T23:05:54.446 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T23:05:54.446 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi005.front.sepia.ceph.com:8001" 2022-09-09T23:05:54.446 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:54.447 INFO:teuthology.orchestra.run.smithi005.stdout: "log_meta": false, 2022-09-09T23:05:54.447 INFO:teuthology.orchestra.run.smithi005.stdout: "log_data": true, 2022-09-09T23:05:54.447 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:05:54.447 INFO:teuthology.orchestra.run.smithi005.stdout: "read_only": false, 2022-09-09T23:05:54.448 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_type": "", 2022-09-09T23:05:54.448 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from_all": true, 2022-09-09T23:05:54.448 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from": [], 2022-09-09T23:05:54.448 INFO:teuthology.orchestra.run.smithi005.stdout: "redirect_zone": "", 2022-09-09T23:05:54.448 INFO:teuthology.orchestra.run.smithi005.stdout: "supported_features": [ 2022-09-09T23:05:54.449 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T23:05:54.449 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:54.449 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:54.449 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:54.450 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T23:05:54.450 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "test-zone2", 2022-09-09T23:05:54.450 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [ 2022-09-09T23:05:54.450 INFO:teuthology.orchestra.run.smithi005.stdout: "http://smithi080.front.sepia.ceph.com:8000" 2022-09-09T23:05:54.451 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:54.451 INFO:teuthology.orchestra.run.smithi005.stdout: "log_meta": false, 2022-09-09T23:05:54.451 INFO:teuthology.orchestra.run.smithi005.stdout: "log_data": true, 2022-09-09T23:05:54.451 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_index_max_shards": 11, 2022-09-09T23:05:54.452 INFO:teuthology.orchestra.run.smithi005.stdout: "read_only": false, 2022-09-09T23:05:54.452 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_type": "", 2022-09-09T23:05:54.452 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from_all": true, 2022-09-09T23:05:54.452 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from": [], 2022-09-09T23:05:54.452 INFO:teuthology.orchestra.run.smithi005.stdout: "redirect_zone": "", 2022-09-09T23:05:54.453 INFO:teuthology.orchestra.run.smithi005.stdout: "supported_features": [ 2022-09-09T23:05:54.453 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T23:05:54.453 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:54.453 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:05:54.454 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:54.454 INFO:teuthology.orchestra.run.smithi005.stdout: "placement_targets": [ 2022-09-09T23:05:54.454 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:54.454 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "default-placement", 2022-09-09T23:05:54.455 INFO:teuthology.orchestra.run.smithi005.stdout: "tags": [], 2022-09-09T23:05:54.455 INFO:teuthology.orchestra.run.smithi005.stdout: "storage_classes": [ 2022-09-09T23:05:54.456 INFO:teuthology.orchestra.run.smithi005.stdout: "STANDARD" 2022-09-09T23:05:54.456 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:54.456 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:05:54.456 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:54.457 INFO:teuthology.orchestra.run.smithi005.stdout: "default_placement": "default-placement", 2022-09-09T23:05:54.457 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T23:05:54.457 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_policy": { 2022-09-09T23:05:54.457 INFO:teuthology.orchestra.run.smithi005.stdout: "groups": [] 2022-09-09T23:05:54.458 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:54.458 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled_features": [ 2022-09-09T23:05:54.458 INFO:teuthology.orchestra.run.smithi005.stdout: "resharding" 2022-09-09T23:05:54.459 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:54.459 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:05:54.459 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2022-09-09T23:05:54.459 INFO:teuthology.orchestra.run.smithi005.stdout: "short_zone_ids": [ 2022-09-09T23:05:54.460 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:54.460 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "3ff609e7-37d9-4b47-8369-34e6c99c4c03", 2022-09-09T23:05:54.460 INFO:teuthology.orchestra.run.smithi005.stdout: "val": 2593176765 2022-09-09T23:05:54.460 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:54.461 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:54.461 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:05:54.461 INFO:teuthology.orchestra.run.smithi005.stdout: "val": 1301724575 2022-09-09T23:05:54.461 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:54.462 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:54.462 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5", 2022-09-09T23:05:54.462 INFO:teuthology.orchestra.run.smithi005.stdout: "val": 2787153217 2022-09-09T23:05:54.462 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:54.463 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:54.463 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T23:05:54.463 INFO:teuthology.orchestra.run.smithi005.stdout: "val": 1899625128 2022-09-09T23:05:54.463 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:05:54.464 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:05:54.464 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:54.464 INFO:teuthology.orchestra.run.smithi005.stdout: "master_zonegroup": "test-zonegroup", 2022-09-09T23:05:54.464 INFO:teuthology.orchestra.run.smithi005.stdout: "master_zone": "4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:05:54.464 INFO:teuthology.orchestra.run.smithi005.stdout: "period_config": { 2022-09-09T23:05:54.465 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_quota": { 2022-09-09T23:05:54.465 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false, 2022-09-09T23:05:54.465 INFO:teuthology.orchestra.run.smithi005.stdout: "check_on_raw": false, 2022-09-09T23:05:54.465 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size": -1, 2022-09-09T23:05:54.466 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size_kb": 0, 2022-09-09T23:05:54.466 INFO:teuthology.orchestra.run.smithi005.stdout: "max_objects": -1 2022-09-09T23:05:54.466 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:54.466 INFO:teuthology.orchestra.run.smithi005.stdout: "user_quota": { 2022-09-09T23:05:54.467 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false, 2022-09-09T23:05:54.467 INFO:teuthology.orchestra.run.smithi005.stdout: "check_on_raw": false, 2022-09-09T23:05:54.467 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size": -1, 2022-09-09T23:05:54.467 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size_kb": 0, 2022-09-09T23:05:54.468 INFO:teuthology.orchestra.run.smithi005.stdout: "max_objects": -1 2022-09-09T23:05:54.468 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:54.468 INFO:teuthology.orchestra.run.smithi005.stdout: "user_ratelimit": { 2022-09-09T23:05:54.468 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_ops": 0, 2022-09-09T23:05:54.469 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_ops": 0, 2022-09-09T23:05:54.469 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_bytes": 0, 2022-09-09T23:05:54.469 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_bytes": 0, 2022-09-09T23:05:54.469 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false 2022-09-09T23:05:54.470 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:54.470 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_ratelimit": { 2022-09-09T23:05:54.470 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_ops": 0, 2022-09-09T23:05:54.470 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_ops": 0, 2022-09-09T23:05:54.471 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_bytes": 0, 2022-09-09T23:05:54.471 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_bytes": 0, 2022-09-09T23:05:54.471 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false 2022-09-09T23:05:54.471 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:54.471 INFO:teuthology.orchestra.run.smithi005.stdout: "anonymous_ratelimit": { 2022-09-09T23:05:54.472 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_ops": 0, 2022-09-09T23:05:54.472 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_ops": 0, 2022-09-09T23:05:54.472 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_bytes": 0, 2022-09-09T23:05:54.472 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_bytes": 0, 2022-09-09T23:05:54.473 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false 2022-09-09T23:05:54.473 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:05:54.473 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:54.473 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_id": "49e55708-9d21-40b2-99ec-1140f8e8ae95", 2022-09-09T23:05:54.474 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_name": "test-realm", 2022-09-09T23:05:54.474 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 4 2022-09-09T23:05:54.474 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T23:05:54.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2022-09-09T23:05:54.476 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jkcgbj-50 2022-09-09T23:05:54.477 DEBUG:boto:path=/jkcgbj-50/ 2022-09-09T23:05:54.478 DEBUG:boto:auth_path=/jkcgbj-50/ 2022-09-09T23:05:54.478 DEBUG:boto:Method: PUT 2022-09-09T23:05:54.479 DEBUG:boto:Path: /jkcgbj-50/ 2022-09-09T23:05:54.479 DEBUG:boto:Data: 2022-09-09T23:05:54.480 DEBUG:boto:Headers: {} 2022-09-09T23:05:54.480 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T23:05:54.481 DEBUG:boto:Port: 8000 2022-09-09T23:05:54.481 DEBUG:boto:Params: {} 2022-09-09T23:05:54.482 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:05:54.483 DEBUG:boto:Token: None 2022-09-09T23:05:54.484 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:05:54 GMT /jkcgbj-50/ 2022-09-09T23:05:54.485 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:KZh1UMsh78o6IMVKwI6l0QCFCtI= 2022-09-09T23:05:54.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:05:54 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:KZh1UMsh78o6IMVKwI6l0QCFCtI=', 'Content-Length': '0'} 2022-09-09T23:05:54.619 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000193ba8add29027d4-00631bc6d2-157e-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:05:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:05:54.621 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jkcgbj-51 2022-09-09T23:05:54.621 DEBUG:boto:path=/jkcgbj-51/ 2022-09-09T23:05:54.622 DEBUG:boto:auth_path=/jkcgbj-51/ 2022-09-09T23:05:54.622 DEBUG:boto:Method: PUT 2022-09-09T23:05:54.623 DEBUG:boto:Path: /jkcgbj-51/ 2022-09-09T23:05:54.623 DEBUG:boto:Data: 2022-09-09T23:05:54.624 DEBUG:boto:Headers: {} 2022-09-09T23:05:54.624 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T23:05:54.625 DEBUG:boto:Port: 8000 2022-09-09T23:05:54.625 DEBUG:boto:Params: {} 2022-09-09T23:05:54.626 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:05:54.627 DEBUG:boto:Token: None 2022-09-09T23:05:54.628 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:05:54 GMT /jkcgbj-51/ 2022-09-09T23:05:54.628 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:k35Joz90fyYv7OntolrFbyc2tTs= 2022-09-09T23:05:54.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:05:54 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:k35Joz90fyYv7OntolrFbyc2tTs=', 'Content-Length': '0'} 2022-09-09T23:05:55.535 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002a94cea874c4a52e-00631bc6d3-16ce-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:05:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:05:55.536 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jkcgbj-52 2022-09-09T23:05:55.536 DEBUG:boto:path=/jkcgbj-52/ 2022-09-09T23:05:55.537 DEBUG:boto:auth_path=/jkcgbj-52/ 2022-09-09T23:05:55.537 DEBUG:boto:Method: PUT 2022-09-09T23:05:55.538 DEBUG:boto:Path: /jkcgbj-52/ 2022-09-09T23:05:55.538 DEBUG:boto:Data: 2022-09-09T23:05:55.539 DEBUG:boto:Headers: {} 2022-09-09T23:05:55.539 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T23:05:55.540 DEBUG:boto:Port: 8001 2022-09-09T23:05:55.540 DEBUG:boto:Params: {} 2022-09-09T23:05:55.541 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-09T23:05:55.541 DEBUG:boto:Token: None 2022-09-09T23:05:55.542 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:05:55 GMT /jkcgbj-52/ 2022-09-09T23:05:55.543 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7JC6Is9WB9YSXCRtd4VKqx6tw7Q= 2022-09-09T23:05:55.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:05:55 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7JC6Is9WB9YSXCRtd4VKqx6tw7Q=', 'Content-Length': '0'} 2022-09-09T23:05:55.602 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b79376980652d96d-00631bc6d3-157c-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:05:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:05:55.603 DEBUG:boto:path=/jkcgbj-50/ 2022-09-09T23:05:55.604 DEBUG:boto:auth_path=/jkcgbj-50/ 2022-09-09T23:05:55.605 DEBUG:boto:path=/jkcgbj-50/?website 2022-09-09T23:05:55.605 DEBUG:boto:auth_path=/jkcgbj-50/?website 2022-09-09T23:05:55.606 DEBUG:boto:Method: PUT 2022-09-09T23:05:55.606 DEBUG:boto:Path: /jkcgbj-50/?website 2022-09-09T23:05:55.607 DEBUG:boto:Data: index.htmlerror.html 2022-09-09T23:05:55.607 DEBUG:boto:Headers: {} 2022-09-09T23:05:55.608 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T23:05:55.608 DEBUG:boto:Port: 8000 2022-09-09T23:05:55.609 DEBUG:boto:Params: {} 2022-09-09T23:05:55.609 DEBUG:boto:Token: None 2022-09-09T23:05:55.610 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:05:55 GMT /jkcgbj-50/?website 2022-09-09T23:05:55.611 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:xVwTZ3ZGlmxNhGZ53Hk50RnRqxU= 2022-09-09T23:05:55.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:05:55 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:xVwTZ3ZGlmxNhGZ53Hk50RnRqxU=', 'Content-Length': '241'} 2022-09-09T23:05:55.612 DEBUG:boto:Response headers: [('Content-Length', '202'), ('x-amz-request-id', 'tx0000073007dbd7e7e0f7b-00631bc6d3-157e-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 09 Sep 2022 23:05:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:05:55.614 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... SKIP: test_set_bucket_website skipped. Requires rgw_enable_static_website = 1. 2022-09-09T23:05:55.616 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jkcgbj-53 2022-09-09T23:05:55.616 DEBUG:boto:path=/jkcgbj-53/ 2022-09-09T23:05:55.617 DEBUG:boto:auth_path=/jkcgbj-53/ 2022-09-09T23:05:55.617 DEBUG:boto:Method: PUT 2022-09-09T23:05:55.618 DEBUG:boto:Path: /jkcgbj-53/ 2022-09-09T23:05:55.618 DEBUG:boto:Data: 2022-09-09T23:05:55.619 DEBUG:boto:Headers: {} 2022-09-09T23:05:55.619 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T23:05:55.620 DEBUG:boto:Port: 8000 2022-09-09T23:05:55.620 DEBUG:boto:Params: {} 2022-09-09T23:05:55.621 DEBUG:boto:Token: None 2022-09-09T23:05:55.621 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:05:55 GMT /jkcgbj-53/ 2022-09-09T23:05:55.622 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:NRD2GPw0h1JJThi7uyX1gi382A0= 2022-09-09T23:05:55.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:05:55 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:NRD2GPw0h1JJThi7uyX1gi382A0=', 'Content-Length': '0'} 2022-09-09T23:05:55.654 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000015b0d1c740761211-00631bc6d3-157e-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:05:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:05:55.655 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jkcgbj-54 2022-09-09T23:05:55.655 DEBUG:boto:path=/jkcgbj-54/ 2022-09-09T23:05:55.656 DEBUG:boto:auth_path=/jkcgbj-54/ 2022-09-09T23:05:55.656 DEBUG:boto:Method: PUT 2022-09-09T23:05:55.657 DEBUG:boto:Path: /jkcgbj-54/ 2022-09-09T23:05:55.657 DEBUG:boto:Data: 2022-09-09T23:05:55.658 DEBUG:boto:Headers: {} 2022-09-09T23:05:55.659 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T23:05:55.659 DEBUG:boto:Port: 8000 2022-09-09T23:05:55.660 DEBUG:boto:Params: {} 2022-09-09T23:05:55.660 DEBUG:boto:Token: None 2022-09-09T23:05:55.661 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:05:55 GMT /jkcgbj-54/ 2022-09-09T23:05:55.661 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9nvSoaQmTJ8l1yBLKiH4XsCIjmk= 2022-09-09T23:05:55.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:05:55 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9nvSoaQmTJ8l1yBLKiH4XsCIjmk=', 'Content-Length': '0'} 2022-09-09T23:05:55.732 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008ef7e8ed2fdcf9b2-00631bc6d3-16ce-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:05:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:05:55.733 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jkcgbj-55 2022-09-09T23:05:55.734 DEBUG:boto:path=/jkcgbj-55/ 2022-09-09T23:05:55.735 DEBUG:boto:auth_path=/jkcgbj-55/ 2022-09-09T23:05:55.736 DEBUG:boto:Method: PUT 2022-09-09T23:05:55.737 DEBUG:boto:Path: /jkcgbj-55/ 2022-09-09T23:05:55.738 DEBUG:boto:Data: 2022-09-09T23:05:55.739 DEBUG:boto:Headers: {} 2022-09-09T23:05:55.740 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T23:05:55.741 DEBUG:boto:Port: 8001 2022-09-09T23:05:55.742 DEBUG:boto:Params: {} 2022-09-09T23:05:55.743 DEBUG:boto:Token: None 2022-09-09T23:05:55.744 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:05:55 GMT /jkcgbj-55/ 2022-09-09T23:05:55.745 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:kH/UDobQjq+6Rp0LF6IHaw75RyA= 2022-09-09T23:05:55.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:05:55 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:kH/UDobQjq+6Rp0LF6IHaw75RyA=', 'Content-Length': '0'} 2022-09-09T23:05:55.798 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000003232dc768985563-00631bc6d3-157c-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:05:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:05:55.799 DEBUG:boto:path=/jkcgbj-53/ 2022-09-09T23:05:55.799 DEBUG:boto:auth_path=/jkcgbj-53/ 2022-09-09T23:05:55.800 DEBUG:boto:path=/jkcgbj-53/?policy 2022-09-09T23:05:55.800 DEBUG:boto:auth_path=/jkcgbj-53/?policy 2022-09-09T23:05:55.801 DEBUG:boto:Method: PUT 2022-09-09T23:05:55.801 DEBUG:boto:Path: /jkcgbj-53/?policy 2022-09-09T23:05:55.802 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2022-09-09T23:05:55.802 DEBUG:boto:Headers: {} 2022-09-09T23:05:55.803 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T23:05:55.803 DEBUG:boto:Port: 8000 2022-09-09T23:05:55.804 DEBUG:boto:Params: {} 2022-09-09T23:05:55.804 DEBUG:boto:Token: None 2022-09-09T23:05:55.805 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:05:55 GMT /jkcgbj-53/?policy 2022-09-09T23:05:55.805 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:EXjjqtYtu3RR+rrO9NSwbHKTQO4= 2022-09-09T23:05:55.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:05:55 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:EXjjqtYtu3RR+rrO9NSwbHKTQO4=', 'Content-Length': '97'} 2022-09-09T23:05:55.817 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008ebce5f3ec48b27c-00631bc6d3-157e-test-zone1'), ('Date', 'Fri, 09 Sep 2022 23:05:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:05:55.818 DEBUG:boto:path=/jkcgbj-53/ 2022-09-09T23:05:55.818 DEBUG:boto:auth_path=/jkcgbj-53/ 2022-09-09T23:05:55.819 DEBUG:boto:path=/jkcgbj-53/?policy 2022-09-09T23:05:55.819 DEBUG:boto:auth_path=/jkcgbj-53/?policy 2022-09-09T23:05:55.820 DEBUG:boto:Method: GET 2022-09-09T23:05:55.820 DEBUG:boto:Path: /jkcgbj-53/?policy 2022-09-09T23:05:55.821 DEBUG:boto:Data: 2022-09-09T23:05:55.821 DEBUG:boto:Headers: {} 2022-09-09T23:05:55.822 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T23:05:55.822 DEBUG:boto:Port: 8000 2022-09-09T23:05:55.823 DEBUG:boto:Params: {} 2022-09-09T23:05:55.823 DEBUG:boto:Token: None 2022-09-09T23:05:55.824 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 23:05:55 GMT /jkcgbj-53/?policy 2022-09-09T23:05:55.824 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:f1/1YHOup8qeD2yTKWZ/10Y2KWc= 2022-09-09T23:05:55.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:05:55 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:f1/1YHOup8qeD2yTKWZ/10Y2KWc=', 'Content-Length': '0'} 2022-09-09T23:05:55.827 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000077da6269657e322d-00631bc6d3-157e-test-zone1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Fri, 09 Sep 2022 23:05:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:05:55.827 DEBUG:boto:path=/jkcgbj-54/ 2022-09-09T23:05:55.828 DEBUG:boto:auth_path=/jkcgbj-54/ 2022-09-09T23:05:55.828 DEBUG:boto:path=/jkcgbj-54/?policy 2022-09-09T23:05:55.829 DEBUG:boto:auth_path=/jkcgbj-54/?policy 2022-09-09T23:05:55.829 DEBUG:boto:Method: PUT 2022-09-09T23:05:55.829 DEBUG:boto:Path: /jkcgbj-54/?policy 2022-09-09T23:05:55.830 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2022-09-09T23:05:55.830 DEBUG:boto:Headers: {} 2022-09-09T23:05:55.831 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T23:05:55.831 DEBUG:boto:Port: 8000 2022-09-09T23:05:55.832 DEBUG:boto:Params: {} 2022-09-09T23:05:55.832 DEBUG:boto:Token: None 2022-09-09T23:05:55.833 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:05:55 GMT /jkcgbj-54/?policy 2022-09-09T23:05:55.834 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:y9erjsrkJ/m1NpBYCcpg59Ictjw= 2022-09-09T23:05:55.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:05:55 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:y9erjsrkJ/m1NpBYCcpg59Ictjw=', 'Content-Length': '97'} 2022-09-09T23:05:56.858 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006f4676f3b93eb744-00631bc6d3-16ce-test-zone2'), ('Date', 'Fri, 09 Sep 2022 23:05:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:05:56.860 DEBUG:boto:path=/jkcgbj-54/ 2022-09-09T23:05:56.860 DEBUG:boto:auth_path=/jkcgbj-54/ 2022-09-09T23:05:56.861 DEBUG:boto:path=/jkcgbj-54/?policy 2022-09-09T23:05:56.862 DEBUG:boto:auth_path=/jkcgbj-54/?policy 2022-09-09T23:05:56.863 DEBUG:boto:Method: GET 2022-09-09T23:05:56.863 DEBUG:boto:Path: /jkcgbj-54/?policy 2022-09-09T23:05:56.864 DEBUG:boto:Data: 2022-09-09T23:05:56.865 DEBUG:boto:Headers: {} 2022-09-09T23:05:56.866 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T23:05:56.866 DEBUG:boto:Port: 8000 2022-09-09T23:05:56.867 DEBUG:boto:Params: {} 2022-09-09T23:05:56.868 DEBUG:boto:Token: None 2022-09-09T23:05:56.868 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 23:05:56 GMT /jkcgbj-54/?policy 2022-09-09T23:05:56.869 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:E3EhOvhcs97cbphnx7zBAigh4QY= 2022-09-09T23:05:56.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:05:56 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:E3EhOvhcs97cbphnx7zBAigh4QY=', 'Content-Length': '0'} 2022-09-09T23:05:56.871 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000024dca1290e3f39f5-00631bc6d4-16ce-test-zone2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Fri, 09 Sep 2022 23:05:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:05:56.872 DEBUG:boto:path=/jkcgbj-55/ 2022-09-09T23:05:56.872 DEBUG:boto:auth_path=/jkcgbj-55/ 2022-09-09T23:05:56.873 DEBUG:boto:path=/jkcgbj-55/?policy 2022-09-09T23:05:56.873 DEBUG:boto:auth_path=/jkcgbj-55/?policy 2022-09-09T23:05:56.874 DEBUG:boto:Method: PUT 2022-09-09T23:05:56.874 DEBUG:boto:Path: /jkcgbj-55/?policy 2022-09-09T23:05:56.875 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2022-09-09T23:05:56.875 DEBUG:boto:Headers: {} 2022-09-09T23:05:56.875 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T23:05:56.876 DEBUG:boto:Port: 8001 2022-09-09T23:05:56.877 DEBUG:boto:Params: {} 2022-09-09T23:05:56.877 DEBUG:boto:Token: None 2022-09-09T23:05:56.878 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:05:56 GMT /jkcgbj-55/?policy 2022-09-09T23:05:56.878 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:5M3iQ3ZfHUYtI4PCHQH5iNQjoNg= 2022-09-09T23:05:56.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:05:56 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:5M3iQ3ZfHUYtI4PCHQH5iNQjoNg=', 'Content-Length': '97'} 2022-09-09T23:05:57.900 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000000e543bb13ee879f-00631bc6d4-157c-test-zone3'), ('Date', 'Fri, 09 Sep 2022 23:05:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:05:57.901 DEBUG:boto:path=/jkcgbj-55/ 2022-09-09T23:05:57.902 DEBUG:boto:auth_path=/jkcgbj-55/ 2022-09-09T23:05:57.903 DEBUG:boto:path=/jkcgbj-55/?policy 2022-09-09T23:05:57.904 DEBUG:boto:auth_path=/jkcgbj-55/?policy 2022-09-09T23:05:57.905 DEBUG:boto:Method: GET 2022-09-09T23:05:57.906 DEBUG:boto:Path: /jkcgbj-55/?policy 2022-09-09T23:05:57.906 DEBUG:boto:Data: 2022-09-09T23:05:57.907 DEBUG:boto:Headers: {} 2022-09-09T23:05:57.908 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T23:05:57.909 DEBUG:boto:Port: 8001 2022-09-09T23:05:57.910 DEBUG:boto:Params: {} 2022-09-09T23:05:57.911 DEBUG:boto:Token: None 2022-09-09T23:05:57.912 DEBUG:boto:StringToSign: GET Fri, 09 Sep 2022 23:05:57 GMT /jkcgbj-55/?policy 2022-09-09T23:05:57.913 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:F1nVedMXS4n0/LeOZmBDuobdhtM= 2022-09-09T23:05:57.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:05:57 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:F1nVedMXS4n0/LeOZmBDuobdhtM=', 'Content-Length': '0'} 2022-09-09T23:05:57.917 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000054b0d05ef5c1cd0-00631bc6d5-157c-test-zone3'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Fri, 09 Sep 2022 23:05:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:05:57.918 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_policy ... ok 2022-09-09T23:05:57.919 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jkcgbj-56 2022-09-09T23:05:57.920 DEBUG:boto:path=/jkcgbj-56/ 2022-09-09T23:05:57.921 DEBUG:boto:auth_path=/jkcgbj-56/ 2022-09-09T23:05:57.922 DEBUG:boto:Method: PUT 2022-09-09T23:05:57.923 DEBUG:boto:Path: /jkcgbj-56/ 2022-09-09T23:05:57.924 DEBUG:boto:Data: 2022-09-09T23:05:57.925 DEBUG:boto:Headers: {} 2022-09-09T23:05:57.926 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T23:05:57.927 DEBUG:boto:Port: 8000 2022-09-09T23:05:57.928 DEBUG:boto:Params: {} 2022-09-09T23:05:57.929 DEBUG:boto:Token: None 2022-09-09T23:05:57.930 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:05:57 GMT /jkcgbj-56/ 2022-09-09T23:05:57.931 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:VW6XiWi1xV3Z9r1l7G5NvulQpcg= 2022-09-09T23:05:57.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:05:57 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:VW6XiWi1xV3Z9r1l7G5NvulQpcg=', 'Content-Length': '0'} 2022-09-09T23:05:57.966 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c2de2ab7841797ea-00631bc6d5-157e-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:05:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:05:57.967 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jkcgbj-57 2022-09-09T23:05:57.968 DEBUG:boto:path=/jkcgbj-57/ 2022-09-09T23:05:57.968 DEBUG:boto:auth_path=/jkcgbj-57/ 2022-09-09T23:05:57.969 DEBUG:boto:Method: PUT 2022-09-09T23:05:57.969 DEBUG:boto:Path: /jkcgbj-57/ 2022-09-09T23:05:57.970 DEBUG:boto:Data: 2022-09-09T23:05:57.970 DEBUG:boto:Headers: {} 2022-09-09T23:05:57.971 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T23:05:57.971 DEBUG:boto:Port: 8000 2022-09-09T23:05:57.972 DEBUG:boto:Params: {} 2022-09-09T23:05:57.972 DEBUG:boto:Token: None 2022-09-09T23:05:57.973 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:05:57 GMT /jkcgbj-57/ 2022-09-09T23:05:57.974 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:kyLIyJD5qKjweHc+QZlt8MY+fVA= 2022-09-09T23:05:57.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:05:57 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:kyLIyJD5qKjweHc+QZlt8MY+fVA=', 'Content-Length': '0'} 2022-09-09T23:05:58.044 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b55efcde88ddbb0b-00631bc6d5-16ce-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:05:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:05:58.046 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jkcgbj-58 2022-09-09T23:05:58.046 DEBUG:boto:path=/jkcgbj-58/ 2022-09-09T23:05:58.047 DEBUG:boto:auth_path=/jkcgbj-58/ 2022-09-09T23:05:58.047 DEBUG:boto:Method: PUT 2022-09-09T23:05:58.048 DEBUG:boto:Path: /jkcgbj-58/ 2022-09-09T23:05:58.049 DEBUG:boto:Data: 2022-09-09T23:05:58.049 DEBUG:boto:Headers: {} 2022-09-09T23:05:58.050 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T23:05:58.050 DEBUG:boto:Port: 8001 2022-09-09T23:05:58.051 DEBUG:boto:Params: {} 2022-09-09T23:05:58.052 DEBUG:boto:Token: None 2022-09-09T23:05:58.052 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:05:58 GMT /jkcgbj-58/ 2022-09-09T23:05:58.053 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:moAH4bk15YGGqF1W9DI8PGdDnTA= 2022-09-09T23:05:58.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:05:58 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:moAH4bk15YGGqF1W9DI8PGdDnTA=', 'Content-Length': '0'} 2022-09-09T23:05:58.106 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000eb1e0035d0cb3344-00631bc6d6-157c-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:05:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:05:58.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:05:58.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:05:58.108 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:05:58.155 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:05:58.156 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:05:58.212 INFO:teuthology.orchestra.run.smithi005.stderr:No --period given, using current period=fd4bc158-06fd-4840-8a27-33fabda67b99 2022-09-09T23:05:58.240 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T23:05:58.241 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:58.241 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T23:05:58.241 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T23:05:58.241 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:58.242 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:58.242 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T23:05:58.242 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T23:05:58.242 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:58.243 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:58.243 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T23:05:58.243 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T23:05:58.243 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:05:58.244 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:05:58.244 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662764758.090037_627.1", 2022-09-09T23:05:58.244 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T23:05:58.090037Z" 2022-09-09T23:05:58.244 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:05:58.245 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T23:05:58.246 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:05:58.246 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-09T23:05:58.247 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:05:58.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:05:58.248 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:05:58.296 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:05:58.296 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:05:58.351 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:05:58.348+0000 7f8e69a67dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:05:58.352 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:05:58.348+0000 7f8e69a67dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:05:58.377 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:05:58.378 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:05:58.378 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:05:58.378 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:05:58.379 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:05:58.379 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:05:58.379 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:05:58.379 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:58.380 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:05:58.380 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:05:58.380 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:05:58.380 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:05:58.381 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:05:58.381 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:05:58.381 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:05:58.381 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:05:58.382 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:05:58.382 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:05:58.382 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:05:58.382 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:05:58.383 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:58.383 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:05:58.383 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:05:58.383 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:05:58.384 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:05:58.384 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:05:58.384 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:05:58.384 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:05:58.384 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:05:58.385 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:05:58.385 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:05:58.385 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:05:58.385 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:58.386 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:05:58.386 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:05:58.386 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:05:58.386 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:05:58.387 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:05:58.387 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:05:58.387 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:05:58.387 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:05:58.388 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:05:58.388 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:05:58.388 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:05:58.388 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:58.389 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:05:58.389 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:05:58.389 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:05:58.389 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:05:58.390 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764501.249520_420.1", 2022-09-09T23:05:58.390 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:05:58.390 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:05:58.390 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:05:58.391 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:01:41.249520Z", 2022-09-09T23:05:58.391 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:05:58.391 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:05:58.391 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:05:58.392 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:05:58.392 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:58.392 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:05:58.392 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:05:58.393 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:05:58.393 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:05:58.393 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:05:58Z" 2022-09-09T23:05:58.393 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:05:58.395 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764501.249520_420.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:01:41.249520Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:05:58Z" } 2022-09-09T23:05:58.395 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764501.249520_420.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:01:41.249520Z', 'realm_epoch': 4}}] 2022-09-09T23:05:58.396 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:05:58.396 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:05:58.397 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764501.249520_420.1 2022-09-09T23:06:03.398 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:06:03.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:06:03.400 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:06:03.448 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:06:03.448 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:06:03.519 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:06:03.516+0000 7f4ebe65ddc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:06:03.521 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:06:03.520+0000 7f4ebe65ddc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:06:03.548 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:06:03.549 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:06:03.549 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:06:03.549 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:06:03.549 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:06:03.550 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:06:03.550 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:06:03.550 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:03.551 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:06:03.551 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:03.551 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:06:03.551 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:03.552 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:03.552 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:06:03.552 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:03.553 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:06:03.553 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:03.553 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:06:03.554 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:03.554 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:03.554 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:03.554 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:03.555 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:06:03.555 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:03.555 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:03.556 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:06:03.556 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:03.556 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:03.556 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:03.557 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:06:03.557 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:03.557 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:03.557 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:03.558 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:03.558 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:06:03.558 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:03.559 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:03.559 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:06:03.559 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:03.570 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:03.571 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:03.571 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:06:03.572 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:03.572 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:03.572 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:03.573 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:03.573 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:06:03.573 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:03.573 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:03.573 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764501.249520_420.1", 2022-09-09T23:06:03.574 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:03.574 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:03.574 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:03.574 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:01:41.249520Z", 2022-09-09T23:06:03.575 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:06:03.575 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:03.575 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:03.575 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:06:03.576 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:03.576 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:06:03.576 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:06:03.577 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:06:03.577 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:03.577 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:06:03Z" 2022-09-09T23:06:03.578 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:06:03.579 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764501.249520_420.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:01:41.249520Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:06:03Z" } 2022-09-09T23:06:03.580 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764501.249520_420.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:01:41.249520Z', 'realm_epoch': 4}}] 2022-09-09T23:06:03.581 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:06:03.581 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:06:03.582 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764501.249520_420.1 2022-09-09T23:06:08.583 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:06:08.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:06:08.585 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:06:08.637 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:06:08.637 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:06:08.691 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:06:08.688+0000 7fa204c29dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:06:08.693 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:06:08.692+0000 7fa204c29dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:06:08.728 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:06:08.728 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:06:08.729 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:06:08.729 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:06:08.729 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:06:08.729 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:06:08.730 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:06:08.730 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:08.730 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:06:08.730 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:08.731 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:06:08.731 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:08.731 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:08.732 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:06:08.732 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:08.732 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:06:08.732 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:08.733 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:06:08.733 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:08.733 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:08.733 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:08.734 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:08.735 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:06:08.735 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:08.736 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:08.736 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:06:08.737 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:08.738 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:08.738 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:08.739 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:06:08.739 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:08.740 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:08.740 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:08.741 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:08.741 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:06:08.742 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:08.743 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:08.743 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:06:08.744 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:08.745 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:08.745 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:08.746 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:06:08.746 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:08.747 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:08.747 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:08.748 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:08.748 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:06:08.749 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:08.749 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:08.750 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764501.249520_420.1", 2022-09-09T23:06:08.750 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:08.751 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:08.751 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:08.752 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:01:41.249520Z", 2022-09-09T23:06:08.752 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:06:08.753 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:08.753 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:08.754 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:06:08.754 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:08.755 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:06:08.755 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:06:08.756 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:06:08.756 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:08.757 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:06:08Z" 2022-09-09T23:06:08.757 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:06:08.759 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764501.249520_420.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:01:41.249520Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:06:08Z" } 2022-09-09T23:06:08.760 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764501.249520_420.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:01:41.249520Z', 'realm_epoch': 4}}] 2022-09-09T23:06:08.761 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:06:08.762 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:06:08.763 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764501.249520_420.1 2022-09-09T23:06:13.765 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:06:13.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:06:13.767 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:06:13.818 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:06:13.818 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:06:13.877 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:06:13.876+0000 7fae1d44cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:06:13.879 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:06:13.876+0000 7fae1d44cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:06:13.908 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:06:13.909 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:06:13.909 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:06:13.909 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:06:13.910 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:06:13.910 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:06:13.910 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:06:13.910 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:13.911 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:06:13.911 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:13.911 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:06:13.911 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:13.912 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:13.912 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:06:13.912 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:13.912 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:06:13.913 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:13.913 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:06:13.913 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:13.913 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:13.914 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:13.914 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:13.914 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:06:13.914 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:13.915 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:13.915 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:06:13.915 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:13.915 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:13.915 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:13.916 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:06:13.916 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:13.916 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:13.917 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:13.917 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:13.917 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:06:13.918 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:13.918 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:13.918 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:06:13.918 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:13.919 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:13.919 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:13.919 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:06:13.919 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:13.920 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:13.920 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:13.920 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:13.920 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:06:13.921 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:13.921 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:13.921 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764501.249520_420.1", 2022-09-09T23:06:13.921 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:13.922 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:13.922 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:13.922 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:01:41.249520Z", 2022-09-09T23:06:13.922 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:06:13.923 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:13.923 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:13.923 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:06:13.923 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:13.924 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:06:13.924 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:06:13.924 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:06:13.924 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:13.925 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:06:13Z" 2022-09-09T23:06:13.925 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:06:13.926 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764501.249520_420.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:01:41.249520Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:06:13Z" } 2022-09-09T23:06:13.927 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764501.249520_420.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:01:41.249520Z', 'realm_epoch': 4}}] 2022-09-09T23:06:13.927 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:06:13.928 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:06:13.928 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764501.249520_420.1 2022-09-09T23:06:18.930 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:06:18.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:06:18.932 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:06:18.980 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:06:18.981 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:06:19.036 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:06:19.032+0000 7f0359606dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:06:19.038 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:06:19.036+0000 7f0359606dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:06:19.067 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:06:19.067 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:06:19.068 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:06:19.068 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:06:19.068 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:06:19.068 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:06:19.069 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:06:19.069 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:19.069 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:06:19.069 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:19.070 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:06:19.070 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:19.070 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:19.071 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:06:19.071 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:19.071 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:06:19.072 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:19.072 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:06:19.072 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:19.072 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:19.073 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:19.073 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:19.073 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:06:19.073 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:19.074 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:19.074 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:06:19.074 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:19.074 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:19.075 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:19.075 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:06:19.075 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:19.076 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:19.076 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:19.076 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:19.076 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:06:19.076 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:19.077 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:19.077 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:06:19.077 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:19.078 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:19.078 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:19.078 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:06:19.078 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:19.079 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:19.079 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:19.079 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:19.079 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:06:19.080 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:19.080 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:19.080 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764501.249520_420.1", 2022-09-09T23:06:19.080 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:19.081 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:19.081 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:19.081 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:01:41.249520Z", 2022-09-09T23:06:19.081 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:06:19.082 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:19.082 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:19.082 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:06:19.082 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:19.083 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:06:19.083 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:06:19.083 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:06:19.083 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:19.084 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:06:19Z" 2022-09-09T23:06:19.084 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:06:19.085 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764501.249520_420.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:01:41.249520Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:06:19Z" } 2022-09-09T23:06:19.086 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764501.249520_420.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:01:41.249520Z', 'realm_epoch': 4}}] 2022-09-09T23:06:19.087 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:06:19.087 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:06:19.087 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764501.249520_420.1 2022-09-09T23:06:24.089 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:06:24.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:06:24.091 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:06:24.137 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:06:24.137 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:06:24.205 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:06:24.204+0000 7f1ef9445dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:06:24.207 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:06:24.204+0000 7f1ef9445dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:06:24.240 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:06:24.241 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:06:24.241 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:06:24.242 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:06:24.242 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:06:24.243 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:06:24.243 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:06:24.244 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:24.244 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:06:24.245 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:24.245 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:06:24.246 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:24.246 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:24.247 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:06:24.247 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:24.247 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:06:24.248 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:24.249 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:06:24.249 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:24.250 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:24.250 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:24.250 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:24.251 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:06:24.251 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:24.252 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:24.252 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:06:24.253 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:24.253 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:24.254 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:24.254 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:06:24.255 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:24.256 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:24.256 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:24.257 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:24.257 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:06:24.258 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:24.259 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:24.259 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:06:24.260 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:24.260 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:24.261 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:24.261 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:06:24.262 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:24.263 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:24.263 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:24.264 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:24.264 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:06:24.265 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:24.265 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:24.266 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764501.249520_420.1", 2022-09-09T23:06:24.267 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:24.267 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:24.268 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:24.268 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:01:41.249520Z", 2022-09-09T23:06:24.269 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:06:24.269 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:24.270 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:24.271 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:06:24.271 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:24.271 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:06:24.271 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:06:24.272 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:06:24.272 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:24.272 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:06:24Z" 2022-09-09T23:06:24.273 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:06:24.274 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764501.249520_420.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:01:41.249520Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:06:24Z" } 2022-09-09T23:06:24.275 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764501.249520_420.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:01:41.249520Z', 'realm_epoch': 4}}] 2022-09-09T23:06:24.275 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:06:24.276 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:06:24.276 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764501.249520_420.1 2022-09-09T23:06:29.278 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:06:29.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:06:29.280 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:06:29.328 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:06:29.329 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:06:29.392 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:06:29.388+0000 7fef9276ddc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:06:29.394 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:06:29.392+0000 7fef9276ddc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:06:29.424 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:06:29.425 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:06:29.425 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:06:29.426 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:06:29.426 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:06:29.426 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:06:29.427 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:06:29.427 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:29.427 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:06:29.427 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:29.428 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:06:29.428 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:29.428 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:29.428 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:06:29.429 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:29.429 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:06:29.429 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:29.429 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:06:29.430 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:29.430 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:29.430 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:29.430 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:29.431 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:06:29.431 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:29.431 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:29.431 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:06:29.432 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:29.432 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:29.434 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:29.434 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:06:29.434 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:29.435 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:29.435 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:29.435 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:29.436 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:06:29.436 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:29.436 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:29.437 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:06:29.437 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:29.437 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:29.437 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:29.438 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:06:29.438 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:29.438 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:29.438 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:29.439 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:29.439 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:06:29.439 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:29.439 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:29.440 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764501.249520_420.1", 2022-09-09T23:06:29.440 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:29.440 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:29.440 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:29.441 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:01:41.249520Z", 2022-09-09T23:06:29.441 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:06:29.441 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:29.442 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:29.442 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:06:29.442 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:29.442 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:06:29.443 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:06:29.443 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:06:29.443 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:29.443 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:06:29Z" 2022-09-09T23:06:29.444 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:06:29.445 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764501.249520_420.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:01:41.249520Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:06:29Z" } 2022-09-09T23:06:29.446 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764501.249520_420.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:01:41.249520Z', 'realm_epoch': 4}}] 2022-09-09T23:06:29.446 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:06:29.447 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:06:29.447 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764501.249520_420.1 2022-09-09T23:06:34.448 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:06:34.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:06:34.450 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:06:34.498 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:06:34.499 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:06:34.573 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:06:34.568+0000 7f25ff2b5dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:06:34.574 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:06:34.572+0000 7f25ff2b5dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:06:34.607 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:06:34.607 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:06:34.608 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:06:34.609 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:06:34.609 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:06:34.610 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:06:34.611 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:06:34.612 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:34.613 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:06:34.614 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:34.614 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:06:34.615 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:34.615 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:34.616 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:06:34.616 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:34.617 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:06:34.618 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:34.618 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:06:34.619 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:34.619 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:34.620 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:34.621 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:34.621 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:06:34.622 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:34.623 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:34.623 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:06:34.625 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:34.625 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:34.625 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:34.626 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:06:34.626 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:34.626 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:34.626 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:34.627 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:34.627 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:06:34.627 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:34.628 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:34.628 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:06:34.628 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:34.628 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:34.629 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:34.629 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:06:34.629 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:34.629 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:34.630 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:34.630 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:34.631 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:06:34.631 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:34.631 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:34.632 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764501.249520_420.1", 2022-09-09T23:06:34.632 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:34.632 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:34.632 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:34.633 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:01:41.249520Z", 2022-09-09T23:06:34.633 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:06:34.633 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:34.633 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:34.634 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:06:34.634 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:34.634 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:06:34.634 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:06:34.635 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:06:34.635 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:34.635 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:06:34Z" 2022-09-09T23:06:34.635 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:06:34.637 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764501.249520_420.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:01:41.249520Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:06:34Z" } 2022-09-09T23:06:34.638 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764501.249520_420.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:01:41.249520Z', 'realm_epoch': 4}}] 2022-09-09T23:06:34.638 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:06:34.639 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:06:34.639 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764501.249520_420.1 2022-09-09T23:06:39.640 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:06:39.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:06:39.643 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:06:39.692 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:06:39.693 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:06:39.749 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:06:39.743+0000 7f4a24ac0dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:06:39.751 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:06:39.747+0000 7f4a24ac0dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:06:39.778 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:06:39.779 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:06:39.780 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:06:39.780 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:06:39.781 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:06:39.781 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:06:39.782 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:06:39.783 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:39.783 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:06:39.783 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:39.783 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:06:39.784 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:39.784 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:39.784 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:06:39.784 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:39.785 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:06:39.785 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:39.785 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:06:39.785 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:39.786 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:39.786 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:39.786 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:39.786 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:06:39.787 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:39.787 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:39.787 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:06:39.787 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:39.788 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:39.788 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:39.788 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:06:39.788 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:39.789 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:39.789 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:39.789 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:39.789 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:06:39.790 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:39.790 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:39.790 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:06:39.790 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:39.791 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:39.791 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:39.791 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:06:39.792 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:39.792 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:39.792 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:39.792 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:39.792 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:06:39.793 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:39.793 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:39.793 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764501.249520_420.1", 2022-09-09T23:06:39.793 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:39.794 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:39.794 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:39.794 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:01:41.249520Z", 2022-09-09T23:06:39.794 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:06:39.795 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:39.795 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:39.795 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:06:39.795 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:39.796 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:06:39.796 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:06:39.796 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:06:39.796 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:39.797 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:06:39Z" 2022-09-09T23:06:39.797 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:06:39.799 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764501.249520_420.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:01:41.249520Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:06:39Z" } 2022-09-09T23:06:39.799 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764501.249520_420.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:01:41.249520Z', 'realm_epoch': 4}}] 2022-09-09T23:06:39.800 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:06:39.800 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:06:39.801 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764501.249520_420.1 2022-09-09T23:06:44.802 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:06:44.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:06:44.804 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:06:44.853 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:06:44.854 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:06:44.923 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:06:44.919+0000 7f389a6f1dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:06:44.925 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:06:44.923+0000 7f389a6f1dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:06:44.958 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:06:44.958 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:06:44.966 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:06:44.967 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:06:44.967 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:06:44.968 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:06:44.969 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:06:44.969 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:44.969 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:06:44.970 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:44.970 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:06:44.971 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:44.972 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:44.972 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:06:44.973 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:44.973 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:06:44.974 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:44.974 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:06:44.975 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:44.975 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:44.976 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:44.976 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:44.977 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:06:44.977 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:44.978 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:44.978 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:06:44.979 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:44.979 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:44.979 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:44.980 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:06:44.980 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:44.981 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:44.981 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:44.982 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:44.982 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:06:44.983 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:44.983 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:44.984 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:06:44.984 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:44.984 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:44.985 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:44.985 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:06:44.985 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:44.985 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:44.986 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:44.986 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:44.986 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:06:44.986 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:44.987 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:44.987 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764501.249520_420.1", 2022-09-09T23:06:44.987 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:44.987 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:44.988 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:44.988 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:01:41.249520Z", 2022-09-09T23:06:44.988 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:06:44.989 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:44.989 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:44.989 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:06:44.989 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:44.990 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:06:44.990 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:06:44.990 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:06:44.990 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:44.991 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:06:44Z" 2022-09-09T23:06:44.991 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:06:44.992 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764501.249520_420.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:01:41.249520Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:06:44Z" } 2022-09-09T23:06:44.993 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764501.249520_420.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:01:41.249520Z', 'realm_epoch': 4}}] 2022-09-09T23:06:44.994 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:06:44.994 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:06:44.995 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764501.249520_420.1 2022-09-09T23:06:49.996 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:06:49.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:06:49.998 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:06:50.045 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:06:50.046 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:06:50.106 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:06:50.103+0000 7ffa324fadc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:06:50.107 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:06:50.103+0000 7ffa324fadc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:06:50.136 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:06:50.136 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:06:50.137 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:06:50.137 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:06:50.138 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:06:50.138 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:06:50.139 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:06:50.139 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:50.140 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:06:50.140 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:50.141 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:06:50.141 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:50.142 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:50.142 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:06:50.143 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:50.143 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:06:50.144 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:50.144 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:06:50.145 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:50.145 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:50.146 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:50.148 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:50.148 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:06:50.149 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:50.150 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:50.150 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:06:50.151 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:50.151 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:50.152 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:50.152 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:06:50.153 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:50.153 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:50.154 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:50.154 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:50.156 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:06:50.156 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:50.157 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:50.157 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:06:50.157 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:50.157 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:50.158 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:50.158 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:06:50.158 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:50.159 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:50.159 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:50.159 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:50.160 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:06:50.160 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:50.161 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:50.162 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764501.249520_420.1", 2022-09-09T23:06:50.162 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:50.163 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:50.163 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:50.164 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:01:41.249520Z", 2022-09-09T23:06:50.164 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:06:50.164 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:50.165 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:50.165 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:06:50.165 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:50.165 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:06:50.166 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:06:50.166 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:06:50.166 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:50.167 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:06:50Z" 2022-09-09T23:06:50.167 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:06:50.168 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764501.249520_420.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:01:41.249520Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:06:50Z" } 2022-09-09T23:06:50.169 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764501.249520_420.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:01:41.249520Z', 'realm_epoch': 4}}] 2022-09-09T23:06:50.170 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:06:50.170 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:06:50.171 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764501.249520_420.1 2022-09-09T23:06:55.173 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:06:55.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:06:55.176 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:06:55.224 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:06:55.225 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:06:55.289 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:06:55.287+0000 7fd55ca1cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:06:55.291 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:06:55.287+0000 7fd55ca1cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:06:55.321 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:06:55.321 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:06:55.321 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:06:55.322 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:06:55.322 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:06:55.322 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:06:55.322 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:06:55.323 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:55.323 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:06:55.323 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:55.323 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:06:55.324 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:55.324 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:55.324 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:06:55.324 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:55.325 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:06:55.325 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:55.325 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:06:55.325 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:55.326 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:55.326 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:55.326 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:55.326 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:06:55.326 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:55.327 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:55.327 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:06:55.327 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:55.327 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:55.328 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:55.328 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:06:55.328 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:55.328 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:55.329 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:55.329 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:55.329 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:06:55.329 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:55.330 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:55.330 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:06:55.330 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:55.330 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:55.331 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:55.331 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:06:55.331 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:06:55.331 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:55.332 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:55.332 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:06:55.332 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:06:55.332 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:06:55.333 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:06:55.333 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764501.249520_420.1", 2022-09-09T23:06:55.333 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:06:55.333 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:06:55.334 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:06:55.334 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:01:41.249520Z", 2022-09-09T23:06:55.334 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:06:55.334 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:55.335 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:06:55.335 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:06:55.335 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:55.335 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:06:55.335 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:06:55.336 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:06:55.336 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:06:55.336 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:06:55Z" 2022-09-09T23:06:55.336 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:06:55.338 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764501.249520_420.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:01:41.249520Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:06:55Z" } 2022-09-09T23:06:55.338 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764501.249520_420.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:01:41.249520Z', 'realm_epoch': 4}}] 2022-09-09T23:06:55.339 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:06:55.339 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:06:55.340 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764501.249520_420.1 2022-09-09T23:07:00.342 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:07:00.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:07:00.343 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:07:00.392 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:07:00.392 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:07:00.456 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:07:00.451+0000 7f85d6c02dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:07:00.458 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:07:00.455+0000 7f85d6c02dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:07:00.485 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:07:00.486 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:07:00.487 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:07:00.487 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:07:00.488 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:07:00.488 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:07:00.489 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:07:00.489 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:00.490 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:07:00.490 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:00.491 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:07:00.491 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:00.492 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:00.492 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:07:00.493 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:00.493 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:07:00.494 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:00.495 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:07:00.495 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:00.496 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:00.496 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:00.497 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:00.497 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:07:00.498 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:00.498 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:00.499 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:07:00.499 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:00.500 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:00.500 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:00.501 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:07:00.501 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:00.502 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:00.502 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:00.503 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:00.504 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:07:00.504 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:00.505 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:00.505 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:07:00.506 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:00.506 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:00.507 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:00.508 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:07:00.508 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:00.509 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:00.509 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:00.510 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:00.510 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:07:00.511 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:00.511 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:00.512 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764501.249520_420.1", 2022-09-09T23:07:00.512 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:00.513 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:00.513 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:00.514 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:01:41.249520Z", 2022-09-09T23:07:00.514 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:07:00.515 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:00.515 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:00.516 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:07:00.516 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:00.517 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:07:00.517 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:07:00.518 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:07:00.518 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:00.519 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:07:00Z" 2022-09-09T23:07:00.519 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:07:00.521 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764501.249520_420.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:01:41.249520Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:07:00Z" } 2022-09-09T23:07:00.523 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764501.249520_420.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:01:41.249520Z', 'realm_epoch': 4}}] 2022-09-09T23:07:00.524 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:07:00.525 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:07:00.526 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764501.249520_420.1 2022-09-09T23:07:05.528 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:07:05.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:07:05.530 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:07:05.577 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:07:05.577 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:07:05.645 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:07:05.643+0000 7fbebce55dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:07:05.648 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:07:05.643+0000 7fbebce55dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:07:05.685 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:07:05.685 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:07:05.686 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:07:05.686 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:07:05.686 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:07:05.686 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:07:05.687 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:07:05.687 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:05.687 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:07:05.688 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:05.688 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:07:05.689 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:05.689 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:05.690 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:07:05.691 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:05.691 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:07:05.692 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:05.692 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:07:05.693 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:05.694 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:05.697 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:05.697 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:05.697 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:07:05.698 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:05.698 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:05.698 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:07:05.698 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:05.699 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:05.699 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:05.699 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:07:05.699 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:05.700 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:05.700 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:05.700 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:05.700 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:07:05.701 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:05.701 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:05.701 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:07:05.702 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:05.702 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:05.702 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:05.702 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:07:05.703 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:05.703 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:05.703 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:05.703 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:05.704 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:07:05.704 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:05.704 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:05.704 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764501.249520_420.1", 2022-09-09T23:07:05.705 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:05.705 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:05.705 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:05.705 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:01:41.249520Z", 2022-09-09T23:07:05.706 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:07:05.706 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:05.706 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:05.706 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:07:05.707 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:05.707 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:07:05.707 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:07:05.707 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:07:05.708 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:05.708 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:07:05Z" 2022-09-09T23:07:05.708 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:07:05.710 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764501.249520_420.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:01:41.249520Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:07:05Z" } 2022-09-09T23:07:05.710 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764501.249520_420.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:01:41.249520Z', 'realm_epoch': 4}}] 2022-09-09T23:07:05.711 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:07:05.711 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:07:05.712 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764501.249520_420.1 2022-09-09T23:07:10.713 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:07:10.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:07:10.715 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:07:10.763 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:07:10.764 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:07:10.827 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:07:10.827+0000 7f850c990dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:07:10.829 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:07:10.831+0000 7f850c990dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:07:10.857 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:07:10.857 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:07:10.858 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:07:10.858 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:07:10.859 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:07:10.859 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:07:10.859 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:07:10.859 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:10.860 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:07:10.860 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:10.860 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:07:10.861 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:10.861 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:10.861 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:07:10.861 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:10.862 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:07:10.862 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:10.862 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:07:10.862 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:10.863 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:10.863 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:10.863 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:10.864 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:07:10.864 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:10.864 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:10.864 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:07:10.865 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:10.865 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:10.865 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:10.865 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:07:10.866 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:10.866 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:10.866 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:10.866 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:10.867 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:07:10.867 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:10.867 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:10.868 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:07:10.868 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:10.869 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:10.869 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:10.869 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:07:10.870 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:10.870 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:10.870 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:10.870 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:10.871 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:07:10.871 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:10.871 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:10.872 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764501.249520_420.1", 2022-09-09T23:07:10.872 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:10.872 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:10.873 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:10.873 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:01:41.249520Z", 2022-09-09T23:07:10.873 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:07:10.873 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:10.874 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:10.874 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:07:10.874 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:10.874 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:07:10.875 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:07:10.875 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:07:10.875 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:10.876 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:07:10Z" 2022-09-09T23:07:10.876 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:07:10.877 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764501.249520_420.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:01:41.249520Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:07:10Z" } 2022-09-09T23:07:10.878 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764501.249520_420.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:01:41.249520Z', 'realm_epoch': 4}}] 2022-09-09T23:07:10.879 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:07:10.879 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:07:10.880 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764501.249520_420.1 2022-09-09T23:07:15.881 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:07:15.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:07:15.885 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:07:15.938 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:07:15.938 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:07:15.998 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:07:15.999+0000 7f2cf0535dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:07:16.000 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:07:15.999+0000 7f2cf0535dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:07:16.031 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:07:16.032 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:07:16.032 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:07:16.032 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:07:16.033 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:07:16.033 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:07:16.034 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:07:16.035 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:16.035 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:07:16.036 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:16.036 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:07:16.037 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:16.037 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:16.038 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:07:16.038 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:16.038 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:07:16.039 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:16.039 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:07:16.039 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:16.040 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:16.040 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:16.040 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:16.041 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:07:16.041 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:16.041 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:16.041 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:07:16.042 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:16.042 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:16.042 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:16.042 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:07:16.043 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:16.043 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:16.043 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:16.043 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:16.044 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:07:16.044 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:16.044 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:16.044 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:07:16.045 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:16.045 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:16.046 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:16.046 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:07:16.046 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:16.047 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:16.047 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:16.047 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:16.047 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:07:16.048 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:16.048 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:16.048 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764501.249520_420.1", 2022-09-09T23:07:16.048 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:16.049 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:16.049 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:16.049 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:01:41.249520Z", 2022-09-09T23:07:16.051 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:07:16.051 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:16.052 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:16.052 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:07:16.053 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:16.054 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:07:16.054 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:07:16.055 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:07:16.055 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:16.056 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:07:16Z" 2022-09-09T23:07:16.056 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:07:16.057 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764501.249520_420.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:01:41.249520Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:07:16Z" } 2022-09-09T23:07:16.058 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764501.249520_420.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:01:41.249520Z', 'realm_epoch': 4}}] 2022-09-09T23:07:16.059 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:07:16.059 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:07:16.059 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764501.249520_420.1 2022-09-09T23:07:21.061 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:07:21.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:07:21.064 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:07:21.112 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:07:21.112 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:07:21.175 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:07:21.175+0000 7f95e30dcdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:07:21.176 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:07:21.179+0000 7f95e30dcdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:07:21.206 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:07:21.206 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:07:21.207 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:07:21.207 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:07:21.208 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:07:21.208 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:07:21.209 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:07:21.209 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:21.210 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:07:21.210 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:21.211 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:07:21.211 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:21.211 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:21.211 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:07:21.212 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:21.212 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:07:21.212 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:21.212 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:07:21.212 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:21.213 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:21.213 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:21.213 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:21.213 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:07:21.214 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:21.214 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:21.214 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:07:21.215 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:21.215 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:21.215 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:21.215 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:07:21.216 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:21.216 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:21.216 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:21.216 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:21.217 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:07:21.217 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:21.217 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:21.218 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:07:21.218 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:21.218 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:21.219 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:21.219 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:07:21.219 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:21.219 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:21.220 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:21.220 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:21.220 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:07:21.220 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:21.221 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:21.221 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764501.249520_420.1", 2022-09-09T23:07:21.221 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:21.221 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:21.222 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:21.222 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:01:41.249520Z", 2022-09-09T23:07:21.222 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:07:21.222 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:21.223 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:21.223 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:07:21.223 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:21.223 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:07:21.224 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:07:21.224 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:07:21.224 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:21.224 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:07:21Z" 2022-09-09T23:07:21.225 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:07:21.226 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764501.249520_420.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:01:41.249520Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:07:21Z" } 2022-09-09T23:07:21.227 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764501.249520_420.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:01:41.249520Z', 'realm_epoch': 4}}] 2022-09-09T23:07:21.227 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:07:21.228 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:07:21.228 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764501.249520_420.1 2022-09-09T23:07:26.229 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:07:26.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:07:26.231 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:07:26.280 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:07:26.281 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:07:26.350 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:07:26.351+0000 7f2052b78dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:07:26.352 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:07:26.351+0000 7f2052b78dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:07:26.383 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:07:26.384 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:07:26.384 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:07:26.385 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:07:26.385 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:07:26.385 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:07:26.386 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:07:26.386 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:26.387 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:07:26.387 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:26.388 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:07:26.388 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:26.389 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:26.389 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:07:26.390 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:26.390 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:07:26.391 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:26.391 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:07:26.392 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:26.393 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:26.393 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:26.393 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:26.394 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:07:26.394 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:26.394 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:26.394 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:07:26.395 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:26.395 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:26.395 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:26.396 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:07:26.396 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:26.396 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:26.396 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:26.397 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:26.397 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:07:26.397 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:26.397 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:26.398 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:07:26.398 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:26.398 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:26.398 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:26.399 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:07:26.399 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:26.399 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:26.399 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:26.400 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:26.400 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:07:26.400 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:26.401 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:26.401 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764501.249520_420.1", 2022-09-09T23:07:26.401 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:26.401 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:26.402 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:26.402 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:01:41.249520Z", 2022-09-09T23:07:26.402 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:07:26.403 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:26.403 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:26.403 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:07:26.403 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:26.404 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:07:26.404 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:07:26.405 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:07:26.405 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:26.406 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:07:26Z" 2022-09-09T23:07:26.406 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:07:26.407 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764501.249520_420.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:01:41.249520Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:07:26Z" } 2022-09-09T23:07:26.408 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764501.249520_420.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:01:41.249520Z', 'realm_epoch': 4}}] 2022-09-09T23:07:26.408 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:07:26.409 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:07:26.410 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764501.249520_420.1 2022-09-09T23:07:31.411 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:07:31.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:07:31.413 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:07:31.460 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:07:31.461 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:07:31.518 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:07:31.518+0000 7f032331edc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:07:31.520 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:07:31.518+0000 7f032331edc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:07:31.551 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:07:31.552 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:07:31.552 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:07:31.552 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:07:31.553 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:07:31.553 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:07:31.553 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:07:31.553 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:31.554 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:07:31.554 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:31.554 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:07:31.555 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:31.555 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:31.555 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:07:31.556 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:31.556 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:07:31.556 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:31.557 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:07:31.557 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:31.557 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:31.558 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:31.558 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:31.558 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:07:31.558 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:31.559 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:31.559 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:07:31.559 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:31.560 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:31.560 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:31.560 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:07:31.560 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:31.561 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:31.561 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:31.561 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:31.561 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:07:31.562 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:31.562 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:31.562 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:07:31.562 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:31.563 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:31.563 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:31.563 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:07:31.563 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:31.564 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:31.564 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:31.564 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:31.564 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:07:31.565 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:31.565 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:31.565 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764501.249520_420.1", 2022-09-09T23:07:31.566 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:31.566 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:31.566 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:31.566 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:01:41.249520Z", 2022-09-09T23:07:31.567 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:07:31.567 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:31.567 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:31.567 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:07:31.568 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:31.568 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:07:31.568 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:07:31.568 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:07:31.569 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:31.569 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:07:31Z" 2022-09-09T23:07:31.569 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:07:31.570 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764501.249520_420.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:01:41.249520Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:07:31Z" } 2022-09-09T23:07:31.571 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764501.249520_420.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:01:41.249520Z', 'realm_epoch': 4}}] 2022-09-09T23:07:31.572 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:07:31.572 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:07:31.573 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764501.249520_420.1 2022-09-09T23:07:36.574 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:07:36.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:07:36.576 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:07:36.625 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:07:36.626 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:07:36.688 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:07:36.686+0000 7f397ea97dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:07:36.689 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:07:36.690+0000 7f397ea97dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:07:36.725 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:07:36.726 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:07:36.726 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:07:36.726 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:07:36.726 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:07:36.727 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:07:36.727 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:07:36.727 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:36.727 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:07:36.728 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:36.728 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:07:36.728 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:36.728 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:36.729 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:07:36.729 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:36.730 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:07:36.730 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:36.731 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:07:36.731 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:36.732 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:36.732 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:36.733 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:36.733 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:07:36.733 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:36.734 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:36.734 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:07:36.735 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:36.735 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:36.736 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:36.736 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:07:36.737 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:36.737 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:36.738 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:36.738 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:36.739 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:07:36.739 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:36.740 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:36.740 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:07:36.741 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:36.741 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:36.742 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:36.742 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:07:36.743 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:36.743 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:36.743 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:36.744 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:36.744 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:07:36.745 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:36.745 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:36.746 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764501.249520_420.1", 2022-09-09T23:07:36.746 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:36.747 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:36.747 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:36.748 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:01:41.249520Z", 2022-09-09T23:07:36.748 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:07:36.749 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:36.749 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:36.750 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:07:36.750 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:36.750 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:07:36.751 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:07:36.751 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:07:36.751 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:36.751 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:07:36Z" 2022-09-09T23:07:36.752 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:07:36.754 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764501.249520_420.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:01:41.249520Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:07:36Z" } 2022-09-09T23:07:36.755 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764501.249520_420.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:01:41.249520Z', 'realm_epoch': 4}}] 2022-09-09T23:07:36.756 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:07:36.757 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:07:36.757 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764501.249520_420.1 2022-09-09T23:07:41.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:07:41.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:07:41.761 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:07:41.804 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:07:41.805 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:07:41.860 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:07:41.858+0000 7f9825003dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:07:41.862 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:07:41.862+0000 7f9825003dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:07:41.890 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:07:41.890 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:07:41.890 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:07:41.891 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:07:41.891 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:07:41.891 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:07:41.891 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:07:41.892 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:41.892 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:07:41.892 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:41.892 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:07:41.893 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:41.893 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:41.893 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:07:41.893 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:41.894 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:07:41.894 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:41.894 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:07:41.894 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:41.895 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:41.895 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:41.895 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:41.895 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:07:41.896 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:41.896 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:41.896 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:07:41.896 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:41.896 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:41.897 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:41.897 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:07:41.897 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:41.897 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:41.898 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:41.898 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:41.898 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:07:41.898 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:41.899 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:41.899 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:07:41.899 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:41.899 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:41.900 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:41.900 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:07:41.900 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:41.900 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:41.901 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:41.901 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:41.901 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:07:41.901 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:41.902 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:41.902 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764501.249520_420.1", 2022-09-09T23:07:41.902 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:41.902 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:41.903 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:41.903 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:01:41.249520Z", 2022-09-09T23:07:41.903 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:07:41.903 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:41.904 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:41.904 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:07:41.904 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:41.904 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:07:41.905 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:07:41.905 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:07:41.905 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:41.905 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:07:41Z" 2022-09-09T23:07:41.905 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:07:41.907 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764501.249520_420.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:01:41.249520Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:07:41Z" } 2022-09-09T23:07:41.908 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764501.249520_420.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:01:41.249520Z', 'realm_epoch': 4}}] 2022-09-09T23:07:41.908 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:07:41.909 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:07:41.909 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764501.249520_420.1 2022-09-09T23:07:46.910 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:07:46.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:07:46.913 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:07:46.962 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:07:46.963 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:07:47.032 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:07:47.030+0000 7fc023fbfdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:07:47.035 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:07:47.034+0000 7fc023fbfdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:07:47.071 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:07:47.071 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:07:47.072 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:07:47.072 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:07:47.072 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:07:47.073 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:07:47.073 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:07:47.073 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:47.074 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:07:47.074 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:47.074 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:07:47.074 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:47.075 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:47.075 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:07:47.075 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:47.075 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:07:47.076 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:47.076 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:07:47.076 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:47.076 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:47.077 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:47.077 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:47.078 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:07:47.078 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:47.078 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:47.079 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:07:47.079 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:47.079 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:47.080 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:47.080 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:07:47.080 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:47.080 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:47.081 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:47.081 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:47.081 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:07:47.081 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:47.082 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:47.082 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:07:47.082 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:47.082 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:47.083 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:47.083 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:07:47.083 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:47.084 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:47.084 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:47.085 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:47.085 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:07:47.086 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:47.086 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:47.087 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764501.249520_420.1", 2022-09-09T23:07:47.087 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:47.088 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:47.089 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:47.089 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:01:41.249520Z", 2022-09-09T23:07:47.090 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:07:47.091 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:47.091 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:47.092 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:07:47.092 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:47.092 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:07:47.093 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:07:47.093 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:07:47.094 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:47.094 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:07:47Z" 2022-09-09T23:07:47.095 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:07:47.096 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764501.249520_420.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:01:41.249520Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:07:47Z" } 2022-09-09T23:07:47.097 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764501.249520_420.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:01:41.249520Z', 'realm_epoch': 4}}] 2022-09-09T23:07:47.098 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:07:47.098 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:07:47.099 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764501.249520_420.1 2022-09-09T23:07:52.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:07:52.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:07:52.103 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:07:52.150 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:07:52.151 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:07:52.222 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:07:52.222+0000 7f3972a6fdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:07:52.224 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:07:52.222+0000 7f3972a6fdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:07:52.254 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:07:52.254 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:07:52.255 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:07:52.255 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:07:52.256 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:07:52.256 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:07:52.257 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:07:52.257 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:52.258 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:07:52.258 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:52.259 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:07:52.259 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:52.259 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:52.260 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:07:52.260 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:52.261 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:07:52.261 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:52.262 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:07:52.262 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:52.262 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:52.263 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:52.263 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:52.264 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:07:52.264 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:52.265 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:52.265 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:07:52.265 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:52.266 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:52.266 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:52.266 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:07:52.266 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:52.267 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:52.267 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:52.267 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:52.267 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:07:52.268 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:52.268 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:52.268 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:07:52.269 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:52.269 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:52.270 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:52.270 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:07:52.270 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:52.271 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:52.271 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:52.272 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:52.272 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:07:52.272 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:52.272 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:52.273 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764501.249520_420.1", 2022-09-09T23:07:52.274 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:52.274 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:52.274 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:52.274 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:01:41.249520Z", 2022-09-09T23:07:52.275 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:07:52.275 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:52.276 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:52.276 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:07:52.276 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:52.277 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:07:52.278 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:07:52.278 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:07:52.278 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:52.279 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:07:52Z" 2022-09-09T23:07:52.279 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:07:52.281 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764501.249520_420.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:01:41.249520Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:07:52Z" } 2022-09-09T23:07:52.282 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764501.249520_420.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:01:41.249520Z', 'realm_epoch': 4}}] 2022-09-09T23:07:52.282 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:07:52.283 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764501.249520_420.1'} 2022-09-09T23:07:52.283 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764501.249520_420.1 2022-09-09T23:07:57.284 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:07:57.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:07:57.286 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:07:57.334 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:07:57.334 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:07:57.394 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:07:57.394+0000 7ff191a06dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:07:57.395 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:07:57.394+0000 7ff191a06dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:07:57.424 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:07:57.425 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:07:57.425 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:07:57.425 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:07:57.426 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:07:57.426 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:07:57.426 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:07:57.426 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:57.427 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:07:57.427 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:57.427 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:07:57.428 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:57.428 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:57.428 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:07:57.429 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:57.429 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:07:57.429 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:57.429 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:07:57.430 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:57.430 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:57.430 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:57.431 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:57.431 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:07:57.431 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:57.431 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:57.432 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:07:57.432 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:57.432 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:57.432 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:57.433 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:07:57.433 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:57.433 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:57.433 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:57.434 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:57.434 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:07:57.434 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:57.435 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:57.435 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:07:57.435 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:57.435 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:57.436 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:57.436 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:07:57.436 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:07:57.436 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:57.437 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:57.437 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:07:57.437 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:07:57.437 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:07:57.438 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:07:57.438 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764755.810272_608.1", 2022-09-09T23:07:57.438 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:07:57.439 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:07:57.439 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:07:57.439 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:05:55.810272Z", 2022-09-09T23:07:57.439 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:07:57.440 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:57.440 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:07:57.440 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:07:57.440 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:57.441 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:07:57.441 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:07:57.441 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:07:57.442 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:07:57.442 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:07:57Z" 2022-09-09T23:07:57.442 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:07:57.443 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764755.810272_608.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:05:55.810272Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:07:57Z" } 2022-09-09T23:07:57.444 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764755.810272_608.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:05:55.810272Z', 'realm_epoch': 4}}] 2022-09-09T23:07:57.445 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:07:57.445 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764755.810272_608.1'} 2022-09-09T23:07:57.446 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764755.810272_608.1 2022-09-09T23:08:02.447 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:08:02.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:08:02.449 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:08:02.504 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:08:02.505 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:08:02.576 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:08:02.574+0000 7fe138735dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:08:02.581 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:08:02.582+0000 7fe138735dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:08:02.652 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:08:02.653 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:08:02.653 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:08:02.654 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:08:02.654 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:08:02.654 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:08:02.655 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:08:02.655 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:02.655 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:08:02.655 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:02.656 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:08:02.656 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:02.656 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:02.656 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:08:02.657 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:02.657 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:08:02.657 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:02.657 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:08:02.658 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:02.658 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:02.658 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:02.659 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:02.659 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:08:02.659 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:02.659 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:02.660 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:08:02.660 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:02.660 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:02.660 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:02.660 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:08:02.661 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:02.661 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:02.661 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:02.661 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:02.662 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:08:02.662 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:02.662 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:02.662 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:08:02.663 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:02.663 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:02.663 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:02.664 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:08:02.664 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:02.664 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:02.664 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:02.665 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:02.666 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:08:02.666 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:02.666 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:02.666 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764755.810272_608.1", 2022-09-09T23:08:02.667 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:02.667 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:02.668 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:02.668 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:05:55.810272Z", 2022-09-09T23:08:02.668 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:08:02.668 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:02.669 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:02.669 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:08:02.669 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:02.670 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:08:02.670 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:08:02.670 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:08:02.671 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:02.671 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:08:02Z" 2022-09-09T23:08:02.671 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:08:02.673 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764755.810272_608.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:05:55.810272Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:08:02Z" } 2022-09-09T23:08:02.674 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764755.810272_608.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:05:55.810272Z', 'realm_epoch': 4}}] 2022-09-09T23:08:02.675 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:08:02.675 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764755.810272_608.1'} 2022-09-09T23:08:02.676 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764755.810272_608.1 2022-09-09T23:08:07.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:08:08.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:08:08.174 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:08:08.223 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:08:08.223 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:08:08.292 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:08:08.290+0000 7f6ad0eb7dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:08:08.294 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:08:08.294+0000 7f6ad0eb7dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:08:08.319 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:08:08.320 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:08:08.320 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:08:08.321 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:08:08.321 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:08:08.322 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:08:08.322 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:08:08.323 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:08.323 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:08:08.324 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:08.324 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:08:08.325 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:08.325 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:08.326 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:08:08.327 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:08.327 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:08:08.328 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:08.328 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:08:08.329 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:08.329 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:08.330 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:08.330 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:08.331 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:08:08.331 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:08.332 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:08.333 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:08:08.333 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:08.334 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:08.334 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:08.335 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:08:08.335 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:08.336 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:08.336 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:08.337 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:08.338 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:08:08.338 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:08.339 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:08.339 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:08:08.340 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:08.340 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:08.341 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:08.341 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:08:08.341 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:08.342 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:08.342 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:08.343 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:08.343 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:08:08.344 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:08.345 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:08.345 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764755.485706_586.1", 2022-09-09T23:08:08.346 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:08.347 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:08.347 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:08.348 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:05:55.485706Z", 2022-09-09T23:08:08.348 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:08:08.349 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:08.349 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:08.350 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:08:08.350 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:08.351 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:08:08.351 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:08:08.352 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:08:08.352 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:08.353 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:08:08Z" 2022-09-09T23:08:08.353 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:08:08.355 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764755.485706_586.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:05:55.485706Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:08:08Z" } 2022-09-09T23:08:08.356 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764755.485706_586.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:05:55.485706Z', 'realm_epoch': 4}}] 2022-09-09T23:08:08.357 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:08:08.358 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764755.485706_586.1'} 2022-09-09T23:08:08.359 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764755.485706_586.1 2022-09-09T23:08:13.361 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:08:13.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:08:13.364 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:08:13.412 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:08:13.413 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:08:13.516 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:08:13.514+0000 7ff086a50dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:08:13.518 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:08:13.518+0000 7ff086a50dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:08:13.545 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:08:13.546 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:08:13.546 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:08:13.547 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:08:13.547 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:08:13.548 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:08:13.548 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:08:13.549 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:13.549 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:08:13.549 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:13.550 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:08:13.550 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:13.551 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:13.551 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:08:13.552 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:13.552 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:08:13.552 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:13.553 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:08:13.553 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:13.553 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:13.553 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:13.554 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:13.554 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:08:13.556 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:13.556 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:13.557 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:08:13.557 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:13.558 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:13.559 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:13.559 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:08:13.560 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:13.560 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:13.561 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:13.561 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:13.562 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:08:13.562 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:13.563 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:13.563 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:08:13.564 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:13.564 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:13.565 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:13.565 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:08:13.566 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:13.566 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:13.567 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:13.567 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:13.568 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:08:13.568 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:13.569 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:13.569 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764755.485706_586.1", 2022-09-09T23:08:13.570 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:13.570 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:13.571 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:13.572 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:05:55.485706Z", 2022-09-09T23:08:13.572 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:08:13.573 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:13.573 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:13.574 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:08:13.574 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:13.575 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:08:13.575 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:08:13.576 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:08:13.576 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:13.577 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:08:13Z" 2022-09-09T23:08:13.577 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:08:13.579 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764755.485706_586.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:05:55.485706Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:08:13Z" } 2022-09-09T23:08:13.581 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764755.485706_586.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:05:55.485706Z', 'realm_epoch': 4}}] 2022-09-09T23:08:13.582 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:08:13.583 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764755.485706_586.1'} 2022-09-09T23:08:13.584 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764755.485706_586.1 2022-09-09T23:08:18.586 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:08:18.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:08:18.588 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:08:18.630 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:08:18.631 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:08:18.691 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:08:18.689+0000 7f655e6cfdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:08:18.692 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:08:18.693+0000 7f655e6cfdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:08:18.724 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:08:18.725 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:08:18.725 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:08:18.726 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:08:18.726 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:08:18.727 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:08:18.727 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:08:18.728 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:18.728 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:08:18.729 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:18.730 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:08:18.730 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:18.731 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:18.732 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:08:18.732 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:18.733 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:08:18.733 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:18.735 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:08:18.735 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:18.736 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:18.737 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:18.737 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:18.738 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:08:18.738 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:18.739 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:18.739 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:08:18.740 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:18.740 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:18.741 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:18.741 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:08:18.741 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:18.742 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:18.742 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:18.742 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:18.743 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:08:18.743 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:18.743 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:18.744 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:08:18.744 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:18.744 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:18.744 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:18.746 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:08:18.746 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:18.746 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:18.747 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:18.748 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:18.748 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:08:18.748 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:18.749 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:18.749 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764755.485706_586.1", 2022-09-09T23:08:18.749 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:18.749 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:18.750 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:18.750 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:05:55.485706Z", 2022-09-09T23:08:18.751 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:08:18.751 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:18.752 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:18.752 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:08:18.753 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:18.753 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:08:18.753 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:08:18.753 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:08:18.754 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:18.754 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:08:18Z" 2022-09-09T23:08:18.754 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:08:18.756 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764755.485706_586.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:05:55.485706Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:08:18Z" } 2022-09-09T23:08:18.756 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764755.485706_586.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:05:55.485706Z', 'realm_epoch': 4}}] 2022-09-09T23:08:18.757 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:08:18.757 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764755.485706_586.1'} 2022-09-09T23:08:18.758 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764755.485706_586.1 2022-09-09T23:08:23.760 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:08:23.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:08:23.761 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:08:23.827 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:08:23.828 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:08:23.879 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:08:23.877+0000 7f3bd463ddc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:08:23.879 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:08:23.877+0000 7f3bd463ddc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:08:23.903 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:08:23.903 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:08:23.903 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:08:23.904 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:08:23.904 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:08:23.904 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:08:23.905 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:08:23.905 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:23.905 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:08:23.905 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:23.906 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:08:23.906 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:23.906 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:23.906 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:08:23.907 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:23.907 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:08:23.907 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:23.907 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:08:23.908 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:23.908 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:23.908 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:23.908 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:23.909 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:08:23.909 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:23.909 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:23.909 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:08:23.910 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:23.910 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:23.910 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:23.910 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:08:23.911 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:23.911 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:23.911 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:23.912 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:23.912 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:08:23.912 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:23.912 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:23.913 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:08:23.913 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:23.913 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:23.914 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:23.914 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:08:23.914 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:23.915 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:23.915 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:23.915 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:23.915 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:08:23.916 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:23.916 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:23.916 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764755.485706_586.1", 2022-09-09T23:08:23.916 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:23.917 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:23.917 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:23.917 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:05:55.485706Z", 2022-09-09T23:08:23.917 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:08:23.918 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:23.918 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:23.918 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:08:23.918 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:23.919 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:08:23.919 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:08:23.919 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:08:23.919 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:23.920 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:08:23Z" 2022-09-09T23:08:23.920 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:08:23.921 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764755.485706_586.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:05:55.485706Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:08:23Z" } 2022-09-09T23:08:23.922 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764755.485706_586.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:05:55.485706Z', 'realm_epoch': 4}}] 2022-09-09T23:08:23.923 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:08:23.923 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764755.485706_586.1'} 2022-09-09T23:08:23.924 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764755.485706_586.1 2022-09-09T23:08:28.925 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:08:28.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:08:28.927 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:08:28.989 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:08:28.990 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:08:29.074 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:08:29.073+0000 7f39dd2ecdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:08:29.075 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:08:29.073+0000 7f39dd2ecdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:08:29.140 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:08:29.141 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:08:29.141 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:08:29.141 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:08:29.141 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:08:29.142 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:08:29.142 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:08:29.142 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:29.142 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:08:29.143 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:29.143 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:08:29.143 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:29.143 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:29.144 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:08:29.144 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:29.144 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:08:29.145 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:29.145 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:08:29.145 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:29.145 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:29.146 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:29.146 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:29.146 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:08:29.146 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:29.147 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:29.147 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:08:29.147 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:29.147 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:29.148 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:29.148 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:08:29.148 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:29.148 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:29.150 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:29.150 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:29.150 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:08:29.150 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:29.151 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:29.151 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:08:29.151 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:29.151 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:29.152 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:29.152 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:08:29.152 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:29.153 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:29.153 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:29.153 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:29.153 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:08:29.154 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:29.154 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:29.154 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764755.571550_590.1", 2022-09-09T23:08:29.154 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:29.155 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:29.155 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:29.155 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:05:55.571550Z", 2022-09-09T23:08:29.155 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:08:29.156 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:29.156 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:29.156 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:08:29.156 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:29.157 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:08:29.157 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:08:29.157 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:08:29.157 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:29.158 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:08:29Z" 2022-09-09T23:08:29.158 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:08:29.159 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764755.571550_590.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:05:55.571550Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:08:29Z" } 2022-09-09T23:08:29.161 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764755.571550_590.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:05:55.571550Z', 'realm_epoch': 4}}] 2022-09-09T23:08:29.161 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:08:29.162 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764755.571550_590.1'} 2022-09-09T23:08:29.162 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764755.571550_590.1 2022-09-09T23:08:34.164 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:08:34.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:08:34.166 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:08:34.206 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:08:34.206 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:08:34.268 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:08:34.265+0000 7f255e66bdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:08:34.270 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:08:34.269+0000 7f255e66bdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:08:34.300 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:08:34.301 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:08:34.301 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:08:34.302 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:08:34.303 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:08:34.303 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:08:34.303 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:08:34.304 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:34.304 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:08:34.305 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:34.305 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:08:34.306 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:34.307 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:34.307 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:08:34.308 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:34.308 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:08:34.309 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:34.309 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:08:34.310 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:34.310 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:34.311 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:34.311 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:34.312 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:08:34.312 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:34.313 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:34.313 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:08:34.314 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:34.314 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:34.315 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:34.316 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:08:34.316 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:34.317 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:34.317 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:34.318 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:34.318 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:08:34.319 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:34.320 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:34.320 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:08:34.321 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:34.321 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:34.322 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:34.322 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:08:34.323 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:34.323 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:34.324 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:34.324 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:34.325 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:08:34.326 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:34.326 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:34.327 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764755.777609_606.1", 2022-09-09T23:08:34.327 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:34.328 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:34.328 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:34.329 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:05:55.777609Z", 2022-09-09T23:08:34.329 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:08:34.330 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:34.330 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:34.331 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:08:34.331 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:34.332 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:08:34.332 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:08:34.333 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:08:34.333 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:34.334 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:08:34Z" 2022-09-09T23:08:34.335 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:08:34.338 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764755.777609_606.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:05:55.777609Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:08:34Z" } 2022-09-09T23:08:34.339 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764755.777609_606.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:05:55.777609Z', 'realm_epoch': 4}}] 2022-09-09T23:08:34.340 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:08:34.341 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764755.777609_606.1'} 2022-09-09T23:08:34.341 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764755.777609_606.1 2022-09-09T23:08:39.344 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:08:39.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:08:39.346 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:08:39.391 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:08:39.392 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:08:39.452 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:08:39.449+0000 7f49a8740dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:08:39.453 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:08:39.453+0000 7f49a8740dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:08:39.477 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:08:39.477 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:08:39.477 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:08:39.478 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:08:39.478 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:08:39.478 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:08:39.479 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:08:39.479 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:39.479 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:08:39.479 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:39.480 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:08:39.480 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:39.480 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:39.480 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:08:39.481 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:39.481 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:08:39.482 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:39.482 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:08:39.482 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:39.483 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:39.483 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:39.484 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:39.484 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:08:39.484 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:39.484 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:39.485 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:08:39.485 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:39.485 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:39.485 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:39.486 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:08:39.486 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:39.486 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:39.486 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:39.487 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:39.487 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:08:39.487 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:39.488 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:39.488 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:08:39.488 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:39.488 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:39.489 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:39.489 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:08:39.489 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:39.489 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:39.490 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:39.490 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:39.490 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:08:39.490 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:39.491 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:39.491 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764755.491248_587.1", 2022-09-09T23:08:39.491 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:39.492 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:39.492 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:39.492 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:05:55.491248Z", 2022-09-09T23:08:39.493 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:08:39.493 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:39.493 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:39.494 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:08:39.494 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:39.494 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:08:39.494 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:08:39.495 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:08:39.495 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:39.495 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:08:39Z" 2022-09-09T23:08:39.495 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:08:39.497 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764755.491248_587.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:05:55.491248Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:08:39Z" } 2022-09-09T23:08:39.497 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764755.491248_587.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:05:55.491248Z', 'realm_epoch': 4}}] 2022-09-09T23:08:39.498 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:08:39.498 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764755.491248_587.1'} 2022-09-09T23:08:39.499 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764755.491248_587.1 2022-09-09T23:08:44.500 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:08:44.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:08:44.502 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:08:44.549 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:08:44.550 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:08:44.613 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:08:44.609+0000 7f0e6a414dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:08:44.614 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:08:44.609+0000 7f0e6a414dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:08:44.643 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:08:44.644 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:08:44.645 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:08:44.645 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:08:44.646 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:08:44.646 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:08:44.647 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:08:44.647 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:44.647 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:08:44.648 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:44.648 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:08:44.649 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:44.649 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:44.649 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:08:44.650 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:44.650 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:08:44.651 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:44.651 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:08:44.651 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:44.652 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:44.652 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:44.653 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:44.653 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:08:44.653 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:44.653 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:44.654 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:08:44.654 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:44.655 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:44.655 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:44.655 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:08:44.655 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:44.656 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:44.656 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:44.656 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:44.657 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:08:44.657 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:44.657 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:44.658 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:08:44.658 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:44.658 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:44.658 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:44.659 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:08:44.659 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:44.659 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:44.659 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:44.660 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:44.660 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:08:44.660 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:44.660 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:44.661 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764755.497379_589.1", 2022-09-09T23:08:44.661 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:44.661 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:44.661 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:44.662 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:05:55.497379Z", 2022-09-09T23:08:44.662 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:08:44.662 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:44.663 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:44.663 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:08:44.663 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:44.663 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:08:44.664 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:08:44.664 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:08:44.664 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:44.664 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:08:44Z" 2022-09-09T23:08:44.665 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:08:44.666 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764755.497379_589.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:05:55.497379Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:08:44Z" } 2022-09-09T23:08:44.667 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764755.497379_589.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:05:55.497379Z', 'realm_epoch': 4}}] 2022-09-09T23:08:44.668 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:08:44.669 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764755.497379_589.1'} 2022-09-09T23:08:44.669 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764755.497379_589.1 2022-09-09T23:08:49.673 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:08:49.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:08:49.675 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:08:49.744 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:08:49.745 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:08:49.796 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:08:49.793+0000 7f747ae59dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:08:49.797 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:08:49.797+0000 7f747ae59dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:08:49.827 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:08:49.828 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:08:49.829 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:08:49.829 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:08:49.831 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:08:49.832 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:08:49.832 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:08:49.833 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:49.833 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:08:49.834 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:49.834 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:08:49.835 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:49.835 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:49.836 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:08:49.836 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:49.837 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:08:49.837 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:49.838 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:08:49.838 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:49.839 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:49.839 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:49.840 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:49.840 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:08:49.842 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:49.842 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:49.843 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:08:49.844 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:49.844 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:49.845 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:49.846 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:08:49.846 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:49.847 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:49.847 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:49.848 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:49.848 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:08:49.849 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:49.849 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:49.849 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:08:49.850 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:49.850 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:49.851 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:49.851 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:08:49.852 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:49.852 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:49.853 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:49.853 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:49.854 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:08:49.855 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:49.855 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:49.856 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764754.608169_568.1", 2022-09-09T23:08:49.856 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:49.857 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:49.857 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:49.858 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:05:54.608169Z", 2022-09-09T23:08:49.858 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:08:49.859 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:49.859 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:49.860 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:08:49.860 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:49.861 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:08:49.861 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:08:49.862 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:08:49.863 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:49.863 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:08:49Z" 2022-09-09T23:08:49.864 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:08:49.866 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764754.608169_568.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:05:54.608169Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:08:49Z" } 2022-09-09T23:08:49.867 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764754.608169_568.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:05:54.608169Z', 'realm_epoch': 4}}] 2022-09-09T23:08:49.869 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:08:49.869 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764754.608169_568.1'} 2022-09-09T23:08:49.870 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764754.608169_568.1 2022-09-09T23:08:54.873 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:08:54.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:08:54.876 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:08:54.932 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:08:54.932 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:08:55.000 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:08:54.997+0000 7f0a9b188dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:08:55.002 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:08:55.001+0000 7f0a9b188dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:08:55.033 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:08:55.034 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:08:55.035 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:08:55.035 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:08:55.036 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:08:55.036 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:08:55.037 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:08:55.037 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:55.037 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:08:55.038 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:55.038 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:08:55.039 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:55.039 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:55.040 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:08:55.040 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:55.041 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:08:55.041 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:55.042 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:08:55.042 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:55.043 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:55.043 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:55.043 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:55.044 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:08:55.044 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:55.045 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:55.045 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:08:55.052 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:55.054 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:55.054 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:55.054 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:08:55.055 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:55.055 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:55.055 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:55.055 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:55.056 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:08:55.056 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:55.056 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:55.056 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:08:55.057 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:55.057 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:55.057 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:55.057 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:08:55.058 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:08:55.058 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:55.058 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:55.058 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:08:55.059 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:08:55.059 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:08:55.059 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:08:55.059 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764754.604834_567.1", 2022-09-09T23:08:55.060 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:08:55.060 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:08:55.060 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:08:55.061 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:05:54.604834Z", 2022-09-09T23:08:55.061 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:08:55.061 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:55.061 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:08:55.062 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:08:55.062 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:55.062 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:08:55.062 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:08:55.063 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:08:55.063 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:08:55.063 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:08:55Z" 2022-09-09T23:08:55.063 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:08:55.065 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764754.604834_567.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:05:54.604834Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:08:55Z" } 2022-09-09T23:08:55.065 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764754.604834_567.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:05:54.604834Z', 'realm_epoch': 4}}] 2022-09-09T23:08:55.066 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:08:55.066 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764754.604834_567.1'} 2022-09-09T23:08:55.067 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764754.604834_567.1 2022-09-09T23:09:00.069 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:00.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:00.070 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:09:00.118 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:09:00.119 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:09:00.179 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:09:00.177+0000 7f3d25be7dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:09:00.179 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:09:00.177+0000 7f3d25be7dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:09:00.256 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:09:00.257 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:09:00.258 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:09:00.258 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:09:00.259 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:09:00.259 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:09:00.260 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:09:00.260 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:00.261 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:09:00.261 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:00.261 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:09:00.262 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:00.262 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:00.262 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:09:00.262 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:00.263 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:09:00.263 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:00.263 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:09:00.263 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:00.264 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:00.264 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:00.264 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:00.265 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:09:00.265 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:00.265 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:00.265 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:09:00.266 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:00.266 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:00.266 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:00.267 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:09:00.267 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:00.267 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:00.268 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:00.268 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:00.268 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:09:00.269 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:00.269 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:00.269 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:09:00.269 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:00.270 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:00.270 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:00.270 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:09:00.271 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:00.271 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:00.271 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:00.272 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:00.272 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:09:00.272 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:00.273 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:00.273 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764754.604834_567.1", 2022-09-09T23:09:00.274 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:00.274 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:00.274 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:00.275 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:05:54.604834Z", 2022-09-09T23:09:00.275 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:09:00.276 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:00.276 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:00.277 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:09:00.277 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:00.278 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:09:00.278 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:09:00.279 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:09:00.279 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:00.280 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:09:00Z" 2022-09-09T23:09:00.280 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:09:00.282 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764754.604834_567.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:05:54.604834Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:09:00Z" } 2022-09-09T23:09:00.283 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764754.604834_567.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:05:54.604834Z', 'realm_epoch': 4}}] 2022-09-09T23:09:00.284 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:09:00.284 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764754.604834_567.1'} 2022-09-09T23:09:00.285 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764754.604834_567.1 2022-09-09T23:09:05.287 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:05.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:05.288 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:09:05.339 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:09:05.340 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:09:05.433 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:09:05.432+0000 7f0e503ccdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:09:05.438 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:09:05.436+0000 7f0e503ccdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:09:05.467 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:09:05.467 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:09:05.468 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:09:05.469 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:09:05.470 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:09:05.471 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:09:05.471 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:09:05.471 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:05.472 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:09:05.472 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:05.473 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:09:05.474 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:05.474 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:05.475 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:09:05.475 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:05.476 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:09:05.476 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:05.477 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:09:05.477 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:05.478 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:05.479 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:05.480 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:05.480 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:09:05.480 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:05.480 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:05.481 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:09:05.481 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:05.481 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:05.481 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:05.482 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:09:05.482 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:05.482 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:05.482 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:05.483 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:05.483 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:09:05.484 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:05.484 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:05.484 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:09:05.484 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:05.485 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:05.485 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:05.485 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:09:05.485 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:05.486 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:05.486 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:05.486 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:05.486 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:09:05.487 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:05.487 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:05.487 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764758.007747_621.1", 2022-09-09T23:09:05.487 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:05.488 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:05.488 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:05.488 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:05:58.007747Z", 2022-09-09T23:09:05.488 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:09:05.489 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:05.489 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:05.489 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:09:05.489 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:05.490 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:09:05.490 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:09:05.491 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:09:05.492 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:05.492 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:09:05Z" 2022-09-09T23:09:05.493 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:09:05.494 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764758.007747_621.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:05:58.007747Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:09:05Z" } 2022-09-09T23:09:05.495 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764758.007747_621.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:05:58.007747Z', 'realm_epoch': 4}}] 2022-09-09T23:09:05.496 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:09:05.496 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764758.007747_621.1'} 2022-09-09T23:09:05.497 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764758.007747_621.1 2022-09-09T23:09:10.498 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:10.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:10.501 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:09:10.554 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:09:10.555 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:09:10.605 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:09:10.604+0000 7f5935b05dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:09:10.607 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:09:10.604+0000 7f5935b05dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:09:10.639 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:09:10.639 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:09:10.639 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:09:10.640 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:09:10.640 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:09:10.640 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:09:10.641 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:09:10.641 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:10.641 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:09:10.641 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:10.642 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:09:10.644 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:10.645 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:10.646 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:09:10.646 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:10.647 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:09:10.647 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:10.648 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:09:10.648 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:10.649 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:10.649 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:10.650 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:10.650 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:09:10.651 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:10.651 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:10.652 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:09:10.652 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:10.653 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:10.653 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:10.653 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:09:10.653 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:10.654 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:10.654 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:10.654 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:10.654 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:09:10.655 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:10.655 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:10.655 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:09:10.655 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:10.656 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:10.656 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:10.656 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:09:10.656 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:10.657 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:10.657 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:10.657 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:10.657 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:09:10.658 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:10.658 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:10.658 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764754.608169_568.1", 2022-09-09T23:09:10.658 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:10.659 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:10.659 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:10.659 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:05:54.608169Z", 2022-09-09T23:09:10.659 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:09:10.659 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:10.660 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:10.660 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:09:10.660 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:10.660 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:09:10.661 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:09:10.661 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:09:10.661 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:10.661 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:09:10Z" 2022-09-09T23:09:10.662 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:09:10.662 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764754.608169_568.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:05:54.608169Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:09:10Z" } 2022-09-09T23:09:10.663 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764754.608169_568.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:05:54.608169Z', 'realm_epoch': 4}}] 2022-09-09T23:09:10.663 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:09:10.664 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764754.608169_568.1'} 2022-09-09T23:09:10.664 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764754.608169_568.1 2022-09-09T23:09:15.665 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:15.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:15.668 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:09:15.724 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:09:15.725 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:09:15.783 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:09:15.780+0000 7f35bced5dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:09:15.784 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:09:15.784+0000 7f35bced5dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:09:15.812 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:09:15.812 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:09:15.813 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:09:15.813 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:09:15.813 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:09:15.814 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:09:15.814 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:09:15.815 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:15.815 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:09:15.815 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:15.816 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:09:15.816 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:15.816 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:15.816 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:09:15.817 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:15.817 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:09:15.817 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:15.817 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:09:15.818 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:15.818 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:15.818 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:15.819 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:15.819 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:09:15.819 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:15.820 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:15.820 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:09:15.820 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:15.821 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:15.821 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:15.822 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:09:15.822 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:15.822 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:15.822 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:15.823 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:15.823 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:09:15.823 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:15.824 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:15.824 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:09:15.824 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:15.825 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:15.825 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:15.825 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:09:15.825 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:15.826 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:15.826 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:15.826 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:15.827 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:09:15.827 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:15.827 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:15.828 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764755.485706_586.1", 2022-09-09T23:09:15.828 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:15.828 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:15.828 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:15.829 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:05:55.485706Z", 2022-09-09T23:09:15.829 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:09:15.829 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:15.829 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:15.830 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:09:15.830 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:15.830 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:09:15.831 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:09:15.831 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:09:15.834 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:15.834 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:09:15Z" 2022-09-09T23:09:15.835 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:09:15.840 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764755.485706_586.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:05:55.485706Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:09:15Z" } 2022-09-09T23:09:15.841 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764755.485706_586.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:05:55.485706Z', 'realm_epoch': 4}}] 2022-09-09T23:09:15.843 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:09:15.844 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764755.485706_586.1'} 2022-09-09T23:09:15.844 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764755.485706_586.1 2022-09-09T23:09:20.847 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:20.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:20.849 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:09:20.897 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:09:20.898 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:09:20.950 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:09:20.948+0000 7fbc53d25dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:09:20.951 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:09:20.948+0000 7fbc53d25dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:09:20.979 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:09:20.979 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:09:20.979 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:09:20.980 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:09:20.980 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:09:20.980 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:09:20.981 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:09:20.981 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:20.981 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:09:20.981 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:20.982 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:09:20.982 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:20.982 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:20.983 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:09:20.983 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:20.984 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:09:20.984 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:20.984 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:09:20.985 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:20.985 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:20.985 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:20.986 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:20.986 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:09:20.987 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:20.987 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:20.988 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:09:20.988 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:20.989 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:20.989 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:20.990 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:09:20.991 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:20.991 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:20.992 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:20.992 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:20.993 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:09:20.993 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:20.993 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:20.994 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:09:20.994 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:20.994 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:20.995 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:20.995 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:09:20.996 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:20.996 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:20.997 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:20.997 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:20.998 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:09:20.998 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:20.999 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:20.999 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764755.810272_608.1", 2022-09-09T23:09:21.000 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:21.001 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:21.001 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:21.002 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:05:55.810272Z", 2022-09-09T23:09:21.002 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:09:21.003 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:21.003 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:21.004 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:09:21.004 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:21.005 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:09:21.005 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:09:21.005 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:09:21.006 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:21.006 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:09:20Z" 2022-09-09T23:09:21.007 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:09:21.009 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764755.810272_608.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:05:55.810272Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:09:20Z" } 2022-09-09T23:09:21.010 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764755.810272_608.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:05:55.810272Z', 'realm_epoch': 4}}] 2022-09-09T23:09:21.011 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:09:21.011 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764755.810272_608.1'} 2022-09-09T23:09:21.012 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764755.810272_608.1 2022-09-09T23:09:26.013 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:26.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:26.015 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:09:26.065 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:09:26.066 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:09:26.133 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:09:26.132+0000 7f4f7e299dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:09:26.135 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:09:26.132+0000 7f4f7e299dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:09:26.164 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:09:26.165 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:09:26.165 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:09:26.166 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:09:26.166 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:09:26.166 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:09:26.167 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:09:26.167 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:26.167 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:09:26.168 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:26.168 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:09:26.169 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:26.169 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:26.170 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:09:26.170 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:26.171 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:09:26.171 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:26.171 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:09:26.172 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:26.172 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:26.172 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:26.172 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:26.173 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:09:26.173 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:26.173 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:26.174 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:09:26.174 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:26.174 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:26.174 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:26.175 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:09:26.175 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:26.175 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:26.176 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:26.176 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:26.176 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:09:26.177 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:26.177 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:26.177 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:09:26.178 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:26.178 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:26.178 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:26.179 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:09:26.179 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:26.179 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:26.180 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:26.180 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:26.181 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:09:26.181 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:26.182 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:26.183 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764754.608169_568.1", 2022-09-09T23:09:26.183 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:26.183 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:26.184 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:26.184 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:05:54.608169Z", 2022-09-09T23:09:26.184 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:09:26.185 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:26.185 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:26.185 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:09:26.186 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:26.186 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:09:26.187 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:09:26.187 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:09:26.187 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:26.188 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:09:26Z" 2022-09-09T23:09:26.188 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:09:26.190 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764754.608169_568.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:05:54.608169Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:09:26Z" } 2022-09-09T23:09:26.191 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764754.608169_568.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:05:54.608169Z', 'realm_epoch': 4}}] 2022-09-09T23:09:26.192 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:09:26.192 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764754.608169_568.1'} 2022-09-09T23:09:26.193 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764754.608169_568.1 2022-09-09T23:09:31.195 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:31.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:31.197 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:09:31.249 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:09:31.249 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:09:31.321 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:09:31.320+0000 7fbc5f35cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:09:31.323 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:09:31.320+0000 7fbc5f35cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:09:31.354 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:09:31.355 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:09:31.355 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:09:31.356 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:09:31.356 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:09:31.357 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:09:31.357 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:09:31.357 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:31.358 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:09:31.358 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:31.358 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:09:31.359 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:31.359 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:31.359 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:09:31.360 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:31.360 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:09:31.361 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:31.361 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:09:31.362 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:31.362 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:31.363 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:31.363 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:31.364 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:09:31.364 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:31.364 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:31.364 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:09:31.365 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:31.365 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:31.365 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:31.365 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:09:31.366 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:31.366 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:31.366 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:31.367 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:31.367 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:09:31.368 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:31.368 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:31.368 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:09:31.369 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:31.369 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:31.370 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:31.370 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:09:31.371 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:31.371 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:31.372 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:31.372 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:31.373 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:09:31.373 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:31.374 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:31.374 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764755.583082_593.1", 2022-09-09T23:09:31.375 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:31.375 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:31.376 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:31.376 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:05:55.583082Z", 2022-09-09T23:09:31.377 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:09:31.377 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:31.378 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:31.378 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:09:31.379 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:31.379 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:09:31.380 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:09:31.380 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:09:31.381 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:31.381 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:09:31Z" 2022-09-09T23:09:31.382 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:09:31.384 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764755.583082_593.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:05:55.583082Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:09:31Z" } 2022-09-09T23:09:31.385 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764755.583082_593.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:05:55.583082Z', 'realm_epoch': 4}}] 2022-09-09T23:09:31.386 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:09:31.386 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764755.583082_593.1'} 2022-09-09T23:09:31.387 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764755.583082_593.1 2022-09-09T23:09:36.389 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:36.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:36.395 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:09:36.450 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:09:36.450 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:09:36.516 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:09:36.512+0000 7fbe67d3edc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:09:36.518 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:09:36.516+0000 7fbe67d3edc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:09:36.548 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:09:36.548 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:09:36.549 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:09:36.549 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:09:36.549 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:09:36.549 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:09:36.550 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:09:36.550 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:36.551 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:09:36.551 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:36.551 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:09:36.552 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:36.552 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:36.552 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:09:36.552 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:36.553 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:09:36.553 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:36.553 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:09:36.553 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:36.554 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:36.554 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:36.554 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:36.555 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:09:36.555 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:36.555 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:36.556 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:09:36.556 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:36.556 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:36.557 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:36.557 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:09:36.557 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:36.558 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:36.558 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:36.559 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:36.559 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:09:36.559 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:36.560 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:36.560 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:09:36.560 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:36.561 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:36.561 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:36.561 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:09:36.562 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:36.562 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:36.562 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:36.563 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:36.563 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:09:36.564 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:36.564 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:36.565 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764755.782138_607.1", 2022-09-09T23:09:36.565 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:36.566 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:36.567 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:36.567 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:05:55.782138Z", 2022-09-09T23:09:36.568 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:09:36.568 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:36.569 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:36.569 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:09:36.570 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:36.570 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:09:36.571 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:09:36.571 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:09:36.572 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:36.572 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:09:36Z" 2022-09-09T23:09:36.573 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:09:36.575 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764755.782138_607.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:05:55.782138Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:09:36Z" } 2022-09-09T23:09:36.576 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764755.782138_607.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:05:55.782138Z', 'realm_epoch': 4}}] 2022-09-09T23:09:36.576 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:09:36.577 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764755.782138_607.1'} 2022-09-09T23:09:36.578 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764755.782138_607.1 2022-09-09T23:09:41.579 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:41.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:41.581 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:09:41.630 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:09:41.631 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:09:41.692 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:09:41.688+0000 7fc12eb52dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:09:41.694 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:09:41.692+0000 7fc12eb52dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:09:41.734 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:09:41.735 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:09:41.735 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:09:41.735 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:09:41.736 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:09:41.736 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:09:41.737 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:09:41.737 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:41.738 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:09:41.738 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:41.738 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:09:41.739 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:41.739 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:41.740 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:09:41.740 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:41.740 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:09:41.741 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:41.742 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:09:41.742 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:41.742 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:41.743 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:41.743 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:41.744 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:09:41.744 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:41.744 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:41.745 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:09:41.745 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:41.746 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:41.746 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:41.747 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:09:41.747 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:41.747 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:41.748 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:41.748 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:41.749 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:09:41.749 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:41.750 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:41.750 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:09:41.750 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:41.751 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:41.751 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:41.752 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:09:41.752 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:41.753 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:41.753 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:41.753 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:41.753 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:09:41.754 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:41.754 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:41.754 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764755.576627_591.1", 2022-09-09T23:09:41.755 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:41.755 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:41.755 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:41.755 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:05:55.576627Z", 2022-09-09T23:09:41.756 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:09:41.756 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:41.756 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:41.756 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:09:41.756 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:41.757 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:09:41.757 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:09:41.757 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:09:41.757 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:41.758 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:09:41Z" 2022-09-09T23:09:41.758 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:09:41.759 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764755.576627_591.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:05:55.576627Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:09:41Z" } 2022-09-09T23:09:41.760 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764755.576627_591.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:05:55.576627Z', 'realm_epoch': 4}}] 2022-09-09T23:09:41.760 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:09:41.761 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764755.576627_591.1'} 2022-09-09T23:09:41.761 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764755.576627_591.1 2022-09-09T23:09:46.763 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:46.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:46.765 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:09:46.812 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:09:46.812 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:09:46.868 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:09:46.864+0000 7fa063546dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:09:46.870 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:09:46.868+0000 7fa063546dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:09:46.899 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:09:46.900 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:09:46.901 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:09:46.901 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:09:46.902 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:09:46.902 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:09:46.903 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:09:46.903 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:46.904 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:09:46.904 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:46.905 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:09:46.905 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:46.906 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:46.906 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:09:46.907 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:46.907 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:09:46.908 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:46.908 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:09:46.909 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:46.909 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:46.910 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:46.910 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:46.911 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:09:46.911 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:46.912 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:46.912 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:09:46.913 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:46.913 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:46.914 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:46.914 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:09:46.915 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:46.915 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:46.916 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:46.916 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:46.917 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:09:46.917 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:46.918 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:46.918 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:09:46.919 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:46.919 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:46.920 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:46.920 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:09:46.921 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:46.921 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:46.922 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:46.922 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:46.923 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:09:46.923 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:46.924 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:46.924 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764755.485706_586.1", 2022-09-09T23:09:46.925 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:46.925 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:46.926 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:46.926 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:05:55.485706Z", 2022-09-09T23:09:46.927 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:09:46.928 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:46.929 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:46.929 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:09:46.930 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:46.930 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:09:46.931 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:09:46.931 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:09:46.932 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:46.932 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:09:46Z" 2022-09-09T23:09:46.933 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:09:46.936 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764755.485706_586.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:05:55.485706Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:09:46Z" } 2022-09-09T23:09:46.937 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764755.485706_586.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:05:55.485706Z', 'realm_epoch': 4}}] 2022-09-09T23:09:46.938 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:09:46.939 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764755.485706_586.1'} 2022-09-09T23:09:46.940 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662764758.090037_627.1 target=1_1662764755.485706_586.1 2022-09-09T23:09:51.941 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:51.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:51.945 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:09:52.003 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:09:52.005 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:09:52.069 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:09:52.068+0000 7fd1c1d65dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:09:52.071 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:09:52.068+0000 7fd1c1d65dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:09:52.103 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:09:52.103 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:09:52.104 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:09:52.104 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:09:52.104 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:09:52.104 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:09:52.105 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:09:52.105 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:52.105 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:09:52.105 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:52.106 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:09:52.106 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:52.106 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:52.106 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:09:52.107 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:52.107 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:09:52.107 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:52.107 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:09:52.108 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:52.108 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:52.108 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:52.109 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:52.109 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:09:52.110 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:52.113 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:52.113 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:09:52.114 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:52.114 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:52.114 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:52.115 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:09:52.115 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:52.115 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:52.115 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:52.116 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:52.117 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:09:52.117 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:52.118 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:52.118 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:09:52.118 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:52.119 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:52.119 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:52.119 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:09:52.119 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:52.120 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:52.120 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:52.120 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:52.120 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:09:52.121 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:52.121 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:52.121 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764758.090037_627.1", 2022-09-09T23:09:52.122 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:52.122 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:52.122 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:52.123 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:05:58.090037Z", 2022-09-09T23:09:52.123 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:09:52.123 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:52.123 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:52.124 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:09:52.124 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:52.124 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:09:52.125 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:09:52.125 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:09:52.125 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:52.125 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:09:52Z" 2022-09-09T23:09:52.126 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:09:52.127 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764758.090037_627.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:05:58.090037Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:09:52Z" } 2022-09-09T23:09:52.128 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764758.090037_627.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:05:58.090037Z', 'realm_epoch': 4}}] 2022-09-09T23:09:52.129 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:09:52.129 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:09:52.130 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-09T23:09:52.130 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-09T23:09:52.131 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:52.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:52.132 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:09:52.188 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:09:52.189 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:09:52.239 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T23:09:52.236+0000 7f8d9b410dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:09:52.240 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T23:09:52.240+0000 7f8d9b410dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:09:52.266 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T23:09:52.267 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T23:09:52.267 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T23:09:52.268 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T23:09:52.268 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T23:09:52.269 INFO:teuthology.orchestra.run.smithi005.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:09:52.269 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 4 2022-09-09T23:09:52.269 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:09:52.269 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T23:09:52.270 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:09:52.270 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T23:09:52.270 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:09:52.271 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T23:09:52.271 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:09:52.271 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:09:52.271 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 1, 2022-09-09T23:09:52.272 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:09:52.272 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:09:52.272 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T23:09:52.272 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:09:52.273 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:09:52.273 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:09:52.273 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T23:09:52.273 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:09:52.274 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T23:09:52.274 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T23:09:52.274 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:09:52.275 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:09:52.275 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:09:52.276 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T23:09:52.276 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T23:09:52.277 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:09:52.277 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:09:52.278 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:09:52.278 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T23:09:52.279 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:09:52.279 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T23:09:52.280 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:09:52.281 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:09:52.281 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:09:52.281 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:09:52.281 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:09:52.282 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T23:09:52.282 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:09:52.282 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:09:52.283 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:09:52.283 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T23:09:52.283 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:09:52.283 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T23:09:52.284 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662764758.090037_627.1", 2022-09-09T23:09:52.284 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:09:52.284 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:09:52.284 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:09:52.285 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T23:05:58.090037Z", 2022-09-09T23:09:52.285 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 4 2022-09-09T23:09:52.285 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:09:52.285 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:09:52.286 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:09:52.286 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:09:52.286 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T23:09:52.286 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 1, 2022-09-09T23:09:52.287 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 1 2022-09-09T23:09:52.287 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:09:52.287 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T23:09:52Z" 2022-09-09T23:09:52.287 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T23:09:52.289 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764758.090037_627.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:05:58.090037Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:09:52Z" } 2022-09-09T23:09:52.290 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764758.090037_627.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:05:58.090037Z', 'realm_epoch': 4}}] 2022-09-09T23:09:52.291 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:09:52.291 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:09:52.292 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-09T23:09:52.293 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-09T23:09:52.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:52.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:52.295 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:09:52.347 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:09:52.347 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:09:52.412 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:09:52.408+0000 7f17cc851dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:09:52.413 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:09:52.412+0000 7f17cc851dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:09:52.441 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:09:52.442 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:09:52.442 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:09:52.443 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:09:52.443 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:09:52.443 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:09:52.444 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:09:52.444 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:52.444 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:09:52.445 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:52.445 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:09:52.445 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:52.446 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:52.446 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:09:52.446 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:52.446 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:09:52.447 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:52.447 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:09:52.447 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:52.447 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:52.448 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:52.448 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:52.448 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:09:52.448 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:52.449 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:52.450 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:09:52.450 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:52.450 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:52.451 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:52.451 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:09:52.451 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:52.451 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:52.452 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:52.452 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:52.452 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:09:52.452 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:52.453 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:52.453 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:09:52.453 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:52.453 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:52.454 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:52.454 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:09:52.454 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:09:52.455 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:52.455 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:52.455 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:52.456 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:09:52.456 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:52.456 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:09:52.456 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662764758.090037_627.1", 2022-09-09T23:09:52.457 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:52.457 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:52.457 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:52.458 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:05:58.090037Z", 2022-09-09T23:09:52.458 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:09:52.459 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:52.459 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:52.460 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:09:52.461 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:52.461 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:09:52.462 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:09:52.462 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:09:52.463 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:52.463 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:09:52Z" 2022-09-09T23:09:52.464 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:09:52.467 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662764758.090037_627.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:05:58.090037Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:09:52Z" } 2022-09-09T23:09:52.468 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662764758.090037_627.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:05:58.090037Z', 'realm_epoch': 4}}] 2022-09-09T23:09:52.469 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:09:52.470 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662764758.090037_627.1'} 2022-09-09T23:09:52.471 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-09T23:09:52.472 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'jkcgbj-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:09:52.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'jkcgbj-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:09:52.475 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket jkcgbj-56 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-09T23:09:52.525 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:09:52.526 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:09:52.665 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'jkcgbj-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:09:52.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'jkcgbj-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:09:52.667 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket jkcgbj-57 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-09T23:09:52.716 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:09:52.716 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:09:52.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'jkcgbj-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:09:52.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'jkcgbj-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-09T23:09:52.856 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket jkcgbj-58 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-09T23:09:52.915 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:09:52.915 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:09:53.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:53.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:53.068 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:09:53.154 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:09:53.155 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:09:53.274 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T23:09:53.275 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:5", 2022-09-09T23:09:53.275 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:6", 2022-09-09T23:09:53.276 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.29", 2022-09-09T23:09:53.276 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000078", 2022-09-09T23:09:53.277 INFO:teuthology.orchestra.run.smithi005.stdout: "meta.history", 2022-09-09T23:09:53.277 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:10", 2022-09-09T23:09:53.278 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:4", 2022-09-09T23:09:53.279 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:10", 2022-09-09T23:09:53.279 INFO:teuthology.orchestra.run.smithi005.stdout: "meta.log.fd4bc158-06fd-4840-8a27-33fabda67b99.3", 2022-09-09T23:09:53.280 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19", 2022-09-09T23:09:53.280 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:5", 2022-09-09T23:09:53.281 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.shard.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5.3", 2022-09-09T23:09:53.281 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:4", 2022-09-09T23:09:53.282 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:10", 2022-09-09T23:09:53.282 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:6", 2022-09-09T23:09:53.283 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.23", 2022-09-09T23:09:53.283 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000070", 2022-09-09T23:09:53.284 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:9", 2022-09-09T23:09:53.284 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:9", 2022-09-09T23:09:53.285 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000104", 2022-09-09T23:09:53.285 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:1", 2022-09-09T23:09:53.286 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:7", 2022-09-09T23:09:53.286 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:6", 2022-09-09T23:09:53.287 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:5", 2022-09-09T23:09:53.287 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000026", 2022-09-09T23:09:53.288 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:6", 2022-09-09T23:09:53.288 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:4", 2022-09-09T23:09:53.289 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:8", 2022-09-09T23:09:53.289 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:2", 2022-09-09T23:09:53.290 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:7", 2022-09-09T23:09:53.290 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:10", 2022-09-09T23:09:53.291 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:5", 2022-09-09T23:09:53.291 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:5", 2022-09-09T23:09:53.292 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:8", 2022-09-09T23:09:53.292 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.shard.cc1fc6ed-f06f-45bc-961e-adec328c8651.2", 2022-09-09T23:09:53.293 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:2", 2022-09-09T23:09:53.293 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:1", 2022-09-09T23:09:53.294 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.8", 2022-09-09T23:09:53.294 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:1", 2022-09-09T23:09:53.295 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000028", 2022-09-09T23:09:53.296 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:3", 2022-09-09T23:09:53.296 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:9", 2022-09-09T23:09:53.297 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:2", 2022-09-09T23:09:53.297 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:7", 2022-09-09T23:09:53.298 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:8", 2022-09-09T23:09:53.298 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:6", 2022-09-09T23:09:53.299 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:2", 2022-09-09T23:09:53.299 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:5", 2022-09-09T23:09:53.300 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.30", 2022-09-09T23:09:53.300 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:6", 2022-09-09T23:09:53.301 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:8", 2022-09-09T23:09:53.301 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:9", 2022-09-09T23:09:53.302 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:6", 2022-09-09T23:09:53.302 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:8", 2022-09-09T23:09:53.303 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000040", 2022-09-09T23:09:53.303 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:5", 2022-09-09T23:09:53.304 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:6", 2022-09-09T23:09:53.304 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.6", 2022-09-09T23:09:53.305 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24", 2022-09-09T23:09:53.305 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000015", 2022-09-09T23:09:53.306 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:5", 2022-09-09T23:09:53.306 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:7", 2022-09-09T23:09:53.307 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000069", 2022-09-09T23:09:53.307 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000095", 2022-09-09T23:09:53.308 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000003", 2022-09-09T23:09:53.309 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000047", 2022-09-09T23:09:53.310 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:4", 2022-09-09T23:09:53.310 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:4", 2022-09-09T23:09:53.310 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:4", 2022-09-09T23:09:53.311 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:3", 2022-09-09T23:09:53.311 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:3", 2022-09-09T23:09:53.311 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:5", 2022-09-09T23:09:53.312 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:1", 2022-09-09T23:09:53.312 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:2", 2022-09-09T23:09:53.312 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000035", 2022-09-09T23:09:53.312 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:3", 2022-09-09T23:09:53.313 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000037", 2022-09-09T23:09:53.313 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:4", 2022-09-09T23:09:53.313 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:7", 2022-09-09T23:09:53.313 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:7", 2022-09-09T23:09:53.314 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:1", 2022-09-09T23:09:53.314 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:7", 2022-09-09T23:09:53.314 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:8", 2022-09-09T23:09:53.315 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:1", 2022-09-09T23:09:53.315 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000024", 2022-09-09T23:09:53.315 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:8", 2022-09-09T23:09:53.315 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:2", 2022-09-09T23:09:53.316 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:6", 2022-09-09T23:09:53.316 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:5", 2022-09-09T23:09:53.316 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:8", 2022-09-09T23:09:53.317 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000000", 2022-09-09T23:09:53.317 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000031", 2022-09-09T23:09:53.318 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:5", 2022-09-09T23:09:53.318 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:3", 2022-09-09T23:09:53.318 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:5", 2022-09-09T23:09:53.319 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:5", 2022-09-09T23:09:53.319 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36", 2022-09-09T23:09:53.321 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:4", 2022-09-09T23:09:53.321 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:3", 2022-09-09T23:09:53.322 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:2", 2022-09-09T23:09:53.322 INFO:teuthology.orchestra.run.smithi005.stdout: "data_log.3", 2022-09-09T23:09:53.322 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:8", 2022-09-09T23:09:53.322 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:5", 2022-09-09T23:09:53.323 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T23:09:53.323 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:2", 2022-09-09T23:09:53.323 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:7", 2022-09-09T23:09:53.324 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:2", 2022-09-09T23:09:53.324 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:0", 2022-09-09T23:09:53.325 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:4", 2022-09-09T23:09:53.325 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:5", 2022-09-09T23:09:53.325 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:1", 2022-09-09T23:09:53.326 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:1", 2022-09-09T23:09:53.326 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:1", 2022-09-09T23:09:53.326 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:8", 2022-09-09T23:09:53.326 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:5", 2022-09-09T23:09:53.327 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:1", 2022-09-09T23:09:53.327 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:4", 2022-09-09T23:09:53.327 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:5", 2022-09-09T23:09:53.327 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:9", 2022-09-09T23:09:53.328 INFO:teuthology.orchestra.run.smithi005.stdout: "data_loggenerations_metadata", 2022-09-09T23:09:53.328 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:10", 2022-09-09T23:09:53.329 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:4", 2022-09-09T23:09:53.329 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:10", 2022-09-09T23:09:53.329 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:2", 2022-09-09T23:09:53.329 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:1", 2022-09-09T23:09:53.330 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:1", 2022-09-09T23:09:53.330 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:6", 2022-09-09T23:09:53.330 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16", 2022-09-09T23:09:53.331 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000076", 2022-09-09T23:09:53.331 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000113", 2022-09-09T23:09:53.332 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000005", 2022-09-09T23:09:53.332 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:3", 2022-09-09T23:09:53.332 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000011", 2022-09-09T23:09:53.333 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000058", 2022-09-09T23:09:53.333 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000052", 2022-09-09T23:09:53.333 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:3", 2022-09-09T23:09:53.333 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:1", 2022-09-09T23:09:53.334 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.21", 2022-09-09T23:09:53.335 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:0", 2022-09-09T23:09:53.335 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5", 2022-09-09T23:09:53.335 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:5", 2022-09-09T23:09:53.336 INFO:teuthology.orchestra.run.smithi005.stdout: "data_log.0.xfyfp/PHd82.0", 2022-09-09T23:09:53.336 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:1", 2022-09-09T23:09:53.337 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.shard.cc1fc6ed-f06f-45bc-961e-adec328c8651.3", 2022-09-09T23:09:53.337 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:5", 2022-09-09T23:09:53.338 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:2", 2022-09-09T23:09:53.338 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:7", 2022-09-09T23:09:53.339 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:5", 2022-09-09T23:09:53.340 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:10", 2022-09-09T23:09:53.340 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:8", 2022-09-09T23:09:53.340 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23", 2022-09-09T23:09:53.341 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:1", 2022-09-09T23:09:53.341 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000088", 2022-09-09T23:09:53.342 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:8", 2022-09-09T23:09:53.342 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000080", 2022-09-09T23:09:53.343 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.shard.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5.1", 2022-09-09T23:09:53.343 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000090", 2022-09-09T23:09:53.343 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:2", 2022-09-09T23:09:53.343 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000110", 2022-09-09T23:09:53.344 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13", 2022-09-09T23:09:53.344 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:8", 2022-09-09T23:09:53.344 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.0", 2022-09-09T23:09:53.345 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:8", 2022-09-09T23:09:53.346 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:3", 2022-09-09T23:09:53.346 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:3", 2022-09-09T23:09:53.346 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22", 2022-09-09T23:09:53.347 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:3", 2022-09-09T23:09:53.347 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000096", 2022-09-09T23:09:53.347 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:10", 2022-09-09T23:09:53.348 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:8", 2022-09-09T23:09:53.348 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:10", 2022-09-09T23:09:53.348 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000087", 2022-09-09T23:09:53.348 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:4", 2022-09-09T23:09:53.349 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:6", 2022-09-09T23:09:53.349 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:0", 2022-09-09T23:09:53.349 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000008", 2022-09-09T23:09:53.349 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15", 2022-09-09T23:09:53.350 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:2", 2022-09-09T23:09:53.350 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:5", 2022-09-09T23:09:53.350 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:7", 2022-09-09T23:09:53.351 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:5", 2022-09-09T23:09:53.351 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:6", 2022-09-09T23:09:53.351 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:4", 2022-09-09T23:09:53.351 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:9", 2022-09-09T23:09:53.352 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:3", 2022-09-09T23:09:53.352 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000006", 2022-09-09T23:09:53.352 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000029", 2022-09-09T23:09:53.352 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:0", 2022-09-09T23:09:53.353 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12", 2022-09-09T23:09:53.353 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:9", 2022-09-09T23:09:53.353 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000089", 2022-09-09T23:09:53.353 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000022", 2022-09-09T23:09:53.354 INFO:teuthology.orchestra.run.smithi005.stdout: "data_log.3./QtcxUcwUfe.0", 2022-09-09T23:09:53.354 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3", 2022-09-09T23:09:53.354 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:6", 2022-09-09T23:09:53.354 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:10", 2022-09-09T23:09:53.355 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:3", 2022-09-09T23:09:53.355 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:6", 2022-09-09T23:09:53.355 INFO:teuthology.orchestra.run.smithi005.stdout: "bilog.trim", 2022-09-09T23:09:53.355 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:10", 2022-09-09T23:09:53.356 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:9", 2022-09-09T23:09:53.356 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000016", 2022-09-09T23:09:53.356 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:2", 2022-09-09T23:09:53.356 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:0", 2022-09-09T23:09:53.357 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:5", 2022-09-09T23:09:53.357 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:6", 2022-09-09T23:09:53.357 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:4", 2022-09-09T23:09:53.357 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:9", 2022-09-09T23:09:53.358 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:8", 2022-09-09T23:09:53.358 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000041", 2022-09-09T23:09:53.358 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:3", 2022-09-09T23:09:53.358 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:7", 2022-09-09T23:09:53.359 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:0", 2022-09-09T23:09:53.359 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.shard.cc1fc6ed-f06f-45bc-961e-adec328c8651.3.retry", 2022-09-09T23:09:53.359 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:9", 2022-09-09T23:09:53.359 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000018", 2022-09-09T23:09:53.360 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000092", 2022-09-09T23:09:53.360 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:2", 2022-09-09T23:09:53.360 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36", 2022-09-09T23:09:53.360 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:7", 2022-09-09T23:09:53.361 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:8", 2022-09-09T23:09:53.361 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:2", 2022-09-09T23:09:53.361 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:10", 2022-09-09T23:09:53.361 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.27", 2022-09-09T23:09:53.362 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:7", 2022-09-09T23:09:53.362 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:6", 2022-09-09T23:09:53.362 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.4", 2022-09-09T23:09:53.362 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:0", 2022-09-09T23:09:53.363 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:2", 2022-09-09T23:09:53.363 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:6", 2022-09-09T23:09:53.363 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20", 2022-09-09T23:09:53.364 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:5", 2022-09-09T23:09:53.364 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000014", 2022-09-09T23:09:53.364 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:1", 2022-09-09T23:09:53.364 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.shard.cc1fc6ed-f06f-45bc-961e-adec328c8651.2.retry", 2022-09-09T23:09:53.365 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:7", 2022-09-09T23:09:53.365 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:10", 2022-09-09T23:09:53.365 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:10", 2022-09-09T23:09:53.365 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000112", 2022-09-09T23:09:53.366 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000007", 2022-09-09T23:09:53.366 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000021", 2022-09-09T23:09:53.366 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.16", 2022-09-09T23:09:53.367 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:2", 2022-09-09T23:09:53.367 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:0", 2022-09-09T23:09:53.367 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:0", 2022-09-09T23:09:53.367 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000064", 2022-09-09T23:09:53.368 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:6", 2022-09-09T23:09:53.368 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:0", 2022-09-09T23:09:53.368 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000071", 2022-09-09T23:09:53.368 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:0", 2022-09-09T23:09:53.369 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:9", 2022-09-09T23:09:53.369 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000074", 2022-09-09T23:09:53.369 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:9", 2022-09-09T23:09:53.370 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:7", 2022-09-09T23:09:53.370 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:4", 2022-09-09T23:09:53.370 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26", 2022-09-09T23:09:53.370 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:9", 2022-09-09T23:09:53.371 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:1", 2022-09-09T23:09:53.371 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:9", 2022-09-09T23:09:53.371 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:1", 2022-09-09T23:09:53.372 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:1", 2022-09-09T23:09:53.372 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000081", 2022-09-09T23:09:53.372 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000009", 2022-09-09T23:09:53.372 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:6", 2022-09-09T23:09:53.373 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000121", 2022-09-09T23:09:53.373 INFO:teuthology.orchestra.run.smithi005.stdout: "data_log.1", 2022-09-09T23:09:53.373 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:0", 2022-09-09T23:09:53.373 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:10", 2022-09-09T23:09:53.374 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.17", 2022-09-09T23:09:53.374 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000125", 2022-09-09T23:09:53.374 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000082", 2022-09-09T23:09:53.374 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:8", 2022-09-09T23:09:53.375 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.shard.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5.0", 2022-09-09T23:09:53.375 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:10", 2022-09-09T23:09:53.375 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24", 2022-09-09T23:09:53.375 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:6", 2022-09-09T23:09:53.376 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:9", 2022-09-09T23:09:53.376 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:10", 2022-09-09T23:09:53.376 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:7", 2022-09-09T23:09:53.376 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:3", 2022-09-09T23:09:53.377 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:4", 2022-09-09T23:09:53.377 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000105", 2022-09-09T23:09:53.377 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000059", 2022-09-09T23:09:53.378 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:9", 2022-09-09T23:09:53.378 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:4", 2022-09-09T23:09:53.378 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:5", 2022-09-09T23:09:53.378 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17", 2022-09-09T23:09:53.379 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:8", 2022-09-09T23:09:53.379 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:2", 2022-09-09T23:09:53.379 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25", 2022-09-09T23:09:53.379 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:2", 2022-09-09T23:09:53.380 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:10", 2022-09-09T23:09:53.380 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000077", 2022-09-09T23:09:53.380 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:1", 2022-09-09T23:09:53.380 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:3", 2022-09-09T23:09:53.381 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:7", 2022-09-09T23:09:53.381 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000032", 2022-09-09T23:09:53.381 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17", 2022-09-09T23:09:53.382 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25", 2022-09-09T23:09:53.382 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:3", 2022-09-09T23:09:53.382 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:8", 2022-09-09T23:09:53.382 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:3", 2022-09-09T23:09:53.383 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:10", 2022-09-09T23:09:53.383 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.24", 2022-09-09T23:09:53.383 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:8", 2022-09-09T23:09:53.383 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:5", 2022-09-09T23:09:53.384 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:6", 2022-09-09T23:09:53.384 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1", 2022-09-09T23:09:53.384 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000053", 2022-09-09T23:09:53.384 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:0", 2022-09-09T23:09:53.385 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000091", 2022-09-09T23:09:53.385 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:9", 2022-09-09T23:09:53.385 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:9", 2022-09-09T23:09:53.385 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:10", 2022-09-09T23:09:53.386 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:7", 2022-09-09T23:09:53.386 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:10", 2022-09-09T23:09:53.386 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:5", 2022-09-09T23:09:53.386 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000065", 2022-09-09T23:09:53.387 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:5", 2022-09-09T23:09:53.387 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:2", 2022-09-09T23:09:53.387 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000083", 2022-09-09T23:09:53.387 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000010", 2022-09-09T23:09:53.388 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:5", 2022-09-09T23:09:53.388 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:7", 2022-09-09T23:09:53.388 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:7", 2022-09-09T23:09:53.389 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5", 2022-09-09T23:09:53.389 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.7", 2022-09-09T23:09:53.390 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:4", 2022-09-09T23:09:53.391 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.3", 2022-09-09T23:09:53.391 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:7", 2022-09-09T23:09:53.391 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:3", 2022-09-09T23:09:53.392 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:0", 2022-09-09T23:09:53.392 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:8", 2022-09-09T23:09:53.392 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:2", 2022-09-09T23:09:53.393 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:9", 2022-09-09T23:09:53.393 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10", 2022-09-09T23:09:53.393 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11", 2022-09-09T23:09:53.393 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:5", 2022-09-09T23:09:53.394 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:6", 2022-09-09T23:09:53.394 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:4", 2022-09-09T23:09:53.394 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:10", 2022-09-09T23:09:53.394 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:6", 2022-09-09T23:09:53.395 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13", 2022-09-09T23:09:53.395 INFO:teuthology.orchestra.run.smithi005.stdout: "mdlog.sync-status.shard.2", 2022-09-09T23:09:53.395 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:4", 2022-09-09T23:09:53.395 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:5", 2022-09-09T23:09:53.396 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.shard.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5.2.retry", 2022-09-09T23:09:53.396 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:6", 2022-09-09T23:09:53.396 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:8", 2022-09-09T23:09:53.396 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:10", 2022-09-09T23:09:53.397 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:7", 2022-09-09T23:09:53.397 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000045", 2022-09-09T23:09:53.397 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000002", 2022-09-09T23:09:53.398 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.12", 2022-09-09T23:09:53.398 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:10", 2022-09-09T23:09:53.398 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:5", 2022-09-09T23:09:53.398 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:9", 2022-09-09T23:09:53.399 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:9", 2022-09-09T23:09:53.400 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000116", 2022-09-09T23:09:53.400 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:10", 2022-09-09T23:09:53.400 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.25", 2022-09-09T23:09:53.401 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:8", 2022-09-09T23:09:53.401 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:2", 2022-09-09T23:09:53.401 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:9", 2022-09-09T23:09:53.401 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.shard.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5.3.retry", 2022-09-09T23:09:53.402 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:3", 2022-09-09T23:09:53.402 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000034", 2022-09-09T23:09:53.402 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:2", 2022-09-09T23:09:53.402 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:3", 2022-09-09T23:09:53.403 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000101", 2022-09-09T23:09:53.403 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:10", 2022-09-09T23:09:53.403 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.13", 2022-09-09T23:09:53.403 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:4", 2022-09-09T23:09:53.404 INFO:teuthology.orchestra.run.smithi005.stdout: "mdlog.sync-status.shard.1", 2022-09-09T23:09:53.404 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:6", 2022-09-09T23:09:53.404 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:8", 2022-09-09T23:09:53.405 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:1", 2022-09-09T23:09:53.405 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:5", 2022-09-09T23:09:53.405 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.shard.cc1fc6ed-f06f-45bc-961e-adec328c8651.0", 2022-09-09T23:09:53.406 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:8", 2022-09-09T23:09:53.406 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:5", 2022-09-09T23:09:53.407 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000079", 2022-09-09T23:09:53.407 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000049", 2022-09-09T23:09:53.407 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:1", 2022-09-09T23:09:53.407 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000117", 2022-09-09T23:09:53.408 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000044", 2022-09-09T23:09:53.408 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:3", 2022-09-09T23:09:53.408 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:10", 2022-09-09T23:09:53.409 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:9", 2022-09-09T23:09:53.409 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:0", 2022-09-09T23:09:53.409 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:10", 2022-09-09T23:09:53.409 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:10", 2022-09-09T23:09:53.410 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:9", 2022-09-09T23:09:53.410 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:2", 2022-09-09T23:09:53.410 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:9", 2022-09-09T23:09:53.411 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:0", 2022-09-09T23:09:53.411 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.14", 2022-09-09T23:09:53.411 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:6", 2022-09-09T23:09:53.411 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000066", 2022-09-09T23:09:53.412 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:0", 2022-09-09T23:09:53.412 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:10", 2022-09-09T23:09:53.412 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:7", 2022-09-09T23:09:53.412 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000068", 2022-09-09T23:09:53.413 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:8", 2022-09-09T23:09:53.413 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:7", 2022-09-09T23:09:53.413 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000085", 2022-09-09T23:09:53.413 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:8", 2022-09-09T23:09:53.414 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:4", 2022-09-09T23:09:53.414 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:8", 2022-09-09T23:09:53.414 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:5", 2022-09-09T23:09:53.414 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:7", 2022-09-09T23:09:53.415 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000073", 2022-09-09T23:09:53.415 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:3", 2022-09-09T23:09:53.415 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000038", 2022-09-09T23:09:53.415 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:10", 2022-09-09T23:09:53.416 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:8", 2022-09-09T23:09:53.416 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:7", 2022-09-09T23:09:53.416 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:2", 2022-09-09T23:09:53.416 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:0", 2022-09-09T23:09:53.417 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.10", 2022-09-09T23:09:53.417 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000118", 2022-09-09T23:09:53.417 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:6", 2022-09-09T23:09:53.417 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:8", 2022-09-09T23:09:53.418 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:10", 2022-09-09T23:09:53.418 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:7", 2022-09-09T23:09:53.418 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000036", 2022-09-09T23:09:53.418 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:0", 2022-09-09T23:09:53.419 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:9", 2022-09-09T23:09:53.419 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000103", 2022-09-09T23:09:53.419 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:4", 2022-09-09T23:09:53.419 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:2", 2022-09-09T23:09:53.419 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:7", 2022-09-09T23:09:53.420 INFO:teuthology.orchestra.run.smithi005.stdout: "mdlog.sync-status.shard.3", 2022-09-09T23:09:53.420 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:3", 2022-09-09T23:09:53.420 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:4", 2022-09-09T23:09:53.420 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000119", 2022-09-09T23:09:53.421 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:10", 2022-09-09T23:09:53.421 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000098", 2022-09-09T23:09:53.421 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:3", 2022-09-09T23:09:53.422 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:2", 2022-09-09T23:09:53.422 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:1", 2022-09-09T23:09:53.422 INFO:teuthology.orchestra.run.smithi005.stdout: "data_log.2.Y0wYvIF876u.0", 2022-09-09T23:09:53.422 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:0", 2022-09-09T23:09:53.423 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.shard.cc1fc6ed-f06f-45bc-961e-adec328c8651.1.retry", 2022-09-09T23:09:53.423 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:7", 2022-09-09T23:09:53.423 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:2", 2022-09-09T23:09:53.424 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:3", 2022-09-09T23:09:53.424 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19", 2022-09-09T23:09:53.424 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:6", 2022-09-09T23:09:53.424 INFO:teuthology.orchestra.run.smithi005.stdout: "mdlog.sync-status.shard.0", 2022-09-09T23:09:53.425 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:5", 2022-09-09T23:09:53.425 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:10", 2022-09-09T23:09:53.425 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:9", 2022-09-09T23:09:53.425 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000027", 2022-09-09T23:09:53.426 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:2", 2022-09-09T23:09:53.426 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:8", 2022-09-09T23:09:53.426 INFO:teuthology.orchestra.run.smithi005.stdout: "mdlog.sync-status", 2022-09-09T23:09:53.426 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000019", 2022-09-09T23:09:53.427 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:0", 2022-09-09T23:09:53.427 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:10", 2022-09-09T23:09:53.427 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:0", 2022-09-09T23:09:53.427 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:0", 2022-09-09T23:09:53.428 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:2", 2022-09-09T23:09:53.428 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000039", 2022-09-09T23:09:53.428 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:9", 2022-09-09T23:09:53.428 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:2", 2022-09-09T23:09:53.428 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:1", 2022-09-09T23:09:53.429 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:5", 2022-09-09T23:09:53.429 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:9", 2022-09-09T23:09:53.429 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:8", 2022-09-09T23:09:53.429 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14", 2022-09-09T23:09:53.430 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:4", 2022-09-09T23:09:53.430 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000100", 2022-09-09T23:09:53.430 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:4", 2022-09-09T23:09:53.430 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:7", 2022-09-09T23:09:53.431 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:0", 2022-09-09T23:09:53.431 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:7", 2022-09-09T23:09:53.431 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:5", 2022-09-09T23:09:53.432 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:8", 2022-09-09T23:09:53.432 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:3", 2022-09-09T23:09:53.432 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:1", 2022-09-09T23:09:53.432 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21", 2022-09-09T23:09:53.433 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:5", 2022-09-09T23:09:53.433 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:8", 2022-09-09T23:09:53.433 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:3", 2022-09-09T23:09:53.433 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000093", 2022-09-09T23:09:53.434 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:4", 2022-09-09T23:09:53.434 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:6", 2022-09-09T23:09:53.434 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:4", 2022-09-09T23:09:53.434 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:3", 2022-09-09T23:09:53.435 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000004", 2022-09-09T23:09:53.435 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000063", 2022-09-09T23:09:53.435 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:1", 2022-09-09T23:09:53.435 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000122", 2022-09-09T23:09:53.436 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:0", 2022-09-09T23:09:53.436 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:2", 2022-09-09T23:09:53.436 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000057", 2022-09-09T23:09:53.436 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:9", 2022-09-09T23:09:53.437 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:3", 2022-09-09T23:09:53.437 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16", 2022-09-09T23:09:53.437 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:9", 2022-09-09T23:09:53.437 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000054", 2022-09-09T23:09:53.438 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000114", 2022-09-09T23:09:53.438 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:7", 2022-09-09T23:09:53.438 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.2", 2022-09-09T23:09:53.438 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:4", 2022-09-09T23:09:53.439 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:0", 2022-09-09T23:09:53.439 INFO:teuthology.orchestra.run.smithi005.stdout: "data_log.0", 2022-09-09T23:09:53.439 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:6", 2022-09-09T23:09:53.439 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:3", 2022-09-09T23:09:53.440 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:0", 2022-09-09T23:09:53.440 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000012", 2022-09-09T23:09:53.441 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:0", 2022-09-09T23:09:53.441 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:5", 2022-09-09T23:09:53.442 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:6", 2022-09-09T23:09:53.442 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.shard.cc1fc6ed-f06f-45bc-961e-adec328c8651.1", 2022-09-09T23:09:53.442 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:0", 2022-09-09T23:09:53.443 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18", 2022-09-09T23:09:53.443 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000084", 2022-09-09T23:09:53.443 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000043", 2022-09-09T23:09:53.443 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:3", 2022-09-09T23:09:53.444 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:4", 2022-09-09T23:09:53.444 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:9", 2022-09-09T23:09:53.444 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:9", 2022-09-09T23:09:53.444 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:5", 2022-09-09T23:09:53.445 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.11", 2022-09-09T23:09:53.445 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:0", 2022-09-09T23:09:53.445 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:2", 2022-09-09T23:09:53.445 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:7", 2022-09-09T23:09:53.446 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22", 2022-09-09T23:09:53.446 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:0", 2022-09-09T23:09:53.446 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000111", 2022-09-09T23:09:53.447 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000048", 2022-09-09T23:09:53.447 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:6", 2022-09-09T23:09:53.447 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:3", 2022-09-09T23:09:53.448 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000020", 2022-09-09T23:09:53.448 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:5", 2022-09-09T23:09:53.448 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:4", 2022-09-09T23:09:53.448 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:5", 2022-09-09T23:09:53.449 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37", 2022-09-09T23:09:53.449 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:0", 2022-09-09T23:09:53.449 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:10", 2022-09-09T23:09:53.449 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:6", 2022-09-09T23:09:53.450 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:2", 2022-09-09T23:09:53.450 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:3", 2022-09-09T23:09:53.450 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000099", 2022-09-09T23:09:53.450 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000056", 2022-09-09T23:09:53.451 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:2", 2022-09-09T23:09:53.451 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.19", 2022-09-09T23:09:53.451 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:9", 2022-09-09T23:09:53.452 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:8", 2022-09-09T23:09:53.452 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:5", 2022-09-09T23:09:53.453 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:0", 2022-09-09T23:09:53.453 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:6", 2022-09-09T23:09:53.453 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.28", 2022-09-09T23:09:53.453 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:3", 2022-09-09T23:09:53.454 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:1", 2022-09-09T23:09:53.454 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:0", 2022-09-09T23:09:53.455 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:5", 2022-09-09T23:09:53.455 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:7", 2022-09-09T23:09:53.455 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000072", 2022-09-09T23:09:53.456 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:2", 2022-09-09T23:09:53.456 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:1", 2022-09-09T23:09:53.456 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:4", 2022-09-09T23:09:53.456 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000062", 2022-09-09T23:09:53.457 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:6", 2022-09-09T23:09:53.457 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:5", 2022-09-09T23:09:53.458 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000115", 2022-09-09T23:09:53.458 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.1", 2022-09-09T23:09:53.458 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:6", 2022-09-09T23:09:53.459 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000033", 2022-09-09T23:09:53.459 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:7", 2022-09-09T23:09:53.460 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:10", 2022-09-09T23:09:53.460 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:1", 2022-09-09T23:09:53.460 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15", 2022-09-09T23:09:53.461 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:1", 2022-09-09T23:09:53.461 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:9", 2022-09-09T23:09:53.461 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:4", 2022-09-09T23:09:53.462 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:4", 2022-09-09T23:09:53.462 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:1", 2022-09-09T23:09:53.462 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:3", 2022-09-09T23:09:53.463 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:4", 2022-09-09T23:09:53.463 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:7", 2022-09-09T23:09:53.463 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000025", 2022-09-09T23:09:53.464 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11", 2022-09-09T23:09:53.464 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000123", 2022-09-09T23:09:53.464 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:1", 2022-09-09T23:09:53.465 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:0", 2022-09-09T23:09:53.465 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.31", 2022-09-09T23:09:53.465 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:0", 2022-09-09T23:09:53.466 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:3", 2022-09-09T23:09:53.466 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:1", 2022-09-09T23:09:53.466 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.18", 2022-09-09T23:09:53.467 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:9", 2022-09-09T23:09:53.467 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:7", 2022-09-09T23:09:53.467 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:8", 2022-09-09T23:09:53.468 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:3", 2022-09-09T23:09:53.468 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:7", 2022-09-09T23:09:53.468 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000108", 2022-09-09T23:09:53.469 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:0", 2022-09-09T23:09:53.469 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000094", 2022-09-09T23:09:53.469 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5", 2022-09-09T23:09:53.469 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:8", 2022-09-09T23:09:53.470 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:4", 2022-09-09T23:09:53.470 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:10", 2022-09-09T23:09:53.470 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:7", 2022-09-09T23:09:53.470 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000060", 2022-09-09T23:09:53.471 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:10", 2022-09-09T23:09:53.471 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:7", 2022-09-09T23:09:53.471 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000109", 2022-09-09T23:09:53.471 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:1", 2022-09-09T23:09:53.472 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.15", 2022-09-09T23:09:53.472 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:3", 2022-09-09T23:09:53.472 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:3", 2022-09-09T23:09:53.472 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:2", 2022-09-09T23:09:53.473 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000013", 2022-09-09T23:09:53.473 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:10", 2022-09-09T23:09:53.474 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:6", 2022-09-09T23:09:53.474 INFO:teuthology.orchestra.run.smithi005.stdout: "data_log.1./4ZDFiUO0/V.0", 2022-09-09T23:09:53.474 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:1", 2022-09-09T23:09:53.474 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37", 2022-09-09T23:09:53.475 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:10", 2022-09-09T23:09:53.475 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:0", 2022-09-09T23:09:53.475 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:3", 2022-09-09T23:09:53.475 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:1", 2022-09-09T23:09:53.476 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000051", 2022-09-09T23:09:53.476 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:2", 2022-09-09T23:09:53.477 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:2", 2022-09-09T23:09:53.477 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:2", 2022-09-09T23:09:53.477 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000106", 2022-09-09T23:09:53.478 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:3", 2022-09-09T23:09:53.478 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:9", 2022-09-09T23:09:53.478 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:2", 2022-09-09T23:09:53.479 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:1", 2022-09-09T23:09:53.479 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:3", 2022-09-09T23:09:53.479 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.shard.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5.2", 2022-09-09T23:09:53.480 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:0", 2022-09-09T23:09:53.480 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:6", 2022-09-09T23:09:53.480 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000017", 2022-09-09T23:09:53.481 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:7", 2022-09-09T23:09:53.481 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:7", 2022-09-09T23:09:53.481 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:9", 2022-09-09T23:09:53.482 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:8", 2022-09-09T23:09:53.482 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:4", 2022-09-09T23:09:53.482 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:3", 2022-09-09T23:09:53.482 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:1", 2022-09-09T23:09:53.483 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:8", 2022-09-09T23:09:53.483 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:9", 2022-09-09T23:09:53.483 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000023", 2022-09-09T23:09:53.483 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:2", 2022-09-09T23:09:53.484 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:6", 2022-09-09T23:09:53.484 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:4", 2022-09-09T23:09:53.484 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:9", 2022-09-09T23:09:53.485 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:10", 2022-09-09T23:09:53.485 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20", 2022-09-09T23:09:53.485 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000107", 2022-09-09T23:09:53.485 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:6", 2022-09-09T23:09:53.486 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000046", 2022-09-09T23:09:53.486 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:2", 2022-09-09T23:09:53.486 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26", 2022-09-09T23:09:53.487 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:9", 2022-09-09T23:09:53.487 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.26", 2022-09-09T23:09:53.487 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:9", 2022-09-09T23:09:53.487 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:0", 2022-09-09T23:09:53.488 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:1", 2022-09-09T23:09:53.488 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:7", 2022-09-09T23:09:53.488 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:6", 2022-09-09T23:09:53.489 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.22", 2022-09-09T23:09:53.489 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:2", 2022-09-09T23:09:53.490 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:1", 2022-09-09T23:09:53.490 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:6", 2022-09-09T23:09:53.490 INFO:teuthology.orchestra.run.smithi005.stdout: "data_log.2", 2022-09-09T23:09:53.490 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:4", 2022-09-09T23:09:53.491 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:9", 2022-09-09T23:09:53.491 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:2", 2022-09-09T23:09:53.491 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000067", 2022-09-09T23:09:53.492 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1", 2022-09-09T23:09:53.492 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:1", 2022-09-09T23:09:53.492 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:7", 2022-09-09T23:09:53.492 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:4", 2022-09-09T23:09:53.493 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:0", 2022-09-09T23:09:53.493 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:4", 2022-09-09T23:09:53.493 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:4", 2022-09-09T23:09:53.493 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:10", 2022-09-09T23:09:53.494 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4", 2022-09-09T23:09:53.494 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:0", 2022-09-09T23:09:53.495 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.9", 2022-09-09T23:09:53.495 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000086", 2022-09-09T23:09:53.496 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:8", 2022-09-09T23:09:53.496 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:4", 2022-09-09T23:09:53.497 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.5", 2022-09-09T23:09:53.497 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:9", 2022-09-09T23:09:53.498 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:1", 2022-09-09T23:09:53.498 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:3", 2022-09-09T23:09:53.499 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:6", 2022-09-09T23:09:53.499 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14", 2022-09-09T23:09:53.500 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.shard.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5.1.retry", 2022-09-09T23:09:53.500 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:7", 2022-09-09T23:09:53.500 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:8", 2022-09-09T23:09:53.500 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:8", 2022-09-09T23:09:53.501 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:8", 2022-09-09T23:09:53.501 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.20", 2022-09-09T23:09:53.501 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:4", 2022-09-09T23:09:53.502 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:0", 2022-09-09T23:09:53.502 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:10", 2022-09-09T23:09:53.502 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12", 2022-09-09T23:09:53.502 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4", 2022-09-09T23:09:53.503 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:1", 2022-09-09T23:09:53.503 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:10", 2022-09-09T23:09:53.503 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:1", 2022-09-09T23:09:53.503 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:0", 2022-09-09T23:09:53.503 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:8", 2022-09-09T23:09:53.504 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:5", 2022-09-09T23:09:53.504 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:1", 2022-09-09T23:09:53.504 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:6", 2022-09-09T23:09:53.504 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000050", 2022-09-09T23:09:53.505 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3", 2022-09-09T23:09:53.505 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000102", 2022-09-09T23:09:53.505 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38", 2022-09-09T23:09:53.505 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:3", 2022-09-09T23:09:53.506 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:0", 2022-09-09T23:09:53.506 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:7", 2022-09-09T23:09:53.506 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:10", 2022-09-09T23:09:53.507 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23", 2022-09-09T23:09:53.507 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21", 2022-09-09T23:09:53.507 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10", 2022-09-09T23:09:53.507 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:8", 2022-09-09T23:09:53.507 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000124", 2022-09-09T23:09:53.508 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:7", 2022-09-09T23:09:53.508 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:1", 2022-09-09T23:09:53.508 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:6", 2022-09-09T23:09:53.509 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:8", 2022-09-09T23:09:53.509 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:6", 2022-09-09T23:09:53.509 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000001", 2022-09-09T23:09:53.509 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:1", 2022-09-09T23:09:53.510 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:3", 2022-09-09T23:09:53.510 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:7", 2022-09-09T23:09:53.510 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000126", 2022-09-09T23:09:53.510 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38", 2022-09-09T23:09:53.511 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000055", 2022-09-09T23:09:53.511 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:5", 2022-09-09T23:09:53.511 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000061", 2022-09-09T23:09:53.511 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:0", 2022-09-09T23:09:53.512 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:2", 2022-09-09T23:09:53.512 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:9", 2022-09-09T23:09:53.512 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:0", 2022-09-09T23:09:53.513 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:2", 2022-09-09T23:09:53.513 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000097", 2022-09-09T23:09:53.513 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:6", 2022-09-09T23:09:53.513 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:9", 2022-09-09T23:09:53.514 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:4", 2022-09-09T23:09:53.514 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:10", 2022-09-09T23:09:53.514 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:9", 2022-09-09T23:09:53.514 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000042", 2022-09-09T23:09:53.515 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18", 2022-09-09T23:09:53.515 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:4", 2022-09-09T23:09:53.515 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:6", 2022-09-09T23:09:53.515 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:3", 2022-09-09T23:09:53.516 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:4", 2022-09-09T23:09:53.516 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000075", 2022-09-09T23:09:53.516 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:0", 2022-09-09T23:09:53.516 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000120", 2022-09-09T23:09:53.517 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:9", 2022-09-09T23:09:53.517 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000030", 2022-09-09T23:09:53.517 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:10", 2022-09-09T23:09:53.517 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:1", 2022-09-09T23:09:53.518 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:4", 2022-09-09T23:09:53.518 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:8" 2022-09-09T23:09:53.518 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T23:09:53.518 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T23:09:53.520 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:53.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:53.521 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:09:53.566 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:09:53.566 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:09:53.685 INFO:teuthology.orchestra.run.smithi080.stdout:[ 2022-09-09T23:09:53.686 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22", 2022-09-09T23:09:53.686 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:6", 2022-09-09T23:09:53.687 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.29", 2022-09-09T23:09:53.687 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000078", 2022-09-09T23:09:53.687 INFO:teuthology.orchestra.run.smithi080.stdout: "meta.history", 2022-09-09T23:09:53.688 INFO:teuthology.orchestra.run.smithi080.stdout: "datalog.sync-status.shard.4b05608c-7197-4e9b-bd29-7989607fa277.1", 2022-09-09T23:09:53.688 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:10", 2022-09-09T23:09:53.689 INFO:teuthology.orchestra.run.smithi080.stdout: "meta.log.fd4bc158-06fd-4840-8a27-33fabda67b99.3", 2022-09-09T23:09:53.689 INFO:teuthology.orchestra.run.smithi080.stdout: "datalog.sync-status.shard.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5.3", 2022-09-09T23:09:53.690 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:6", 2022-09-09T23:09:53.690 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.23", 2022-09-09T23:09:53.691 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000070", 2022-09-09T23:09:53.691 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:9", 2022-09-09T23:09:53.692 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000104", 2022-09-09T23:09:53.692 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:7", 2022-09-09T23:09:53.693 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:5", 2022-09-09T23:09:53.693 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:0", 2022-09-09T23:09:53.694 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000026", 2022-09-09T23:09:53.694 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:6", 2022-09-09T23:09:53.695 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:4", 2022-09-09T23:09:53.695 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:6", 2022-09-09T23:09:53.695 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:8", 2022-09-09T23:09:53.696 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:2", 2022-09-09T23:09:53.697 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:7", 2022-09-09T23:09:53.697 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:10", 2022-09-09T23:09:53.698 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:9", 2022-09-09T23:09:53.698 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:10", 2022-09-09T23:09:53.698 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:8", 2022-09-09T23:09:53.698 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:2", 2022-09-09T23:09:53.699 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:1", 2022-09-09T23:09:53.699 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.8", 2022-09-09T23:09:53.699 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:3", 2022-09-09T23:09:53.699 INFO:teuthology.orchestra.run.smithi080.stdout: "datalog.sync-status.shard.4b05608c-7197-4e9b-bd29-7989607fa277.3.retry", 2022-09-09T23:09:53.700 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:1", 2022-09-09T23:09:53.700 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000028", 2022-09-09T23:09:53.700 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:9", 2022-09-09T23:09:53.700 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:8", 2022-09-09T23:09:53.701 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:4", 2022-09-09T23:09:53.701 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:5", 2022-09-09T23:09:53.701 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:6", 2022-09-09T23:09:53.701 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:3", 2022-09-09T23:09:53.702 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:6", 2022-09-09T23:09:53.702 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:3", 2022-09-09T23:09:53.702 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.30", 2022-09-09T23:09:53.702 INFO:teuthology.orchestra.run.smithi080.stdout: "data_log.2.r0bR2uCsobK.0", 2022-09-09T23:09:53.703 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:9", 2022-09-09T23:09:53.703 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:1", 2022-09-09T23:09:53.703 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:0", 2022-09-09T23:09:53.703 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:8", 2022-09-09T23:09:53.704 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000040", 2022-09-09T23:09:53.704 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:6", 2022-09-09T23:09:53.704 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.6", 2022-09-09T23:09:53.704 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:1", 2022-09-09T23:09:53.705 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:7", 2022-09-09T23:09:53.705 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24", 2022-09-09T23:09:53.705 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:10", 2022-09-09T23:09:53.705 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000015", 2022-09-09T23:09:53.706 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:5", 2022-09-09T23:09:53.706 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36", 2022-09-09T23:09:53.706 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000069", 2022-09-09T23:09:53.707 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000095", 2022-09-09T23:09:53.707 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:2", 2022-09-09T23:09:53.707 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:2", 2022-09-09T23:09:53.707 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000003", 2022-09-09T23:09:53.708 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000047", 2022-09-09T23:09:53.708 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:4", 2022-09-09T23:09:53.708 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:4", 2022-09-09T23:09:53.708 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:10", 2022-09-09T23:09:53.709 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:3", 2022-09-09T23:09:53.709 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:3", 2022-09-09T23:09:53.709 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:5", 2022-09-09T23:09:53.709 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:1", 2022-09-09T23:09:53.710 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000035", 2022-09-09T23:09:53.710 INFO:teuthology.orchestra.run.smithi080.stdout: "datalog.sync-status.shard.4b05608c-7197-4e9b-bd29-7989607fa277.0", 2022-09-09T23:09:53.710 INFO:teuthology.orchestra.run.smithi080.stdout: "datalog.sync-status.shard.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5.0.retry", 2022-09-09T23:09:53.710 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:3", 2022-09-09T23:09:53.711 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000037", 2022-09-09T23:09:53.711 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:2", 2022-09-09T23:09:53.711 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:8", 2022-09-09T23:09:53.711 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:6", 2022-09-09T23:09:53.712 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:4", 2022-09-09T23:09:53.712 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:8", 2022-09-09T23:09:53.712 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:7", 2022-09-09T23:09:53.712 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:8", 2022-09-09T23:09:53.713 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:1", 2022-09-09T23:09:53.713 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:2", 2022-09-09T23:09:53.713 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:5", 2022-09-09T23:09:53.713 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000024", 2022-09-09T23:09:53.714 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:5", 2022-09-09T23:09:53.714 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:2", 2022-09-09T23:09:53.714 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:6", 2022-09-09T23:09:53.714 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:7", 2022-09-09T23:09:53.715 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:8", 2022-09-09T23:09:53.715 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:0", 2022-09-09T23:09:53.715 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:8", 2022-09-09T23:09:53.715 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000000", 2022-09-09T23:09:53.716 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000031", 2022-09-09T23:09:53.716 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:5", 2022-09-09T23:09:53.716 INFO:teuthology.orchestra.run.smithi080.stdout: "datalog.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:09:53.717 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:5", 2022-09-09T23:09:53.717 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36", 2022-09-09T23:09:53.717 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:4", 2022-09-09T23:09:53.717 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:3", 2022-09-09T23:09:53.718 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:2", 2022-09-09T23:09:53.718 INFO:teuthology.orchestra.run.smithi080.stdout: "data_log.3", 2022-09-09T23:09:53.718 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:0", 2022-09-09T23:09:53.718 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:2", 2022-09-09T23:09:53.719 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:0", 2022-09-09T23:09:53.719 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:4", 2022-09-09T23:09:53.719 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:5", 2022-09-09T23:09:53.720 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:1", 2022-09-09T23:09:53.720 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:0", 2022-09-09T23:09:53.720 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:1", 2022-09-09T23:09:53.721 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:0", 2022-09-09T23:09:53.721 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:10", 2022-09-09T23:09:53.721 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:8", 2022-09-09T23:09:53.721 INFO:teuthology.orchestra.run.smithi080.stdout: "data_loggenerations_metadata", 2022-09-09T23:09:53.721 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:10", 2022-09-09T23:09:53.722 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000076", 2022-09-09T23:09:53.722 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000113", 2022-09-09T23:09:53.722 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000005", 2022-09-09T23:09:53.723 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:2", 2022-09-09T23:09:53.723 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000011", 2022-09-09T23:09:53.723 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000058", 2022-09-09T23:09:53.723 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:8", 2022-09-09T23:09:53.724 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000052", 2022-09-09T23:09:53.724 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:5", 2022-09-09T23:09:53.724 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:10", 2022-09-09T23:09:53.724 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:1", 2022-09-09T23:09:53.725 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.21", 2022-09-09T23:09:53.725 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:0", 2022-09-09T23:09:53.725 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:5", 2022-09-09T23:09:53.725 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:0", 2022-09-09T23:09:53.726 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:0", 2022-09-09T23:09:53.726 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:5", 2022-09-09T23:09:53.726 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:0", 2022-09-09T23:09:53.726 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:1", 2022-09-09T23:09:53.727 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:2", 2022-09-09T23:09:53.727 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:5", 2022-09-09T23:09:53.727 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:8", 2022-09-09T23:09:53.727 INFO:teuthology.orchestra.run.smithi080.stdout: "datalog.sync-status.shard.4b05608c-7197-4e9b-bd29-7989607fa277.2.retry", 2022-09-09T23:09:53.728 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000088", 2022-09-09T23:09:53.728 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:9", 2022-09-09T23:09:53.728 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:8", 2022-09-09T23:09:53.728 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000080", 2022-09-09T23:09:53.729 INFO:teuthology.orchestra.run.smithi080.stdout: "datalog.sync-status.shard.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5.1", 2022-09-09T23:09:53.729 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000090", 2022-09-09T23:09:53.729 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:2", 2022-09-09T23:09:53.729 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000110", 2022-09-09T23:09:53.730 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:6", 2022-09-09T23:09:53.730 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:8", 2022-09-09T23:09:53.730 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:5", 2022-09-09T23:09:53.730 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:3", 2022-09-09T23:09:53.731 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:3", 2022-09-09T23:09:53.731 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:2", 2022-09-09T23:09:53.731 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:1", 2022-09-09T23:09:53.731 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:6", 2022-09-09T23:09:53.732 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000096", 2022-09-09T23:09:53.732 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:5", 2022-09-09T23:09:53.732 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:7", 2022-09-09T23:09:53.732 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:10", 2022-09-09T23:09:53.733 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000087", 2022-09-09T23:09:53.733 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:6", 2022-09-09T23:09:53.733 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:0", 2022-09-09T23:09:53.733 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000008", 2022-09-09T23:09:53.734 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15", 2022-09-09T23:09:53.734 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:2", 2022-09-09T23:09:53.734 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:4", 2022-09-09T23:09:53.735 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:4", 2022-09-09T23:09:53.735 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000006", 2022-09-09T23:09:53.735 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000029", 2022-09-09T23:09:53.735 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:0", 2022-09-09T23:09:53.736 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:6", 2022-09-09T23:09:53.736 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:9", 2022-09-09T23:09:53.736 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000089", 2022-09-09T23:09:53.736 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:1", 2022-09-09T23:09:53.737 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000022", 2022-09-09T23:09:53.737 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3", 2022-09-09T23:09:53.737 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:10", 2022-09-09T23:09:53.737 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:6", 2022-09-09T23:09:53.738 INFO:teuthology.orchestra.run.smithi080.stdout: "bilog.trim", 2022-09-09T23:09:53.738 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000016", 2022-09-09T23:09:53.738 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:0", 2022-09-09T23:09:53.738 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:5", 2022-09-09T23:09:53.739 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:1", 2022-09-09T23:09:53.739 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:3", 2022-09-09T23:09:53.739 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:6", 2022-09-09T23:09:53.739 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:9", 2022-09-09T23:09:53.740 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:8", 2022-09-09T23:09:53.740 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000041", 2022-09-09T23:09:53.740 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:3", 2022-09-09T23:09:53.740 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:7", 2022-09-09T23:09:53.741 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:7", 2022-09-09T23:09:53.741 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:9", 2022-09-09T23:09:53.741 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000018", 2022-09-09T23:09:53.741 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000092", 2022-09-09T23:09:53.742 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:2", 2022-09-09T23:09:53.742 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:1", 2022-09-09T23:09:53.742 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:8", 2022-09-09T23:09:53.742 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:8", 2022-09-09T23:09:53.743 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:7", 2022-09-09T23:09:53.743 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:10", 2022-09-09T23:09:53.743 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:4", 2022-09-09T23:09:53.743 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.27", 2022-09-09T23:09:53.744 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:7", 2022-09-09T23:09:53.744 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:6", 2022-09-09T23:09:53.744 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.4", 2022-09-09T23:09:53.744 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24", 2022-09-09T23:09:53.745 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:9", 2022-09-09T23:09:53.745 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:5", 2022-09-09T23:09:53.745 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:4", 2022-09-09T23:09:53.745 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:3", 2022-09-09T23:09:53.746 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20", 2022-09-09T23:09:53.746 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:10", 2022-09-09T23:09:53.746 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:5", 2022-09-09T23:09:53.747 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000014", 2022-09-09T23:09:53.747 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11", 2022-09-09T23:09:53.747 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:1", 2022-09-09T23:09:53.747 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:6", 2022-09-09T23:09:53.748 INFO:teuthology.orchestra.run.smithi080.stdout: "datalog.sync-status.shard.4b05608c-7197-4e9b-bd29-7989607fa277.2", 2022-09-09T23:09:53.748 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:6", 2022-09-09T23:09:53.748 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:10", 2022-09-09T23:09:53.748 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:10", 2022-09-09T23:09:53.749 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:0", 2022-09-09T23:09:53.749 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000112", 2022-09-09T23:09:53.749 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:8", 2022-09-09T23:09:53.749 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000007", 2022-09-09T23:09:53.750 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000021", 2022-09-09T23:09:53.750 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.16", 2022-09-09T23:09:53.750 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:2", 2022-09-09T23:09:53.750 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:6", 2022-09-09T23:09:53.751 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:4", 2022-09-09T23:09:53.751 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:0", 2022-09-09T23:09:53.751 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000064", 2022-09-09T23:09:53.751 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:6", 2022-09-09T23:09:53.752 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000071", 2022-09-09T23:09:53.752 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:8", 2022-09-09T23:09:53.752 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:0", 2022-09-09T23:09:53.752 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16", 2022-09-09T23:09:53.753 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000074", 2022-09-09T23:09:53.753 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:6", 2022-09-09T23:09:53.753 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:9", 2022-09-09T23:09:53.753 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:7", 2022-09-09T23:09:53.754 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:5", 2022-09-09T23:09:53.754 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:8", 2022-09-09T23:09:53.754 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:3", 2022-09-09T23:09:53.754 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:1", 2022-09-09T23:09:53.755 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:4", 2022-09-09T23:09:53.755 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26", 2022-09-09T23:09:53.755 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:9", 2022-09-09T23:09:53.755 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:1", 2022-09-09T23:09:53.756 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:9", 2022-09-09T23:09:53.756 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:5", 2022-09-09T23:09:53.756 INFO:teuthology.orchestra.run.smithi080.stdout: "datalog.sync-status.shard.4b05608c-7197-4e9b-bd29-7989607fa277.3", 2022-09-09T23:09:53.757 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000081", 2022-09-09T23:09:53.757 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000009", 2022-09-09T23:09:53.757 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:10", 2022-09-09T23:09:53.757 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:8", 2022-09-09T23:09:53.758 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:6", 2022-09-09T23:09:53.758 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000121", 2022-09-09T23:09:53.758 INFO:teuthology.orchestra.run.smithi080.stdout: "data_log.1", 2022-09-09T23:09:53.758 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14", 2022-09-09T23:09:53.759 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:3", 2022-09-09T23:09:53.759 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:0", 2022-09-09T23:09:53.759 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:8", 2022-09-09T23:09:53.759 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.17", 2022-09-09T23:09:53.760 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23", 2022-09-09T23:09:53.760 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000125", 2022-09-09T23:09:53.760 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:4", 2022-09-09T23:09:53.760 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000082", 2022-09-09T23:09:53.761 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:8", 2022-09-09T23:09:53.761 INFO:teuthology.orchestra.run.smithi080.stdout: "datalog.sync-status.shard.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5.0", 2022-09-09T23:09:53.761 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:5", 2022-09-09T23:09:53.761 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:10", 2022-09-09T23:09:53.762 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:4", 2022-09-09T23:09:53.762 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000105", 2022-09-09T23:09:53.762 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000059", 2022-09-09T23:09:53.763 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:5", 2022-09-09T23:09:53.763 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:4", 2022-09-09T23:09:53.763 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:2", 2022-09-09T23:09:53.763 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:8", 2022-09-09T23:09:53.764 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:2", 2022-09-09T23:09:53.764 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:3", 2022-09-09T23:09:53.764 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:10", 2022-09-09T23:09:53.764 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000077", 2022-09-09T23:09:53.765 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:1", 2022-09-09T23:09:53.765 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:3", 2022-09-09T23:09:53.765 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000032", 2022-09-09T23:09:53.765 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:6", 2022-09-09T23:09:53.765 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17", 2022-09-09T23:09:53.766 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25", 2022-09-09T23:09:53.766 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:0", 2022-09-09T23:09:53.766 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:6", 2022-09-09T23:09:53.767 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:3", 2022-09-09T23:09:53.767 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:8", 2022-09-09T23:09:53.767 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:3", 2022-09-09T23:09:53.767 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:3", 2022-09-09T23:09:53.767 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:9", 2022-09-09T23:09:53.768 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:6", 2022-09-09T23:09:53.768 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:1", 2022-09-09T23:09:53.768 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:4", 2022-09-09T23:09:53.768 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.24", 2022-09-09T23:09:53.769 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:7", 2022-09-09T23:09:53.769 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:1", 2022-09-09T23:09:53.769 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:6", 2022-09-09T23:09:53.769 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:9", 2022-09-09T23:09:53.770 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:3", 2022-09-09T23:09:53.770 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1", 2022-09-09T23:09:53.770 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000053", 2022-09-09T23:09:53.771 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000091", 2022-09-09T23:09:53.771 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:9", 2022-09-09T23:09:53.771 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:4", 2022-09-09T23:09:53.771 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000065", 2022-09-09T23:09:53.771 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:5", 2022-09-09T23:09:53.772 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000083", 2022-09-09T23:09:53.772 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000010", 2022-09-09T23:09:53.772 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:7", 2022-09-09T23:09:53.772 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:7", 2022-09-09T23:09:53.773 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5", 2022-09-09T23:09:53.773 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:8", 2022-09-09T23:09:53.773 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.7", 2022-09-09T23:09:53.773 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.3", 2022-09-09T23:09:53.774 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:7", 2022-09-09T23:09:53.774 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:3", 2022-09-09T23:09:53.774 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:0", 2022-09-09T23:09:53.774 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:3", 2022-09-09T23:09:53.775 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:2", 2022-09-09T23:09:53.775 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:2", 2022-09-09T23:09:53.775 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10", 2022-09-09T23:09:53.775 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11", 2022-09-09T23:09:53.776 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:5", 2022-09-09T23:09:53.776 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:6", 2022-09-09T23:09:53.776 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:2", 2022-09-09T23:09:53.777 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:4", 2022-09-09T23:09:53.777 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26", 2022-09-09T23:09:53.777 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:6", 2022-09-09T23:09:53.777 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13", 2022-09-09T23:09:53.778 INFO:teuthology.orchestra.run.smithi080.stdout: "mdlog.sync-status.shard.2", 2022-09-09T23:09:53.778 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:5", 2022-09-09T23:09:53.778 INFO:teuthology.orchestra.run.smithi080.stdout: "datalog.sync-status.shard.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5.2.retry", 2022-09-09T23:09:53.778 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:6", 2022-09-09T23:09:53.779 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:4", 2022-09-09T23:09:53.779 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:8", 2022-09-09T23:09:53.779 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:8", 2022-09-09T23:09:53.779 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:6", 2022-09-09T23:09:53.780 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000045", 2022-09-09T23:09:53.780 INFO:teuthology.orchestra.run.smithi080.stdout: "datalog.sync-status.shard.4b05608c-7197-4e9b-bd29-7989607fa277.0.retry", 2022-09-09T23:09:53.780 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:4", 2022-09-09T23:09:53.780 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000002", 2022-09-09T23:09:53.781 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:0", 2022-09-09T23:09:53.781 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:0", 2022-09-09T23:09:53.781 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.12", 2022-09-09T23:09:53.781 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:4", 2022-09-09T23:09:53.782 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:9", 2022-09-09T23:09:53.782 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:5", 2022-09-09T23:09:53.782 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:9", 2022-09-09T23:09:53.782 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:0", 2022-09-09T23:09:53.783 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15", 2022-09-09T23:09:53.783 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000116", 2022-09-09T23:09:53.783 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1", 2022-09-09T23:09:53.783 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.25", 2022-09-09T23:09:53.784 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:3", 2022-09-09T23:09:53.784 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000034", 2022-09-09T23:09:53.784 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13", 2022-09-09T23:09:53.784 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:3", 2022-09-09T23:09:53.785 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000101", 2022-09-09T23:09:53.785 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:7", 2022-09-09T23:09:53.785 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.13", 2022-09-09T23:09:53.785 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21", 2022-09-09T23:09:53.786 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:2", 2022-09-09T23:09:53.786 INFO:teuthology.orchestra.run.smithi080.stdout: "mdlog.sync-status.shard.1", 2022-09-09T23:09:53.786 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:8", 2022-09-09T23:09:53.787 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:5", 2022-09-09T23:09:53.787 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000079", 2022-09-09T23:09:53.787 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000049", 2022-09-09T23:09:53.787 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37", 2022-09-09T23:09:53.788 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:1", 2022-09-09T23:09:53.788 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000117", 2022-09-09T23:09:53.788 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000044", 2022-09-09T23:09:53.788 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:3", 2022-09-09T23:09:53.789 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:10", 2022-09-09T23:09:53.789 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:9", 2022-09-09T23:09:53.789 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:0", 2022-09-09T23:09:53.789 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:10", 2022-09-09T23:09:53.790 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:10", 2022-09-09T23:09:53.790 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:7", 2022-09-09T23:09:53.790 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:10", 2022-09-09T23:09:53.790 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:9", 2022-09-09T23:09:53.791 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.14", 2022-09-09T23:09:53.791 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:6", 2022-09-09T23:09:53.791 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000066", 2022-09-09T23:09:53.791 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:7", 2022-09-09T23:09:53.792 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:9", 2022-09-09T23:09:53.792 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:9", 2022-09-09T23:09:53.792 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:10", 2022-09-09T23:09:53.792 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:5", 2022-09-09T23:09:53.793 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:9", 2022-09-09T23:09:53.793 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:6", 2022-09-09T23:09:53.793 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000068", 2022-09-09T23:09:53.793 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:8", 2022-09-09T23:09:53.794 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:2", 2022-09-09T23:09:53.794 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000085", 2022-09-09T23:09:53.794 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:3", 2022-09-09T23:09:53.794 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:9", 2022-09-09T23:09:53.795 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000073", 2022-09-09T23:09:53.795 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000038", 2022-09-09T23:09:53.795 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:10", 2022-09-09T23:09:53.795 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:3", 2022-09-09T23:09:53.796 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:8", 2022-09-09T23:09:53.796 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:7", 2022-09-09T23:09:53.796 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:0", 2022-09-09T23:09:53.796 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.10", 2022-09-09T23:09:53.797 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000118", 2022-09-09T23:09:53.797 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:8", 2022-09-09T23:09:53.797 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19", 2022-09-09T23:09:53.797 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000036", 2022-09-09T23:09:53.798 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:6", 2022-09-09T23:09:53.798 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:9", 2022-09-09T23:09:53.798 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000103", 2022-09-09T23:09:53.799 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:7", 2022-09-09T23:09:53.799 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:0", 2022-09-09T23:09:53.799 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:7", 2022-09-09T23:09:53.799 INFO:teuthology.orchestra.run.smithi080.stdout: "mdlog.sync-status.shard.3", 2022-09-09T23:09:53.800 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:4", 2022-09-09T23:09:53.800 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:2", 2022-09-09T23:09:53.800 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000119", 2022-09-09T23:09:53.800 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:9", 2022-09-09T23:09:53.801 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:9", 2022-09-09T23:09:53.801 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:10", 2022-09-09T23:09:53.801 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000098", 2022-09-09T23:09:53.801 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:3", 2022-09-09T23:09:53.802 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:9", 2022-09-09T23:09:53.802 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:1", 2022-09-09T23:09:53.802 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:0", 2022-09-09T23:09:53.802 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:2", 2022-09-09T23:09:53.803 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:2", 2022-09-09T23:09:53.803 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19", 2022-09-09T23:09:53.803 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:6", 2022-09-09T23:09:53.803 INFO:teuthology.orchestra.run.smithi080.stdout: "mdlog.sync-status.shard.0", 2022-09-09T23:09:53.804 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:6", 2022-09-09T23:09:53.804 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:10", 2022-09-09T23:09:53.804 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12", 2022-09-09T23:09:53.804 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:8", 2022-09-09T23:09:53.805 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000027", 2022-09-09T23:09:53.805 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:2", 2022-09-09T23:09:53.805 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:8", 2022-09-09T23:09:53.805 INFO:teuthology.orchestra.run.smithi080.stdout: "mdlog.sync-status", 2022-09-09T23:09:53.806 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000019", 2022-09-09T23:09:53.806 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:7", 2022-09-09T23:09:53.806 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:10", 2022-09-09T23:09:53.806 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:0", 2022-09-09T23:09:53.807 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000039", 2022-09-09T23:09:53.807 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:4", 2022-09-09T23:09:53.807 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:7", 2022-09-09T23:09:53.807 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:1", 2022-09-09T23:09:53.808 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:5", 2022-09-09T23:09:53.808 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:9", 2022-09-09T23:09:53.808 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:8", 2022-09-09T23:09:53.808 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:4", 2022-09-09T23:09:53.809 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:0", 2022-09-09T23:09:53.809 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:9", 2022-09-09T23:09:53.809 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000100", 2022-09-09T23:09:53.810 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:3", 2022-09-09T23:09:53.810 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:10", 2022-09-09T23:09:53.810 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:7", 2022-09-09T23:09:53.810 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:0", 2022-09-09T23:09:53.811 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:2", 2022-09-09T23:09:53.811 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:5", 2022-09-09T23:09:53.811 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:7", 2022-09-09T23:09:53.811 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:8", 2022-09-09T23:09:53.812 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21", 2022-09-09T23:09:53.812 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000093", 2022-09-09T23:09:53.812 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:7", 2022-09-09T23:09:53.812 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18", 2022-09-09T23:09:53.813 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:3", 2022-09-09T23:09:53.813 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000004", 2022-09-09T23:09:53.813 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000063", 2022-09-09T23:09:53.813 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:7", 2022-09-09T23:09:53.814 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:1", 2022-09-09T23:09:53.814 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000122", 2022-09-09T23:09:53.814 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:7", 2022-09-09T23:09:53.814 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000057", 2022-09-09T23:09:53.815 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:5", 2022-09-09T23:09:53.815 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16", 2022-09-09T23:09:53.815 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:2", 2022-09-09T23:09:53.815 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:9", 2022-09-09T23:09:53.816 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:10", 2022-09-09T23:09:53.816 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:7", 2022-09-09T23:09:53.816 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000054", 2022-09-09T23:09:53.816 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000114", 2022-09-09T23:09:53.817 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:0", 2022-09-09T23:09:53.817 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.2", 2022-09-09T23:09:53.817 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:4", 2022-09-09T23:09:53.817 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:0", 2022-09-09T23:09:53.818 INFO:teuthology.orchestra.run.smithi080.stdout: "data_log.0", 2022-09-09T23:09:53.818 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:6", 2022-09-09T23:09:53.818 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:7", 2022-09-09T23:09:53.819 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:1", 2022-09-09T23:09:53.819 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:10", 2022-09-09T23:09:53.819 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:5", 2022-09-09T23:09:53.819 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:8", 2022-09-09T23:09:53.820 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000012", 2022-09-09T23:09:53.820 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:9", 2022-09-09T23:09:53.820 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:6", 2022-09-09T23:09:53.820 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:5", 2022-09-09T23:09:53.821 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:6", 2022-09-09T23:09:53.821 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:0", 2022-09-09T23:09:53.821 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18", 2022-09-09T23:09:53.821 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000084", 2022-09-09T23:09:53.822 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000043", 2022-09-09T23:09:53.822 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:3", 2022-09-09T23:09:53.822 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:8", 2022-09-09T23:09:53.822 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38", 2022-09-09T23:09:53.823 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:5", 2022-09-09T23:09:53.823 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.11", 2022-09-09T23:09:53.823 INFO:teuthology.orchestra.run.smithi080.stdout: "datalog.sync-status.shard.4b05608c-7197-4e9b-bd29-7989607fa277.1.retry", 2022-09-09T23:09:53.823 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:2", 2022-09-09T23:09:53.824 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:7", 2022-09-09T23:09:53.824 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:7", 2022-09-09T23:09:53.824 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:10", 2022-09-09T23:09:53.824 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:3", 2022-09-09T23:09:53.825 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:10", 2022-09-09T23:09:53.825 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:9", 2022-09-09T23:09:53.825 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:5", 2022-09-09T23:09:53.825 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22", 2022-09-09T23:09:53.826 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:4", 2022-09-09T23:09:53.826 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:1", 2022-09-09T23:09:53.826 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:0", 2022-09-09T23:09:53.826 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000111", 2022-09-09T23:09:53.827 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000048", 2022-09-09T23:09:53.827 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:9", 2022-09-09T23:09:53.827 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:3", 2022-09-09T23:09:53.827 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:0", 2022-09-09T23:09:53.828 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000020", 2022-09-09T23:09:53.828 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:5", 2022-09-09T23:09:53.828 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:7", 2022-09-09T23:09:53.828 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:8", 2022-09-09T23:09:53.829 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:1", 2022-09-09T23:09:53.829 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:1", 2022-09-09T23:09:53.829 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:5", 2022-09-09T23:09:53.829 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:10", 2022-09-09T23:09:53.830 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:2", 2022-09-09T23:09:53.830 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:3", 2022-09-09T23:09:53.830 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000099", 2022-09-09T23:09:53.830 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000056", 2022-09-09T23:09:53.831 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:2", 2022-09-09T23:09:53.831 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:0", 2022-09-09T23:09:53.831 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:2", 2022-09-09T23:09:53.831 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.19", 2022-09-09T23:09:53.832 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:9", 2022-09-09T23:09:53.832 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:8", 2022-09-09T23:09:53.832 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:5", 2022-09-09T23:09:53.832 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:6", 2022-09-09T23:09:53.833 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:2", 2022-09-09T23:09:53.833 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.28", 2022-09-09T23:09:53.833 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:3", 2022-09-09T23:09:53.833 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:1", 2022-09-09T23:09:53.834 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:0", 2022-09-09T23:09:53.834 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:1", 2022-09-09T23:09:53.834 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000072", 2022-09-09T23:09:53.834 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:1", 2022-09-09T23:09:53.835 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:10", 2022-09-09T23:09:53.835 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10", 2022-09-09T23:09:53.835 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:4", 2022-09-09T23:09:53.835 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000062", 2022-09-09T23:09:53.836 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:9", 2022-09-09T23:09:53.836 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:6", 2022-09-09T23:09:53.836 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:1", 2022-09-09T23:09:53.836 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:5", 2022-09-09T23:09:53.837 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:10", 2022-09-09T23:09:53.837 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000115", 2022-09-09T23:09:53.837 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.1", 2022-09-09T23:09:53.837 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:1", 2022-09-09T23:09:53.838 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000033", 2022-09-09T23:09:53.838 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:7", 2022-09-09T23:09:53.838 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:4", 2022-09-09T23:09:53.838 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:7", 2022-09-09T23:09:53.839 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:9", 2022-09-09T23:09:53.839 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:9", 2022-09-09T23:09:53.839 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:4", 2022-09-09T23:09:53.839 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:5", 2022-09-09T23:09:53.840 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:8", 2022-09-09T23:09:53.840 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:0", 2022-09-09T23:09:53.840 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:4", 2022-09-09T23:09:53.840 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:1", 2022-09-09T23:09:53.841 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:3", 2022-09-09T23:09:53.841 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:4", 2022-09-09T23:09:53.841 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000025", 2022-09-09T23:09:53.841 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:10", 2022-09-09T23:09:53.842 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000123", 2022-09-09T23:09:53.842 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:0", 2022-09-09T23:09:53.842 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:0", 2022-09-09T23:09:53.842 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.31", 2022-09-09T23:09:53.843 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:6", 2022-09-09T23:09:53.843 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:2", 2022-09-09T23:09:53.843 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:1", 2022-09-09T23:09:53.843 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17", 2022-09-09T23:09:53.844 INFO:teuthology.orchestra.run.smithi080.stdout: "data_log.3.ahkJMzNFLGM.0", 2022-09-09T23:09:53.844 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.18", 2022-09-09T23:09:53.844 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:7", 2022-09-09T23:09:53.844 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:3", 2022-09-09T23:09:53.845 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000108", 2022-09-09T23:09:53.845 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:5", 2022-09-09T23:09:53.845 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:3", 2022-09-09T23:09:53.845 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:5", 2022-09-09T23:09:53.846 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000094", 2022-09-09T23:09:53.846 INFO:teuthology.orchestra.run.smithi080.stdout: "datalog.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5", 2022-09-09T23:09:53.846 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:8", 2022-09-09T23:09:53.847 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:10", 2022-09-09T23:09:53.847 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:7", 2022-09-09T23:09:53.847 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:5", 2022-09-09T23:09:53.847 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:4", 2022-09-09T23:09:53.848 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:7", 2022-09-09T23:09:53.848 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:2", 2022-09-09T23:09:53.848 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000060", 2022-09-09T23:09:53.848 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:4", 2022-09-09T23:09:53.849 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:10", 2022-09-09T23:09:53.849 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:7", 2022-09-09T23:09:53.849 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000109", 2022-09-09T23:09:53.849 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:1", 2022-09-09T23:09:53.850 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.15", 2022-09-09T23:09:53.850 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:8", 2022-09-09T23:09:53.850 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:3", 2022-09-09T23:09:53.850 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:2", 2022-09-09T23:09:53.851 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:0", 2022-09-09T23:09:53.851 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:10", 2022-09-09T23:09:53.851 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000013", 2022-09-09T23:09:53.851 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:10", 2022-09-09T23:09:53.852 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:3", 2022-09-09T23:09:53.852 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:3", 2022-09-09T23:09:53.852 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37", 2022-09-09T23:09:53.852 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:10", 2022-09-09T23:09:53.853 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:7", 2022-09-09T23:09:53.853 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4", 2022-09-09T23:09:53.853 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:4", 2022-09-09T23:09:53.853 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:3", 2022-09-09T23:09:53.854 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:10", 2022-09-09T23:09:53.854 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:5", 2022-09-09T23:09:53.854 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000051", 2022-09-09T23:09:53.854 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:2", 2022-09-09T23:09:53.855 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:2", 2022-09-09T23:09:53.855 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000106", 2022-09-09T23:09:53.855 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:3", 2022-09-09T23:09:53.855 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:9", 2022-09-09T23:09:53.856 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:2", 2022-09-09T23:09:53.856 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:2", 2022-09-09T23:09:53.856 INFO:teuthology.orchestra.run.smithi080.stdout: "data_log.0.WKp4KNuDy6a.0", 2022-09-09T23:09:53.856 INFO:teuthology.orchestra.run.smithi080.stdout: "datalog.sync-status.shard.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5.2", 2022-09-09T23:09:53.857 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:0", 2022-09-09T23:09:53.857 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:1", 2022-09-09T23:09:53.857 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000017", 2022-09-09T23:09:53.857 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:7", 2022-09-09T23:09:53.858 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:7", 2022-09-09T23:09:53.858 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:9", 2022-09-09T23:09:53.858 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:5", 2022-09-09T23:09:53.858 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:4", 2022-09-09T23:09:53.859 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:3", 2022-09-09T23:09:53.859 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000023", 2022-09-09T23:09:53.859 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:2", 2022-09-09T23:09:53.859 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:4", 2022-09-09T23:09:53.860 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:6", 2022-09-09T23:09:53.860 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:9", 2022-09-09T23:09:53.860 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:10", 2022-09-09T23:09:53.860 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000107", 2022-09-09T23:09:53.861 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000046", 2022-09-09T23:09:53.861 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:10", 2022-09-09T23:09:53.861 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:5", 2022-09-09T23:09:53.861 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.26", 2022-09-09T23:09:53.862 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:8", 2022-09-09T23:09:53.862 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:0", 2022-09-09T23:09:53.862 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:7", 2022-09-09T23:09:53.862 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.22", 2022-09-09T23:09:53.863 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:2", 2022-09-09T23:09:53.863 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:6", 2022-09-09T23:09:53.863 INFO:teuthology.orchestra.run.smithi080.stdout: "data_log.1.PHolLc1pPUA.0", 2022-09-09T23:09:53.863 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:1", 2022-09-09T23:09:53.864 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:2", 2022-09-09T23:09:53.864 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:1", 2022-09-09T23:09:53.864 INFO:teuthology.orchestra.run.smithi080.stdout: "data_log.2", 2022-09-09T23:09:53.864 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:4", 2022-09-09T23:09:53.865 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000067", 2022-09-09T23:09:53.865 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:3", 2022-09-09T23:09:53.865 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:1", 2022-09-09T23:09:53.865 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:4", 2022-09-09T23:09:53.866 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:0", 2022-09-09T23:09:53.866 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:4", 2022-09-09T23:09:53.866 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:10", 2022-09-09T23:09:53.866 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4", 2022-09-09T23:09:53.867 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:2", 2022-09-09T23:09:53.867 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.9", 2022-09-09T23:09:53.867 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000086", 2022-09-09T23:09:53.867 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:4", 2022-09-09T23:09:53.868 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.5", 2022-09-09T23:09:53.868 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25", 2022-09-09T23:09:53.868 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:9", 2022-09-09T23:09:53.868 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:3", 2022-09-09T23:09:53.869 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:2", 2022-09-09T23:09:53.869 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:6", 2022-09-09T23:09:53.869 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14", 2022-09-09T23:09:53.869 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:1", 2022-09-09T23:09:53.870 INFO:teuthology.orchestra.run.smithi080.stdout: "datalog.sync-status.shard.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5.1.retry", 2022-09-09T23:09:53.870 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:7", 2022-09-09T23:09:53.870 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:10", 2022-09-09T23:09:53.870 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:4", 2022-09-09T23:09:53.871 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:2", 2022-09-09T23:09:53.871 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:1", 2022-09-09T23:09:53.871 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:10", 2022-09-09T23:09:53.871 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:8", 2022-09-09T23:09:53.872 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.20", 2022-09-09T23:09:53.872 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:7", 2022-09-09T23:09:53.872 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12", 2022-09-09T23:09:53.872 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:9", 2022-09-09T23:09:53.873 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:1", 2022-09-09T23:09:53.873 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:10", 2022-09-09T23:09:53.873 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:1", 2022-09-09T23:09:53.873 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5", 2022-09-09T23:09:53.874 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:6", 2022-09-09T23:09:53.874 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:1", 2022-09-09T23:09:53.874 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:4", 2022-09-09T23:09:53.874 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:1", 2022-09-09T23:09:53.875 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000050", 2022-09-09T23:09:53.875 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:9", 2022-09-09T23:09:53.875 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000102", 2022-09-09T23:09:53.875 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:5", 2022-09-09T23:09:53.876 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38", 2022-09-09T23:09:53.876 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:0", 2022-09-09T23:09:53.877 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:7", 2022-09-09T23:09:53.877 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23", 2022-09-09T23:09:53.878 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:6", 2022-09-09T23:09:53.878 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:8", 2022-09-09T23:09:53.879 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000124", 2022-09-09T23:09:53.879 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:7", 2022-09-09T23:09:53.880 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:1", 2022-09-09T23:09:53.880 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:8", 2022-09-09T23:09:53.881 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000001", 2022-09-09T23:09:53.881 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:0", 2022-09-09T23:09:53.881 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000126", 2022-09-09T23:09:53.882 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:9", 2022-09-09T23:09:53.882 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000055", 2022-09-09T23:09:53.882 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:5", 2022-09-09T23:09:53.883 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000061", 2022-09-09T23:09:53.883 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:9", 2022-09-09T23:09:53.884 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:0", 2022-09-09T23:09:53.884 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:2", 2022-09-09T23:09:53.884 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20", 2022-09-09T23:09:53.885 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000097", 2022-09-09T23:09:53.885 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:6", 2022-09-09T23:09:53.886 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:9", 2022-09-09T23:09:53.886 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:7", 2022-09-09T23:09:53.886 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:9", 2022-09-09T23:09:53.887 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:3", 2022-09-09T23:09:53.887 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:6", 2022-09-09T23:09:53.887 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000042", 2022-09-09T23:09:53.887 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:10", 2022-09-09T23:09:53.887 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:4", 2022-09-09T23:09:53.888 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:8", 2022-09-09T23:09:53.888 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3", 2022-09-09T23:09:53.888 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:4", 2022-09-09T23:09:53.888 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:1", 2022-09-09T23:09:53.889 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000075", 2022-09-09T23:09:53.889 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:0", 2022-09-09T23:09:53.889 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000120", 2022-09-09T23:09:53.889 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:9", 2022-09-09T23:09:53.890 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000030", 2022-09-09T23:09:53.890 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:3", 2022-09-09T23:09:53.890 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.79c29b7c-0f9b-4fa5-b619-8f0b8cba5cf5:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:4", 2022-09-09T23:09:53.890 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:4", 2022-09-09T23:09:53.891 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:4" 2022-09-09T23:09:53.891 INFO:teuthology.orchestra.run.smithi080.stdout:] 2022-09-09T23:09:53.891 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T23:09:53.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:53.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:53.894 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:09:53.938 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:09:53.938 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:09:54.050 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T23:09:54.051 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:5", 2022-09-09T23:09:54.051 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22", 2022-09-09T23:09:54.051 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000078", 2022-09-09T23:09:54.052 INFO:teuthology.orchestra.run.smithi005.stdout: "meta.history", 2022-09-09T23:09:54.052 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:10", 2022-09-09T23:09:54.053 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.shard.4b05608c-7197-4e9b-bd29-7989607fa277.1", 2022-09-09T23:09:54.053 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:4", 2022-09-09T23:09:54.053 INFO:teuthology.orchestra.run.smithi005.stdout: "meta.log.fd4bc158-06fd-4840-8a27-33fabda67b99.3", 2022-09-09T23:09:54.054 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19", 2022-09-09T23:09:54.054 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:5", 2022-09-09T23:09:54.054 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:4", 2022-09-09T23:09:54.055 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:10", 2022-09-09T23:09:54.055 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000070", 2022-09-09T23:09:54.055 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:9", 2022-09-09T23:09:54.055 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000104", 2022-09-09T23:09:54.056 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:1", 2022-09-09T23:09:54.056 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:6", 2022-09-09T23:09:54.056 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:0", 2022-09-09T23:09:54.057 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000026", 2022-09-09T23:09:54.057 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:6", 2022-09-09T23:09:54.057 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:10", 2022-09-09T23:09:54.057 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:9", 2022-09-09T23:09:54.058 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:5", 2022-09-09T23:09:54.058 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:5", 2022-09-09T23:09:54.059 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.shard.cc1fc6ed-f06f-45bc-961e-adec328c8651.2", 2022-09-09T23:09:54.059 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.8", 2022-09-09T23:09:54.059 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:3", 2022-09-09T23:09:54.060 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.shard.4b05608c-7197-4e9b-bd29-7989607fa277.3.retry", 2022-09-09T23:09:54.060 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000028", 2022-09-09T23:09:54.060 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:3", 2022-09-09T23:09:54.061 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:2", 2022-09-09T23:09:54.061 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:7", 2022-09-09T23:09:54.061 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:4", 2022-09-09T23:09:54.061 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:5", 2022-09-09T23:09:54.062 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:3", 2022-09-09T23:09:54.062 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:6", 2022-09-09T23:09:54.062 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:2", 2022-09-09T23:09:54.063 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:5", 2022-09-09T23:09:54.063 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:3", 2022-09-09T23:09:54.063 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:6", 2022-09-09T23:09:54.064 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:8", 2022-09-09T23:09:54.064 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:1", 2022-09-09T23:09:54.064 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:6", 2022-09-09T23:09:54.064 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:0", 2022-09-09T23:09:54.065 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000040", 2022-09-09T23:09:54.065 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:5", 2022-09-09T23:09:54.065 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.6", 2022-09-09T23:09:54.066 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:1", 2022-09-09T23:09:54.066 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:7", 2022-09-09T23:09:54.067 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:10", 2022-09-09T23:09:54.067 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000015", 2022-09-09T23:09:54.067 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36", 2022-09-09T23:09:54.067 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:7", 2022-09-09T23:09:54.068 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000069", 2022-09-09T23:09:54.068 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000095", 2022-09-09T23:09:54.068 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:2", 2022-09-09T23:09:54.068 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:2", 2022-09-09T23:09:54.069 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000003", 2022-09-09T23:09:54.069 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000047", 2022-09-09T23:09:54.069 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:4", 2022-09-09T23:09:54.070 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:10", 2022-09-09T23:09:54.070 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:2", 2022-09-09T23:09:54.071 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000035", 2022-09-09T23:09:54.071 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.shard.4b05608c-7197-4e9b-bd29-7989607fa277.0", 2022-09-09T23:09:54.071 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000037", 2022-09-09T23:09:54.072 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:2", 2022-09-09T23:09:54.072 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:8", 2022-09-09T23:09:54.072 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:6", 2022-09-09T23:09:54.072 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:7", 2022-09-09T23:09:54.073 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:7", 2022-09-09T23:09:54.073 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:8", 2022-09-09T23:09:54.073 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:1", 2022-09-09T23:09:54.074 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:2", 2022-09-09T23:09:54.074 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:5", 2022-09-09T23:09:54.074 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000024", 2022-09-09T23:09:54.075 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:8", 2022-09-09T23:09:54.075 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:5", 2022-09-09T23:09:54.075 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:5", 2022-09-09T23:09:54.075 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:7", 2022-09-09T23:09:54.076 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:8", 2022-09-09T23:09:54.076 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:0", 2022-09-09T23:09:54.076 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000000", 2022-09-09T23:09:54.077 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000031", 2022-09-09T23:09:54.077 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:5", 2022-09-09T23:09:54.077 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:3", 2022-09-09T23:09:54.077 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:09:54.078 INFO:teuthology.orchestra.run.smithi005.stdout: "data_log.3", 2022-09-09T23:09:54.078 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:0", 2022-09-09T23:09:54.079 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:8", 2022-09-09T23:09:54.079 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:5", 2022-09-09T23:09:54.079 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651", 2022-09-09T23:09:54.080 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:2", 2022-09-09T23:09:54.080 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:7", 2022-09-09T23:09:54.080 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:1", 2022-09-09T23:09:54.081 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:0", 2022-09-09T23:09:54.081 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:8", 2022-09-09T23:09:54.082 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:5", 2022-09-09T23:09:54.082 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:1", 2022-09-09T23:09:54.082 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:4", 2022-09-09T23:09:54.082 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:5", 2022-09-09T23:09:54.083 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:0", 2022-09-09T23:09:54.083 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:10", 2022-09-09T23:09:54.083 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:9", 2022-09-09T23:09:54.084 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:8", 2022-09-09T23:09:54.084 INFO:teuthology.orchestra.run.smithi005.stdout: "data_loggenerations_metadata", 2022-09-09T23:09:54.084 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:10", 2022-09-09T23:09:54.085 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:4", 2022-09-09T23:09:54.085 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:2", 2022-09-09T23:09:54.085 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:1", 2022-09-09T23:09:54.085 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:1", 2022-09-09T23:09:54.086 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:6", 2022-09-09T23:09:54.086 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16", 2022-09-09T23:09:54.086 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000076", 2022-09-09T23:09:54.086 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000113", 2022-09-09T23:09:54.087 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000005", 2022-09-09T23:09:54.087 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:3", 2022-09-09T23:09:54.087 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:2", 2022-09-09T23:09:54.087 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000011", 2022-09-09T23:09:54.088 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000058", 2022-09-09T23:09:54.088 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:8", 2022-09-09T23:09:54.088 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000052", 2022-09-09T23:09:54.089 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:5", 2022-09-09T23:09:54.089 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:3", 2022-09-09T23:09:54.089 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:10", 2022-09-09T23:09:54.089 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:5", 2022-09-09T23:09:54.090 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:0", 2022-09-09T23:09:54.090 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5", 2022-09-09T23:09:54.090 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:0", 2022-09-09T23:09:54.091 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:0", 2022-09-09T23:09:54.091 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.shard.cc1fc6ed-f06f-45bc-961e-adec328c8651.3", 2022-09-09T23:09:54.091 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:5", 2022-09-09T23:09:54.092 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:7", 2022-09-09T23:09:54.092 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:10", 2022-09-09T23:09:54.093 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.shard.4b05608c-7197-4e9b-bd29-7989607fa277.2.retry", 2022-09-09T23:09:54.093 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23", 2022-09-09T23:09:54.093 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:1", 2022-09-09T23:09:54.093 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000088", 2022-09-09T23:09:54.094 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:9", 2022-09-09T23:09:54.094 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000080", 2022-09-09T23:09:54.094 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000090", 2022-09-09T23:09:54.094 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000110", 2022-09-09T23:09:54.095 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13", 2022-09-09T23:09:54.095 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:6", 2022-09-09T23:09:54.095 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:8", 2022-09-09T23:09:54.096 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:3", 2022-09-09T23:09:54.096 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:5", 2022-09-09T23:09:54.096 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22", 2022-09-09T23:09:54.096 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:2", 2022-09-09T23:09:54.097 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:1", 2022-09-09T23:09:54.097 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:6", 2022-09-09T23:09:54.097 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000096", 2022-09-09T23:09:54.098 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:10", 2022-09-09T23:09:54.098 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:5", 2022-09-09T23:09:54.098 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:7", 2022-09-09T23:09:54.099 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:8", 2022-09-09T23:09:54.099 INFO:teuthology.orchestra.run.smithi005.stdout: "data_log.0.JiHUkM9DIZD.0", 2022-09-09T23:09:54.099 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000087", 2022-09-09T23:09:54.100 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:4", 2022-09-09T23:09:54.100 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000008", 2022-09-09T23:09:54.100 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:4", 2022-09-09T23:09:54.101 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:5", 2022-09-09T23:09:54.101 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:7", 2022-09-09T23:09:54.101 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:5", 2022-09-09T23:09:54.101 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:6", 2022-09-09T23:09:54.102 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:9", 2022-09-09T23:09:54.102 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:3", 2022-09-09T23:09:54.102 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000006", 2022-09-09T23:09:54.103 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000029", 2022-09-09T23:09:54.103 INFO:teuthology.orchestra.run.smithi005.stdout: "data_log.2.zBIw1g0HhnY.0", 2022-09-09T23:09:54.103 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:6", 2022-09-09T23:09:54.104 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12", 2022-09-09T23:09:54.104 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000089", 2022-09-09T23:09:54.104 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:1", 2022-09-09T23:09:54.105 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000022", 2022-09-09T23:09:54.105 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:6", 2022-09-09T23:09:54.105 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:3", 2022-09-09T23:09:54.106 INFO:teuthology.orchestra.run.smithi005.stdout: "bilog.trim", 2022-09-09T23:09:54.106 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:10", 2022-09-09T23:09:54.106 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:9", 2022-09-09T23:09:54.107 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000016", 2022-09-09T23:09:54.107 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:2", 2022-09-09T23:09:54.107 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:1", 2022-09-09T23:09:54.108 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:3", 2022-09-09T23:09:54.108 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:4", 2022-09-09T23:09:54.108 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000041", 2022-09-09T23:09:54.108 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:0", 2022-09-09T23:09:54.109 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.shard.cc1fc6ed-f06f-45bc-961e-adec328c8651.3.retry", 2022-09-09T23:09:54.109 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:7", 2022-09-09T23:09:54.109 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000018", 2022-09-09T23:09:54.110 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000092", 2022-09-09T23:09:54.110 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:1", 2022-09-09T23:09:54.110 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:8", 2022-09-09T23:09:54.110 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:8", 2022-09-09T23:09:54.111 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36", 2022-09-09T23:09:54.111 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:8", 2022-09-09T23:09:54.111 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:2", 2022-09-09T23:09:54.112 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:4", 2022-09-09T23:09:54.112 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.4", 2022-09-09T23:09:54.112 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:0", 2022-09-09T23:09:54.112 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24", 2022-09-09T23:09:54.113 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:9", 2022-09-09T23:09:54.113 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:2", 2022-09-09T23:09:54.113 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:5", 2022-09-09T23:09:54.113 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:4", 2022-09-09T23:09:54.114 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:3", 2022-09-09T23:09:54.114 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:6", 2022-09-09T23:09:54.114 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:10", 2022-09-09T23:09:54.115 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000014", 2022-09-09T23:09:54.115 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11", 2022-09-09T23:09:54.116 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:6", 2022-09-09T23:09:54.116 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.shard.cc1fc6ed-f06f-45bc-961e-adec328c8651.2.retry", 2022-09-09T23:09:54.116 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.shard.4b05608c-7197-4e9b-bd29-7989607fa277.2", 2022-09-09T23:09:54.116 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:7", 2022-09-09T23:09:54.117 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:6", 2022-09-09T23:09:54.117 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:0", 2022-09-09T23:09:54.117 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000112", 2022-09-09T23:09:54.118 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:8", 2022-09-09T23:09:54.118 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000007", 2022-09-09T23:09:54.118 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000021", 2022-09-09T23:09:54.119 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.16", 2022-09-09T23:09:54.119 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:6", 2022-09-09T23:09:54.119 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:4", 2022-09-09T23:09:54.119 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:0", 2022-09-09T23:09:54.120 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000064", 2022-09-09T23:09:54.120 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:0", 2022-09-09T23:09:54.120 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000071", 2022-09-09T23:09:54.121 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:8", 2022-09-09T23:09:54.121 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:9", 2022-09-09T23:09:54.121 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16", 2022-09-09T23:09:54.121 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000074", 2022-09-09T23:09:54.122 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:6", 2022-09-09T23:09:54.122 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:5", 2022-09-09T23:09:54.122 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:8", 2022-09-09T23:09:54.122 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:3", 2022-09-09T23:09:54.123 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:1", 2022-09-09T23:09:54.123 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:9", 2022-09-09T23:09:54.123 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:1", 2022-09-09T23:09:54.123 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:1", 2022-09-09T23:09:54.124 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:9", 2022-09-09T23:09:54.124 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:5", 2022-09-09T23:09:54.124 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.shard.4b05608c-7197-4e9b-bd29-7989607fa277.3", 2022-09-09T23:09:54.124 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000081", 2022-09-09T23:09:54.125 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000009", 2022-09-09T23:09:54.125 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:10", 2022-09-09T23:09:54.125 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:8", 2022-09-09T23:09:54.126 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000121", 2022-09-09T23:09:54.126 INFO:teuthology.orchestra.run.smithi005.stdout: "data_log.1", 2022-09-09T23:09:54.126 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14", 2022-09-09T23:09:54.126 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:3", 2022-09-09T23:09:54.127 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:10", 2022-09-09T23:09:54.127 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:8", 2022-09-09T23:09:54.127 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23", 2022-09-09T23:09:54.128 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000125", 2022-09-09T23:09:54.128 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:4", 2022-09-09T23:09:54.128 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000082", 2022-09-09T23:09:54.128 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:5", 2022-09-09T23:09:54.129 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24", 2022-09-09T23:09:54.129 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:6", 2022-09-09T23:09:54.129 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:9", 2022-09-09T23:09:54.129 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:10", 2022-09-09T23:09:54.130 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:7", 2022-09-09T23:09:54.130 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:3", 2022-09-09T23:09:54.130 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000105", 2022-09-09T23:09:54.130 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000059", 2022-09-09T23:09:54.131 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:9", 2022-09-09T23:09:54.131 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:4", 2022-09-09T23:09:54.131 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:4", 2022-09-09T23:09:54.131 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17", 2022-09-09T23:09:54.132 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:8", 2022-09-09T23:09:54.132 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25", 2022-09-09T23:09:54.132 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:8", 2022-09-09T23:09:54.132 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:3", 2022-09-09T23:09:54.133 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000077", 2022-09-09T23:09:54.133 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:1", 2022-09-09T23:09:54.133 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:1", 2022-09-09T23:09:54.133 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:7", 2022-09-09T23:09:54.134 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000032", 2022-09-09T23:09:54.134 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:6", 2022-09-09T23:09:54.134 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:0", 2022-09-09T23:09:54.135 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:6", 2022-09-09T23:09:54.135 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:3", 2022-09-09T23:09:54.135 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:10", 2022-09-09T23:09:54.135 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:9", 2022-09-09T23:09:54.136 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:6", 2022-09-09T23:09:54.136 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:1", 2022-09-09T23:09:54.136 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:4", 2022-09-09T23:09:54.136 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:8", 2022-09-09T23:09:54.137 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:5", 2022-09-09T23:09:54.137 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:7", 2022-09-09T23:09:54.137 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:1", 2022-09-09T23:09:54.137 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:9", 2022-09-09T23:09:54.138 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:3", 2022-09-09T23:09:54.138 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000053", 2022-09-09T23:09:54.138 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:0", 2022-09-09T23:09:54.138 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000091", 2022-09-09T23:09:54.139 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:9", 2022-09-09T23:09:54.139 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:10", 2022-09-09T23:09:54.139 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:4", 2022-09-09T23:09:54.139 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:7", 2022-09-09T23:09:54.140 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:10", 2022-09-09T23:09:54.140 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:5", 2022-09-09T23:09:54.140 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000065", 2022-09-09T23:09:54.140 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:2", 2022-09-09T23:09:54.141 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000083", 2022-09-09T23:09:54.141 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000010", 2022-09-09T23:09:54.141 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:5", 2022-09-09T23:09:54.141 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:8", 2022-09-09T23:09:54.142 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.7", 2022-09-09T23:09:54.142 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:4", 2022-09-09T23:09:54.142 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.3", 2022-09-09T23:09:54.143 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:3", 2022-09-09T23:09:54.143 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:8", 2022-09-09T23:09:54.143 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:2", 2022-09-09T23:09:54.143 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:9", 2022-09-09T23:09:54.144 INFO:teuthology.orchestra.run.smithi005.stdout: "meta.log.503b1d61-4513-48c8-a308-17b76219610c.2", 2022-09-09T23:09:54.144 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:2", 2022-09-09T23:09:54.144 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:10", 2022-09-09T23:09:54.144 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26", 2022-09-09T23:09:54.145 INFO:teuthology.orchestra.run.smithi005.stdout: "mdlog.sync-status.shard.2", 2022-09-09T23:09:54.145 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:4", 2022-09-09T23:09:54.145 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:4", 2022-09-09T23:09:54.145 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:8", 2022-09-09T23:09:54.146 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:6", 2022-09-09T23:09:54.146 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:10", 2022-09-09T23:09:54.146 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:7", 2022-09-09T23:09:54.146 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000045", 2022-09-09T23:09:54.147 INFO:teuthology.orchestra.run.smithi005.stdout: "meta.log.2efdc3a1-023b-4e78-aaa2-0e069a034dac.3", 2022-09-09T23:09:54.147 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:4", 2022-09-09T23:09:54.147 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000002", 2022-09-09T23:09:54.147 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:0", 2022-09-09T23:09:54.148 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:0", 2022-09-09T23:09:54.148 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.12", 2022-09-09T23:09:54.148 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:10", 2022-09-09T23:09:54.148 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:4", 2022-09-09T23:09:54.149 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:5", 2022-09-09T23:09:54.149 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:5", 2022-09-09T23:09:54.149 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:0", 2022-09-09T23:09:54.150 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15", 2022-09-09T23:09:54.150 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000116", 2022-09-09T23:09:54.150 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1", 2022-09-09T23:09:54.150 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:10", 2022-09-09T23:09:54.151 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:8", 2022-09-09T23:09:54.151 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:2", 2022-09-09T23:09:54.151 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:9", 2022-09-09T23:09:54.151 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000034", 2022-09-09T23:09:54.152 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13", 2022-09-09T23:09:54.152 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:2", 2022-09-09T23:09:54.152 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000101", 2022-09-09T23:09:54.152 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:7", 2022-09-09T23:09:54.153 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:10", 2022-09-09T23:09:54.153 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.13", 2022-09-09T23:09:54.153 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21", 2022-09-09T23:09:54.154 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:2", 2022-09-09T23:09:54.154 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:4", 2022-09-09T23:09:54.154 INFO:teuthology.orchestra.run.smithi005.stdout: "mdlog.sync-status.shard.1", 2022-09-09T23:09:54.154 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:6", 2022-09-09T23:09:54.155 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:1", 2022-09-09T23:09:54.155 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:5", 2022-09-09T23:09:54.155 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.shard.cc1fc6ed-f06f-45bc-961e-adec328c8651.0", 2022-09-09T23:09:54.155 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:8", 2022-09-09T23:09:54.156 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000079", 2022-09-09T23:09:54.156 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000049", 2022-09-09T23:09:54.156 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37", 2022-09-09T23:09:54.156 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000117", 2022-09-09T23:09:54.157 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000044", 2022-09-09T23:09:54.157 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:9", 2022-09-09T23:09:54.157 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:7", 2022-09-09T23:09:54.157 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:2", 2022-09-09T23:09:54.158 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:10", 2022-09-09T23:09:54.158 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:0", 2022-09-09T23:09:54.158 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.14", 2022-09-09T23:09:54.159 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000066", 2022-09-09T23:09:54.159 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:7", 2022-09-09T23:09:54.159 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:9", 2022-09-09T23:09:54.159 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:9", 2022-09-09T23:09:54.160 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:0", 2022-09-09T23:09:54.160 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:5", 2022-09-09T23:09:54.160 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:9", 2022-09-09T23:09:54.161 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:6", 2022-09-09T23:09:54.161 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:7", 2022-09-09T23:09:54.161 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000068", 2022-09-09T23:09:54.161 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:2", 2022-09-09T23:09:54.162 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:7", 2022-09-09T23:09:54.162 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000085", 2022-09-09T23:09:54.162 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:8", 2022-09-09T23:09:54.162 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:3", 2022-09-09T23:09:54.163 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:9", 2022-09-09T23:09:54.163 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:4", 2022-09-09T23:09:54.163 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:8", 2022-09-09T23:09:54.163 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:5", 2022-09-09T23:09:54.164 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:7", 2022-09-09T23:09:54.164 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000073", 2022-09-09T23:09:54.164 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:3", 2022-09-09T23:09:54.165 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000038", 2022-09-09T23:09:54.165 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:3", 2022-09-09T23:09:54.165 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:2", 2022-09-09T23:09:54.165 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.10", 2022-09-09T23:09:54.166 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000118", 2022-09-09T23:09:54.166 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:6", 2022-09-09T23:09:54.166 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:10", 2022-09-09T23:09:54.166 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:7", 2022-09-09T23:09:54.167 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19", 2022-09-09T23:09:54.167 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000036", 2022-09-09T23:09:54.167 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:0", 2022-09-09T23:09:54.167 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:6", 2022-09-09T23:09:54.168 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:9", 2022-09-09T23:09:54.168 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:9", 2022-09-09T23:09:54.168 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000103", 2022-09-09T23:09:54.168 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:4", 2022-09-09T23:09:54.169 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:2", 2022-09-09T23:09:54.169 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:7", 2022-09-09T23:09:54.169 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:0", 2022-09-09T23:09:54.170 INFO:teuthology.orchestra.run.smithi005.stdout: "mdlog.sync-status.shard.3", 2022-09-09T23:09:54.170 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:3", 2022-09-09T23:09:54.170 INFO:teuthology.orchestra.run.smithi005.stdout: "meta.log.503b1d61-4513-48c8-a308-17b76219610c.0", 2022-09-09T23:09:54.170 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:2", 2022-09-09T23:09:54.171 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000119", 2022-09-09T23:09:54.171 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:9", 2022-09-09T23:09:54.171 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:9", 2022-09-09T23:09:54.172 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000098", 2022-09-09T23:09:54.172 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:3", 2022-09-09T23:09:54.172 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:3", 2022-09-09T23:09:54.173 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:2", 2022-09-09T23:09:54.173 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:9", 2022-09-09T23:09:54.173 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.shard.cc1fc6ed-f06f-45bc-961e-adec328c8651.1.retry", 2022-09-09T23:09:54.173 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:7", 2022-09-09T23:09:54.174 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:2", 2022-09-09T23:09:54.174 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:3", 2022-09-09T23:09:54.174 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:6", 2022-09-09T23:09:54.174 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:6", 2022-09-09T23:09:54.175 INFO:teuthology.orchestra.run.smithi005.stdout: "mdlog.sync-status.shard.0", 2022-09-09T23:09:54.175 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:6", 2022-09-09T23:09:54.175 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:10", 2022-09-09T23:09:54.176 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:5", 2022-09-09T23:09:54.176 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:10", 2022-09-09T23:09:54.176 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12", 2022-09-09T23:09:54.176 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:9", 2022-09-09T23:09:54.177 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:8", 2022-09-09T23:09:54.177 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000027", 2022-09-09T23:09:54.177 INFO:teuthology.orchestra.run.smithi005.stdout: "mdlog.sync-status", 2022-09-09T23:09:54.177 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000019", 2022-09-09T23:09:54.178 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:0", 2022-09-09T23:09:54.178 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:7", 2022-09-09T23:09:54.178 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:10", 2022-09-09T23:09:54.178 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:10", 2022-09-09T23:09:54.179 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:0", 2022-09-09T23:09:54.179 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:2", 2022-09-09T23:09:54.179 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000039", 2022-09-09T23:09:54.180 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:9", 2022-09-09T23:09:54.180 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:2", 2022-09-09T23:09:54.180 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:4", 2022-09-09T23:09:54.180 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:7", 2022-09-09T23:09:54.181 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14", 2022-09-09T23:09:54.181 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:0", 2022-09-09T23:09:54.181 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:9", 2022-09-09T23:09:54.181 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000100", 2022-09-09T23:09:54.182 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:3", 2022-09-09T23:09:54.182 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:4", 2022-09-09T23:09:54.182 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:10", 2022-09-09T23:09:54.182 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:7", 2022-09-09T23:09:54.183 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:2", 2022-09-09T23:09:54.183 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:7", 2022-09-09T23:09:54.183 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:3", 2022-09-09T23:09:54.183 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:1", 2022-09-09T23:09:54.184 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:5", 2022-09-09T23:09:54.184 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:8", 2022-09-09T23:09:54.184 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:3", 2022-09-09T23:09:54.184 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000093", 2022-09-09T23:09:54.185 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:7", 2022-09-09T23:09:54.185 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:4", 2022-09-09T23:09:54.185 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:6", 2022-09-09T23:09:54.185 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18", 2022-09-09T23:09:54.186 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:4", 2022-09-09T23:09:54.186 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000004", 2022-09-09T23:09:54.186 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000063", 2022-09-09T23:09:54.186 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:7", 2022-09-09T23:09:54.187 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000122", 2022-09-09T23:09:54.187 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:0", 2022-09-09T23:09:54.187 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:7", 2022-09-09T23:09:54.188 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:2", 2022-09-09T23:09:54.188 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000057", 2022-09-09T23:09:54.188 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:9", 2022-09-09T23:09:54.188 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:5", 2022-09-09T23:09:54.189 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:3", 2022-09-09T23:09:54.189 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:2", 2022-09-09T23:09:54.189 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:10", 2022-09-09T23:09:54.189 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:7", 2022-09-09T23:09:54.190 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000054", 2022-09-09T23:09:54.190 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000114", 2022-09-09T23:09:54.190 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:7", 2022-09-09T23:09:54.190 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:0", 2022-09-09T23:09:54.191 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.2", 2022-09-09T23:09:54.191 INFO:teuthology.orchestra.run.smithi005.stdout: "data_log.0", 2022-09-09T23:09:54.191 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:7", 2022-09-09T23:09:54.191 INFO:teuthology.orchestra.run.smithi005.stdout: "data_log.1.vG30I9/OOsJ.0", 2022-09-09T23:09:54.192 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:1", 2022-09-09T23:09:54.192 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:3", 2022-09-09T23:09:54.192 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:10", 2022-09-09T23:09:54.192 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:5", 2022-09-09T23:09:54.193 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:8", 2022-09-09T23:09:54.193 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:0", 2022-09-09T23:09:54.193 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000012", 2022-09-09T23:09:54.194 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:9", 2022-09-09T23:09:54.194 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:6", 2022-09-09T23:09:54.194 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:0", 2022-09-09T23:09:54.194 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.shard.cc1fc6ed-f06f-45bc-961e-adec328c8651.1", 2022-09-09T23:09:54.195 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000084", 2022-09-09T23:09:54.195 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000043", 2022-09-09T23:09:54.195 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:8", 2022-09-09T23:09:54.195 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38", 2022-09-09T23:09:54.196 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:4", 2022-09-09T23:09:54.196 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:9", 2022-09-09T23:09:54.196 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:9", 2022-09-09T23:09:54.196 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.11", 2022-09-09T23:09:54.197 INFO:teuthology.orchestra.run.smithi005.stdout: "datalog.sync-status.shard.4b05608c-7197-4e9b-bd29-7989607fa277.1.retry", 2022-09-09T23:09:54.197 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:0", 2022-09-09T23:09:54.197 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:7", 2022-09-09T23:09:54.197 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:10", 2022-09-09T23:09:54.198 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:3", 2022-09-09T23:09:54.198 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:10", 2022-09-09T23:09:54.198 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:9", 2022-09-09T23:09:54.199 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:5", 2022-09-09T23:09:54.199 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:4", 2022-09-09T23:09:54.199 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:1", 2022-09-09T23:09:54.199 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000111", 2022-09-09T23:09:54.200 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000048", 2022-09-09T23:09:54.200 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:9", 2022-09-09T23:09:54.200 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:6", 2022-09-09T23:09:54.200 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:0", 2022-09-09T23:09:54.201 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000020", 2022-09-09T23:09:54.201 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:7", 2022-09-09T23:09:54.201 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:4", 2022-09-09T23:09:54.201 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:8", 2022-09-09T23:09:54.202 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:1", 2022-09-09T23:09:54.202 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:1", 2022-09-09T23:09:54.202 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37", 2022-09-09T23:09:54.202 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:0", 2022-09-09T23:09:54.203 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:6", 2022-09-09T23:09:54.203 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000099", 2022-09-09T23:09:54.203 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000056", 2022-09-09T23:09:54.204 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:0", 2022-09-09T23:09:54.204 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:2", 2022-09-09T23:09:54.204 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:0", 2022-09-09T23:09:54.204 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:2", 2022-09-09T23:09:54.205 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:1", 2022-09-09T23:09:54.205 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:5", 2022-09-09T23:09:54.205 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:7", 2022-09-09T23:09:54.205 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000072", 2022-09-09T23:09:54.206 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:2", 2022-09-09T23:09:54.206 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:10", 2022-09-09T23:09:54.206 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10", 2022-09-09T23:09:54.207 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000062", 2022-09-09T23:09:54.207 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:9", 2022-09-09T23:09:54.207 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:1", 2022-09-09T23:09:54.208 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:10", 2022-09-09T23:09:54.208 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000115", 2022-09-09T23:09:54.208 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.1", 2022-09-09T23:09:54.208 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:6", 2022-09-09T23:09:54.209 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:1", 2022-09-09T23:09:54.209 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000033", 2022-09-09T23:09:54.209 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:10", 2022-09-09T23:09:54.210 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:1", 2022-09-09T23:09:54.210 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:4", 2022-09-09T23:09:54.210 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:7", 2022-09-09T23:09:54.210 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:9", 2022-09-09T23:09:54.211 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15", 2022-09-09T23:09:54.211 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:1", 2022-09-09T23:09:54.211 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:5", 2022-09-09T23:09:54.211 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:8", 2022-09-09T23:09:54.212 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:0", 2022-09-09T23:09:54.212 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:3", 2022-09-09T23:09:54.212 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:3", 2022-09-09T23:09:54.212 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:7", 2022-09-09T23:09:54.213 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000025", 2022-09-09T23:09:54.213 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11", 2022-09-09T23:09:54.213 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:10", 2022-09-09T23:09:54.213 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000123", 2022-09-09T23:09:54.214 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:1", 2022-09-09T23:09:54.214 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:0", 2022-09-09T23:09:54.214 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:0", 2022-09-09T23:09:54.215 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:3", 2022-09-09T23:09:54.215 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:6", 2022-09-09T23:09:54.215 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:2", 2022-09-09T23:09:54.215 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17", 2022-09-09T23:09:54.216 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:9", 2022-09-09T23:09:54.216 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:8", 2022-09-09T23:09:54.216 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:7", 2022-09-09T23:09:54.216 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000108", 2022-09-09T23:09:54.217 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:5", 2022-09-09T23:09:54.217 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:3", 2022-09-09T23:09:54.217 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:0", 2022-09-09T23:09:54.218 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:5", 2022-09-09T23:09:54.218 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000094", 2022-09-09T23:09:54.218 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:8", 2022-09-09T23:09:54.218 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:8", 2022-09-09T23:09:54.219 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:4", 2022-09-09T23:09:54.219 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:5", 2022-09-09T23:09:54.219 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:4", 2022-09-09T23:09:54.219 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:7", 2022-09-09T23:09:54.220 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:2", 2022-09-09T23:09:54.220 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000060", 2022-09-09T23:09:54.220 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:4", 2022-09-09T23:09:54.220 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000109", 2022-09-09T23:09:54.221 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.15", 2022-09-09T23:09:54.221 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:3", 2022-09-09T23:09:54.221 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:3", 2022-09-09T23:09:54.221 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:8", 2022-09-09T23:09:54.222 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:3", 2022-09-09T23:09:54.222 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:0", 2022-09-09T23:09:54.222 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:10", 2022-09-09T23:09:54.223 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000013", 2022-09-09T23:09:54.223 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:6", 2022-09-09T23:09:54.223 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:3", 2022-09-09T23:09:54.223 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:1", 2022-09-09T23:09:54.224 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:3", 2022-09-09T23:09:54.224 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:7", 2022-09-09T23:09:54.224 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4", 2022-09-09T23:09:54.224 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:0", 2022-09-09T23:09:54.225 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:4", 2022-09-09T23:09:54.225 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:10", 2022-09-09T23:09:54.225 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:5", 2022-09-09T23:09:54.225 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:1", 2022-09-09T23:09:54.226 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000051", 2022-09-09T23:09:54.226 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:2", 2022-09-09T23:09:54.226 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000106", 2022-09-09T23:09:54.227 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:2", 2022-09-09T23:09:54.227 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:2", 2022-09-09T23:09:54.227 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:2", 2022-09-09T23:09:54.227 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:1", 2022-09-09T23:09:54.228 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:3", 2022-09-09T23:09:54.228 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:1", 2022-09-09T23:09:54.228 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:6", 2022-09-09T23:09:54.228 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000017", 2022-09-09T23:09:54.229 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:5", 2022-09-09T23:09:54.229 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:8", 2022-09-09T23:09:54.229 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:1", 2022-09-09T23:09:54.230 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:8", 2022-09-09T23:09:54.230 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:9", 2022-09-09T23:09:54.230 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000023", 2022-09-09T23:09:54.230 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:4", 2022-09-09T23:09:54.231 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:4", 2022-09-09T23:09:54.231 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20", 2022-09-09T23:09:54.231 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000107", 2022-09-09T23:09:54.232 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:6", 2022-09-09T23:09:54.232 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000046", 2022-09-09T23:09:54.232 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:2", 2022-09-09T23:09:54.232 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:10", 2022-09-09T23:09:54.233 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26", 2022-09-09T23:09:54.233 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:9", 2022-09-09T23:09:54.233 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:5", 2022-09-09T23:09:54.234 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:8", 2022-09-09T23:09:54.234 INFO:teuthology.orchestra.run.smithi005.stdout: "meta.log.503b1d61-4513-48c8-a308-17b76219610c.3", 2022-09-09T23:09:54.234 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:9", 2022-09-09T23:09:54.235 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:1", 2022-09-09T23:09:54.235 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:6", 2022-09-09T23:09:54.235 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:6", 2022-09-09T23:09:54.236 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:2", 2022-09-09T23:09:54.236 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:1", 2022-09-09T23:09:54.236 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:6", 2022-09-09T23:09:54.237 INFO:teuthology.orchestra.run.smithi005.stdout: "data_log.2", 2022-09-09T23:09:54.237 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:9", 2022-09-09T23:09:54.237 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:2", 2022-09-09T23:09:54.237 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000067", 2022-09-09T23:09:54.238 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:3", 2022-09-09T23:09:54.238 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1", 2022-09-09T23:09:54.239 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:1", 2022-09-09T23:09:54.239 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:1", 2022-09-09T23:09:54.239 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:7", 2022-09-09T23:09:54.240 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:4", 2022-09-09T23:09:54.240 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:2", 2022-09-09T23:09:54.240 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:0", 2022-09-09T23:09:54.241 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.9", 2022-09-09T23:09:54.241 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000086", 2022-09-09T23:09:54.241 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:8", 2022-09-09T23:09:54.241 INFO:teuthology.orchestra.run.smithi005.stdout: "sync.error-log.5", 2022-09-09T23:09:54.242 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25", 2022-09-09T23:09:54.242 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:1", 2022-09-09T23:09:54.243 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:2", 2022-09-09T23:09:54.243 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:1", 2022-09-09T23:09:54.243 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:10", 2022-09-09T23:09:54.244 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:4", 2022-09-09T23:09:54.244 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:2", 2022-09-09T23:09:54.244 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:8", 2022-09-09T23:09:54.244 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:1", 2022-09-09T23:09:54.245 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:10", 2022-09-09T23:09:54.245 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:8", 2022-09-09T23:09:54.245 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:4", 2022-09-09T23:09:54.245 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:0", 2022-09-09T23:09:54.246 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:10", 2022-09-09T23:09:54.246 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:7", 2022-09-09T23:09:54.246 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4", 2022-09-09T23:09:54.246 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:9", 2022-09-09T23:09:54.247 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5", 2022-09-09T23:09:54.247 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:6", 2022-09-09T23:09:54.247 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:0", 2022-09-09T23:09:54.247 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:8", 2022-09-09T23:09:54.248 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:5", 2022-09-09T23:09:54.248 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:4", 2022-09-09T23:09:54.248 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:1", 2022-09-09T23:09:54.249 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:6", 2022-09-09T23:09:54.249 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000050", 2022-09-09T23:09:54.249 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3", 2022-09-09T23:09:54.249 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:9", 2022-09-09T23:09:54.250 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000102", 2022-09-09T23:09:54.250 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:5", 2022-09-09T23:09:54.250 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:3", 2022-09-09T23:09:54.251 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:10", 2022-09-09T23:09:54.251 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21", 2022-09-09T23:09:54.251 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10", 2022-09-09T23:09:54.251 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:6", 2022-09-09T23:09:54.252 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000124", 2022-09-09T23:09:54.252 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:6", 2022-09-09T23:09:54.252 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:6", 2022-09-09T23:09:54.252 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000001", 2022-09-09T23:09:54.253 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:1", 2022-09-09T23:09:54.253 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:3", 2022-09-09T23:09:54.253 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:7", 2022-09-09T23:09:54.253 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:0", 2022-09-09T23:09:54.254 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000126", 2022-09-09T23:09:54.254 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38", 2022-09-09T23:09:54.254 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:9", 2022-09-09T23:09:54.254 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000055", 2022-09-09T23:09:54.255 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000061", 2022-09-09T23:09:54.255 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:0", 2022-09-09T23:09:54.255 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:2", 2022-09-09T23:09:54.255 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20", 2022-09-09T23:09:54.256 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000097", 2022-09-09T23:09:54.256 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:7", 2022-09-09T23:09:54.256 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:4", 2022-09-09T23:09:54.257 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:9", 2022-09-09T23:09:54.257 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:10", 2022-09-09T23:09:54.257 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:3", 2022-09-09T23:09:54.257 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:6", 2022-09-09T23:09:54.258 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:9", 2022-09-09T23:09:54.258 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000042", 2022-09-09T23:09:54.258 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:10", 2022-09-09T23:09:54.259 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18", 2022-09-09T23:09:54.259 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:6", 2022-09-09T23:09:54.259 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:8", 2022-09-09T23:09:54.259 INFO:teuthology.orchestra.run.smithi005.stdout: "data_log.3.CWGQuPAV4cl.0", 2022-09-09T23:09:54.260 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3", 2022-09-09T23:09:54.260 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:4", 2022-09-09T23:09:54.260 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:1", 2022-09-09T23:09:54.261 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:3", 2022-09-09T23:09:54.261 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:4", 2022-09-09T23:09:54.261 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000075", 2022-09-09T23:09:54.262 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:0", 2022-09-09T23:09:54.262 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:0", 2022-09-09T23:09:54.262 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000120", 2022-09-09T23:09:54.262 INFO:teuthology.orchestra.run.smithi005.stdout: "obj_delete_at_hint.0000000030", 2022-09-09T23:09:54.263 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:3", 2022-09-09T23:09:54.263 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:10", 2022-09-09T23:09:54.263 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:1", 2022-09-09T23:09:54.264 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.cc1fc6ed-f06f-45bc-961e-adec328c8651:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:8", 2022-09-09T23:09:54.264 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:4", 2022-09-09T23:09:54.264 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:4" 2022-09-09T23:09:54.264 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T23:09:54.265 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-09-09T23:09:54.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:54.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:54.268 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:09:54.321 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:09:54.322 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:09:54.427 INFO:teuthology.orchestra.run.smithi080.stdout:[ 2022-09-09T23:09:54.427 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22", 2022-09-09T23:09:54.427 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000078", 2022-09-09T23:09:54.428 INFO:teuthology.orchestra.run.smithi080.stdout: "meta.history", 2022-09-09T23:09:54.428 INFO:teuthology.orchestra.run.smithi080.stdout: "datalog.sync-status.shard.4b05608c-7197-4e9b-bd29-7989607fa277.1", 2022-09-09T23:09:54.428 INFO:teuthology.orchestra.run.smithi080.stdout: "meta.log.fd4bc158-06fd-4840-8a27-33fabda67b99.3", 2022-09-09T23:09:54.429 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000070", 2022-09-09T23:09:54.429 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000104", 2022-09-09T23:09:54.429 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:0", 2022-09-09T23:09:54.429 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000026", 2022-09-09T23:09:54.430 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:6", 2022-09-09T23:09:54.430 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:10", 2022-09-09T23:09:54.431 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:9", 2022-09-09T23:09:54.431 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.8", 2022-09-09T23:09:54.431 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:3", 2022-09-09T23:09:54.432 INFO:teuthology.orchestra.run.smithi080.stdout: "datalog.sync-status.shard.4b05608c-7197-4e9b-bd29-7989607fa277.3.retry", 2022-09-09T23:09:54.432 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000028", 2022-09-09T23:09:54.432 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:4", 2022-09-09T23:09:54.432 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:5", 2022-09-09T23:09:54.433 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:3", 2022-09-09T23:09:54.433 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:6", 2022-09-09T23:09:54.433 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:3", 2022-09-09T23:09:54.433 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:1", 2022-09-09T23:09:54.434 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:0", 2022-09-09T23:09:54.434 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000040", 2022-09-09T23:09:54.434 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.6", 2022-09-09T23:09:54.435 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:1", 2022-09-09T23:09:54.435 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:7", 2022-09-09T23:09:54.435 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:10", 2022-09-09T23:09:54.435 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000015", 2022-09-09T23:09:54.436 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36", 2022-09-09T23:09:54.436 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000069", 2022-09-09T23:09:54.436 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000095", 2022-09-09T23:09:54.436 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:2", 2022-09-09T23:09:54.437 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:2", 2022-09-09T23:09:54.437 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000003", 2022-09-09T23:09:54.437 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000047", 2022-09-09T23:09:54.438 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:10", 2022-09-09T23:09:54.438 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000035", 2022-09-09T23:09:54.438 INFO:teuthology.orchestra.run.smithi080.stdout: "datalog.sync-status.shard.4b05608c-7197-4e9b-bd29-7989607fa277.0", 2022-09-09T23:09:54.438 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000037", 2022-09-09T23:09:54.439 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:2", 2022-09-09T23:09:54.439 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:8", 2022-09-09T23:09:54.439 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:6", 2022-09-09T23:09:54.439 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:8", 2022-09-09T23:09:54.440 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:2", 2022-09-09T23:09:54.440 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:5", 2022-09-09T23:09:54.440 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000024", 2022-09-09T23:09:54.441 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:5", 2022-09-09T23:09:54.441 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:7", 2022-09-09T23:09:54.441 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:8", 2022-09-09T23:09:54.441 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:0", 2022-09-09T23:09:54.442 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000000", 2022-09-09T23:09:54.442 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000031", 2022-09-09T23:09:54.442 INFO:teuthology.orchestra.run.smithi080.stdout: "datalog.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277", 2022-09-09T23:09:54.442 INFO:teuthology.orchestra.run.smithi080.stdout: "data_log.3", 2022-09-09T23:09:54.443 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:0", 2022-09-09T23:09:54.443 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:0", 2022-09-09T23:09:54.443 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:0", 2022-09-09T23:09:54.444 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:10", 2022-09-09T23:09:54.444 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:8", 2022-09-09T23:09:54.444 INFO:teuthology.orchestra.run.smithi080.stdout: "data_loggenerations_metadata", 2022-09-09T23:09:54.444 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000076", 2022-09-09T23:09:54.445 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000113", 2022-09-09T23:09:54.445 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000005", 2022-09-09T23:09:54.446 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:2", 2022-09-09T23:09:54.446 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000011", 2022-09-09T23:09:54.446 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000058", 2022-09-09T23:09:54.447 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:8", 2022-09-09T23:09:54.447 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000052", 2022-09-09T23:09:54.447 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:5", 2022-09-09T23:09:54.447 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:10", 2022-09-09T23:09:54.448 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:5", 2022-09-09T23:09:54.448 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:0", 2022-09-09T23:09:54.448 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:0", 2022-09-09T23:09:54.448 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:0", 2022-09-09T23:09:54.449 INFO:teuthology.orchestra.run.smithi080.stdout: "datalog.sync-status.shard.4b05608c-7197-4e9b-bd29-7989607fa277.2.retry", 2022-09-09T23:09:54.449 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000088", 2022-09-09T23:09:54.449 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:9", 2022-09-09T23:09:54.449 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000080", 2022-09-09T23:09:54.450 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000090", 2022-09-09T23:09:54.450 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000110", 2022-09-09T23:09:54.450 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:6", 2022-09-09T23:09:54.451 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:5", 2022-09-09T23:09:54.451 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:2", 2022-09-09T23:09:54.451 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:1", 2022-09-09T23:09:54.451 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:6", 2022-09-09T23:09:54.452 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000096", 2022-09-09T23:09:54.452 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:5", 2022-09-09T23:09:54.452 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:7", 2022-09-09T23:09:54.453 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000087", 2022-09-09T23:09:54.453 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000008", 2022-09-09T23:09:54.453 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:4", 2022-09-09T23:09:54.453 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000006", 2022-09-09T23:09:54.454 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000029", 2022-09-09T23:09:54.454 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:6", 2022-09-09T23:09:54.454 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000089", 2022-09-09T23:09:54.455 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:1", 2022-09-09T23:09:54.455 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000022", 2022-09-09T23:09:54.455 INFO:teuthology.orchestra.run.smithi080.stdout: "bilog.trim", 2022-09-09T23:09:54.455 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000016", 2022-09-09T23:09:54.456 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:1", 2022-09-09T23:09:54.456 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:3", 2022-09-09T23:09:54.456 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000041", 2022-09-09T23:09:54.457 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:7", 2022-09-09T23:09:54.457 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000018", 2022-09-09T23:09:54.457 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000092", 2022-09-09T23:09:54.457 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:1", 2022-09-09T23:09:54.458 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:8", 2022-09-09T23:09:54.458 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:8", 2022-09-09T23:09:54.458 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:4", 2022-09-09T23:09:54.458 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.4", 2022-09-09T23:09:54.459 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24", 2022-09-09T23:09:54.459 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:9", 2022-09-09T23:09:54.459 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:5", 2022-09-09T23:09:54.460 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:4", 2022-09-09T23:09:54.460 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:3", 2022-09-09T23:09:54.460 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:10", 2022-09-09T23:09:54.460 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000014", 2022-09-09T23:09:54.461 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11", 2022-09-09T23:09:54.461 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:6", 2022-09-09T23:09:54.461 INFO:teuthology.orchestra.run.smithi080.stdout: "datalog.sync-status.shard.4b05608c-7197-4e9b-bd29-7989607fa277.2", 2022-09-09T23:09:54.461 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:6", 2022-09-09T23:09:54.462 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:0", 2022-09-09T23:09:54.462 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000112", 2022-09-09T23:09:54.462 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:8", 2022-09-09T23:09:54.462 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000007", 2022-09-09T23:09:54.463 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000021", 2022-09-09T23:09:54.463 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:6", 2022-09-09T23:09:54.463 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:4", 2022-09-09T23:09:54.464 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000064", 2022-09-09T23:09:54.464 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000071", 2022-09-09T23:09:54.464 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:8", 2022-09-09T23:09:54.464 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16", 2022-09-09T23:09:54.465 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000074", 2022-09-09T23:09:54.465 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:6", 2022-09-09T23:09:54.465 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:5", 2022-09-09T23:09:54.465 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:8", 2022-09-09T23:09:54.466 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:3", 2022-09-09T23:09:54.466 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:1", 2022-09-09T23:09:54.466 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:9", 2022-09-09T23:09:54.467 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:5", 2022-09-09T23:09:54.467 INFO:teuthology.orchestra.run.smithi080.stdout: "datalog.sync-status.shard.4b05608c-7197-4e9b-bd29-7989607fa277.3", 2022-09-09T23:09:54.467 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000081", 2022-09-09T23:09:54.467 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000009", 2022-09-09T23:09:54.468 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:10", 2022-09-09T23:09:54.468 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:8", 2022-09-09T23:09:54.468 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000121", 2022-09-09T23:09:54.468 INFO:teuthology.orchestra.run.smithi080.stdout: "data_log.1", 2022-09-09T23:09:54.469 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14", 2022-09-09T23:09:54.469 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:3", 2022-09-09T23:09:54.469 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:8", 2022-09-09T23:09:54.469 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23", 2022-09-09T23:09:54.470 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000125", 2022-09-09T23:09:54.470 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:4", 2022-09-09T23:09:54.471 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000082", 2022-09-09T23:09:54.471 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:5", 2022-09-09T23:09:54.472 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000105", 2022-09-09T23:09:54.472 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000059", 2022-09-09T23:09:54.472 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:4", 2022-09-09T23:09:54.472 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:8", 2022-09-09T23:09:54.473 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:3", 2022-09-09T23:09:54.473 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000077", 2022-09-09T23:09:54.473 INFO:teuthology.orchestra.run.smithi080.stdout: "data_log.0.mYLAxmiHK9+.0", 2022-09-09T23:09:54.474 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:1", 2022-09-09T23:09:54.474 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000032", 2022-09-09T23:09:54.474 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:6", 2022-09-09T23:09:54.474 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:0", 2022-09-09T23:09:54.475 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:6", 2022-09-09T23:09:54.475 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:3", 2022-09-09T23:09:54.475 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:9", 2022-09-09T23:09:54.476 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:6", 2022-09-09T23:09:54.476 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:1", 2022-09-09T23:09:54.477 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:4", 2022-09-09T23:09:54.477 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:7", 2022-09-09T23:09:54.477 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:1", 2022-09-09T23:09:54.477 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:9", 2022-09-09T23:09:54.478 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:3", 2022-09-09T23:09:54.478 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000053", 2022-09-09T23:09:54.478 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000091", 2022-09-09T23:09:54.478 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:4", 2022-09-09T23:09:54.479 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000065", 2022-09-09T23:09:54.479 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000083", 2022-09-09T23:09:54.479 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000010", 2022-09-09T23:09:54.479 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:8", 2022-09-09T23:09:54.480 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.7", 2022-09-09T23:09:54.480 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.3", 2022-09-09T23:09:54.480 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:3", 2022-09-09T23:09:54.480 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:2", 2022-09-09T23:09:54.481 INFO:teuthology.orchestra.run.smithi080.stdout: "meta.log.503b1d61-4513-48c8-a308-17b76219610c.2", 2022-09-09T23:09:54.481 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:2", 2022-09-09T23:09:54.482 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26", 2022-09-09T23:09:54.482 INFO:teuthology.orchestra.run.smithi080.stdout: "mdlog.sync-status.shard.2", 2022-09-09T23:09:54.482 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:4", 2022-09-09T23:09:54.483 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:8", 2022-09-09T23:09:54.484 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:6", 2022-09-09T23:09:54.484 INFO:teuthology.orchestra.run.smithi080.stdout: "data_log.3.0LSM9tE1aH/.0", 2022-09-09T23:09:54.484 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000045", 2022-09-09T23:09:54.485 INFO:teuthology.orchestra.run.smithi080.stdout: "meta.log.2efdc3a1-023b-4e78-aaa2-0e069a034dac.3", 2022-09-09T23:09:54.485 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:4", 2022-09-09T23:09:54.486 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000002", 2022-09-09T23:09:54.486 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:0", 2022-09-09T23:09:54.486 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:0", 2022-09-09T23:09:54.487 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.12", 2022-09-09T23:09:54.488 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:4", 2022-09-09T23:09:54.488 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:5", 2022-09-09T23:09:54.489 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:0", 2022-09-09T23:09:54.489 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15", 2022-09-09T23:09:54.490 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000116", 2022-09-09T23:09:54.490 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1", 2022-09-09T23:09:54.491 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000034", 2022-09-09T23:09:54.491 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13", 2022-09-09T23:09:54.492 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000101", 2022-09-09T23:09:54.492 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:7", 2022-09-09T23:09:54.493 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.13", 2022-09-09T23:09:54.493 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21", 2022-09-09T23:09:54.493 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:2", 2022-09-09T23:09:54.494 INFO:teuthology.orchestra.run.smithi080.stdout: "mdlog.sync-status.shard.1", 2022-09-09T23:09:54.494 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000079", 2022-09-09T23:09:54.495 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000049", 2022-09-09T23:09:54.495 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37", 2022-09-09T23:09:54.496 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000117", 2022-09-09T23:09:54.496 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000044", 2022-09-09T23:09:54.496 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:7", 2022-09-09T23:09:54.497 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:10", 2022-09-09T23:09:54.497 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.14", 2022-09-09T23:09:54.497 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000066", 2022-09-09T23:09:54.498 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:7", 2022-09-09T23:09:54.498 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:9", 2022-09-09T23:09:54.498 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:9", 2022-09-09T23:09:54.498 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:5", 2022-09-09T23:09:54.499 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:9", 2022-09-09T23:09:54.499 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:6", 2022-09-09T23:09:54.499 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000068", 2022-09-09T23:09:54.500 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:2", 2022-09-09T23:09:54.500 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000085", 2022-09-09T23:09:54.500 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:3", 2022-09-09T23:09:54.500 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:9", 2022-09-09T23:09:54.501 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000073", 2022-09-09T23:09:54.501 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000038", 2022-09-09T23:09:54.501 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:3", 2022-09-09T23:09:54.502 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.10", 2022-09-09T23:09:54.502 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000118", 2022-09-09T23:09:54.502 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19", 2022-09-09T23:09:54.502 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000036", 2022-09-09T23:09:54.503 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:6", 2022-09-09T23:09:54.503 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:9", 2022-09-09T23:09:54.503 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000103", 2022-09-09T23:09:54.504 INFO:teuthology.orchestra.run.smithi080.stdout: "data_log.2.QlCpXXmhl8W.0", 2022-09-09T23:09:54.504 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:7", 2022-09-09T23:09:54.504 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:0", 2022-09-09T23:09:54.504 INFO:teuthology.orchestra.run.smithi080.stdout: "mdlog.sync-status.shard.3", 2022-09-09T23:09:54.505 INFO:teuthology.orchestra.run.smithi080.stdout: "meta.log.503b1d61-4513-48c8-a308-17b76219610c.0", 2022-09-09T23:09:54.505 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:2", 2022-09-09T23:09:54.505 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000119", 2022-09-09T23:09:54.505 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:9", 2022-09-09T23:09:54.506 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:9", 2022-09-09T23:09:54.506 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000098", 2022-09-09T23:09:54.506 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:3", 2022-09-09T23:09:54.506 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:9", 2022-09-09T23:09:54.507 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:2", 2022-09-09T23:09:54.507 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:6", 2022-09-09T23:09:54.507 INFO:teuthology.orchestra.run.smithi080.stdout: "mdlog.sync-status.shard.0", 2022-09-09T23:09:54.508 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:6", 2022-09-09T23:09:54.508 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:10", 2022-09-09T23:09:54.508 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12", 2022-09-09T23:09:54.509 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:8", 2022-09-09T23:09:54.509 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000027", 2022-09-09T23:09:54.509 INFO:teuthology.orchestra.run.smithi080.stdout: "mdlog.sync-status", 2022-09-09T23:09:54.509 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000019", 2022-09-09T23:09:54.510 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:7", 2022-09-09T23:09:54.510 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:10", 2022-09-09T23:09:54.510 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000039", 2022-09-09T23:09:54.511 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:4", 2022-09-09T23:09:54.511 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:7", 2022-09-09T23:09:54.511 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:0", 2022-09-09T23:09:54.511 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:9", 2022-09-09T23:09:54.512 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000100", 2022-09-09T23:09:54.512 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:3", 2022-09-09T23:09:54.512 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:10", 2022-09-09T23:09:54.512 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:2", 2022-09-09T23:09:54.513 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:7", 2022-09-09T23:09:54.513 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000093", 2022-09-09T23:09:54.513 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:7", 2022-09-09T23:09:54.514 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18", 2022-09-09T23:09:54.514 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000004", 2022-09-09T23:09:54.515 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000063", 2022-09-09T23:09:54.515 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:7", 2022-09-09T23:09:54.515 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000122", 2022-09-09T23:09:54.516 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:7", 2022-09-09T23:09:54.516 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000057", 2022-09-09T23:09:54.516 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:5", 2022-09-09T23:09:54.516 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:2", 2022-09-09T23:09:54.517 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:10", 2022-09-09T23:09:54.517 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:7", 2022-09-09T23:09:54.517 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000054", 2022-09-09T23:09:54.518 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000114", 2022-09-09T23:09:54.518 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:0", 2022-09-09T23:09:54.518 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.2", 2022-09-09T23:09:54.518 INFO:teuthology.orchestra.run.smithi080.stdout: "data_log.0", 2022-09-09T23:09:54.519 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:7", 2022-09-09T23:09:54.519 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:1", 2022-09-09T23:09:54.519 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:10", 2022-09-09T23:09:54.519 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:5", 2022-09-09T23:09:54.520 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:8", 2022-09-09T23:09:54.520 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000012", 2022-09-09T23:09:54.520 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:9", 2022-09-09T23:09:54.520 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:6", 2022-09-09T23:09:54.521 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000084", 2022-09-09T23:09:54.521 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000043", 2022-09-09T23:09:54.521 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:8", 2022-09-09T23:09:54.521 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38", 2022-09-09T23:09:54.522 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.11", 2022-09-09T23:09:54.522 INFO:teuthology.orchestra.run.smithi080.stdout: "datalog.sync-status.shard.4b05608c-7197-4e9b-bd29-7989607fa277.1.retry", 2022-09-09T23:09:54.522 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:7", 2022-09-09T23:09:54.522 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:10", 2022-09-09T23:09:54.523 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:3", 2022-09-09T23:09:54.523 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:10", 2022-09-09T23:09:54.523 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:9", 2022-09-09T23:09:54.523 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:5", 2022-09-09T23:09:54.524 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:4", 2022-09-09T23:09:54.524 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:1", 2022-09-09T23:09:54.525 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000111", 2022-09-09T23:09:54.525 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000048", 2022-09-09T23:09:54.525 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:9", 2022-09-09T23:09:54.525 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:0", 2022-09-09T23:09:54.526 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000020", 2022-09-09T23:09:54.526 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:7", 2022-09-09T23:09:54.526 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:8", 2022-09-09T23:09:54.526 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:1", 2022-09-09T23:09:54.527 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:1", 2022-09-09T23:09:54.527 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000099", 2022-09-09T23:09:54.527 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000056", 2022-09-09T23:09:54.527 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:0", 2022-09-09T23:09:54.528 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:2", 2022-09-09T23:09:54.528 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:2", 2022-09-09T23:09:54.528 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:1", 2022-09-09T23:09:54.528 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000072", 2022-09-09T23:09:54.529 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:10", 2022-09-09T23:09:54.529 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10", 2022-09-09T23:09:54.529 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000062", 2022-09-09T23:09:54.529 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:9", 2022-09-09T23:09:54.530 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:1", 2022-09-09T23:09:54.530 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:10", 2022-09-09T23:09:54.530 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000115", 2022-09-09T23:09:54.530 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.1", 2022-09-09T23:09:54.531 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:1", 2022-09-09T23:09:54.531 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000033", 2022-09-09T23:09:54.531 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:4", 2022-09-09T23:09:54.531 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:7", 2022-09-09T23:09:54.532 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:9", 2022-09-09T23:09:54.532 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:5", 2022-09-09T23:09:54.532 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:8", 2022-09-09T23:09:54.532 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:0", 2022-09-09T23:09:54.533 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:3", 2022-09-09T23:09:54.533 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000025", 2022-09-09T23:09:54.533 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:10", 2022-09-09T23:09:54.533 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000123", 2022-09-09T23:09:54.534 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:0", 2022-09-09T23:09:54.534 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:6", 2022-09-09T23:09:54.534 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-37:4b05608c-7197-4e9b-bd29-7989607fa277.4354.37:2", 2022-09-09T23:09:54.534 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17", 2022-09-09T23:09:54.535 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000108", 2022-09-09T23:09:54.535 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:5", 2022-09-09T23:09:54.535 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:3", 2022-09-09T23:09:54.536 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:5", 2022-09-09T23:09:54.536 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000094", 2022-09-09T23:09:54.536 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:8", 2022-09-09T23:09:54.536 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:5", 2022-09-09T23:09:54.537 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:4", 2022-09-09T23:09:54.537 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:7", 2022-09-09T23:09:54.537 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:2", 2022-09-09T23:09:54.537 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000060", 2022-09-09T23:09:54.538 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:4", 2022-09-09T23:09:54.538 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000109", 2022-09-09T23:09:54.538 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:8", 2022-09-09T23:09:54.538 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:3", 2022-09-09T23:09:54.539 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-15:4b05608c-7197-4e9b-bd29-7989607fa277.4354.15:0", 2022-09-09T23:09:54.539 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:10", 2022-09-09T23:09:54.539 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000013", 2022-09-09T23:09:54.539 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4:3", 2022-09-09T23:09:54.540 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:3", 2022-09-09T23:09:54.540 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5:7", 2022-09-09T23:09:54.540 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-47:4b05608c-7197-4e9b-bd29-7989607fa277.5119.4", 2022-09-09T23:09:54.540 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:4", 2022-09-09T23:09:54.541 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:10", 2022-09-09T23:09:54.541 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:5", 2022-09-09T23:09:54.541 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000051", 2022-09-09T23:09:54.541 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000106", 2022-09-09T23:09:54.542 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:2", 2022-09-09T23:09:54.542 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:2", 2022-09-09T23:09:54.542 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:1", 2022-09-09T23:09:54.542 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000017", 2022-09-09T23:09:54.543 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:5", 2022-09-09T23:09:54.543 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000023", 2022-09-09T23:09:54.543 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-10:4b05608c-7197-4e9b-bd29-7989607fa277.4354.10:4", 2022-09-09T23:09:54.543 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000107", 2022-09-09T23:09:54.544 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000046", 2022-09-09T23:09:54.544 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-13:4b05608c-7197-4e9b-bd29-7989607fa277.4354.13:10", 2022-09-09T23:09:54.544 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-16:4b05608c-7197-4e9b-bd29-7989607fa277.4354.16:5", 2022-09-09T23:09:54.544 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:8", 2022-09-09T23:09:54.545 INFO:teuthology.orchestra.run.smithi080.stdout: "meta.log.503b1d61-4513-48c8-a308-17b76219610c.3", 2022-09-09T23:09:54.545 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:6", 2022-09-09T23:09:54.545 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:2", 2022-09-09T23:09:54.545 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:1", 2022-09-09T23:09:54.546 INFO:teuthology.orchestra.run.smithi080.stdout: "data_log.2", 2022-09-09T23:09:54.546 INFO:teuthology.orchestra.run.smithi080.stdout: "data_log.1.jFxiXliNQ6e.0", 2022-09-09T23:09:54.546 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000067", 2022-09-09T23:09:54.546 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-38:4b05608c-7197-4e9b-bd29-7989607fa277.4354.38:3", 2022-09-09T23:09:54.547 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:1", 2022-09-09T23:09:54.547 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:2", 2022-09-09T23:09:54.547 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.9", 2022-09-09T23:09:54.547 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000086", 2022-09-09T23:09:54.548 INFO:teuthology.orchestra.run.smithi080.stdout: "sync.error-log.5", 2022-09-09T23:09:54.548 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25", 2022-09-09T23:09:54.548 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:2", 2022-09-09T23:09:54.548 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:1", 2022-09-09T23:09:54.548 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:10", 2022-09-09T23:09:54.549 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:4", 2022-09-09T23:09:54.549 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:2", 2022-09-09T23:09:54.549 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:1", 2022-09-09T23:09:54.549 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-11:4b05608c-7197-4e9b-bd29-7989607fa277.4354.11:10", 2022-09-09T23:09:54.550 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:7", 2022-09-09T23:09:54.550 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3:9", 2022-09-09T23:09:54.550 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-48:4b05608c-7197-4e9b-bd29-7989607fa277.5119.5", 2022-09-09T23:09:54.550 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20:6", 2022-09-09T23:09:54.551 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:4", 2022-09-09T23:09:54.551 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:1", 2022-09-09T23:09:54.551 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000050", 2022-09-09T23:09:54.551 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-25:4b05608c-7197-4e9b-bd29-7989607fa277.4354.25:9", 2022-09-09T23:09:54.552 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000102", 2022-09-09T23:09:54.552 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:5", 2022-09-09T23:09:54.552 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:6", 2022-09-09T23:09:54.552 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000124", 2022-09-09T23:09:54.553 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000001", 2022-09-09T23:09:54.553 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-17:4b05608c-7197-4e9b-bd29-7989607fa277.4354.17:0", 2022-09-09T23:09:54.553 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000126", 2022-09-09T23:09:54.553 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:9", 2022-09-09T23:09:54.554 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000055", 2022-09-09T23:09:54.554 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000061", 2022-09-09T23:09:54.554 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-20:4b05608c-7197-4e9b-bd29-7989607fa277.4354.20", 2022-09-09T23:09:54.554 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000097", 2022-09-09T23:09:54.555 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-14:4b05608c-7197-4e9b-bd29-7989607fa277.4354.14:7", 2022-09-09T23:09:54.555 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-26:4b05608c-7197-4e9b-bd29-7989607fa277.4354.26:9", 2022-09-09T23:09:54.555 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-12:4b05608c-7197-4e9b-bd29-7989607fa277.4354.12:3", 2022-09-09T23:09:54.555 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-21:4b05608c-7197-4e9b-bd29-7989607fa277.4354.21:6", 2022-09-09T23:09:54.555 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000042", 2022-09-09T23:09:54.556 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-19:4b05608c-7197-4e9b-bd29-7989607fa277.4354.19:10", 2022-09-09T23:09:54.556 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-22:4b05608c-7197-4e9b-bd29-7989607fa277.4354.22:8", 2022-09-09T23:09:54.556 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.full-sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-46:4b05608c-7197-4e9b-bd29-7989607fa277.5119.3", 2022-09-09T23:09:54.556 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-36:4b05608c-7197-4e9b-bd29-7989607fa277.4354.36:4", 2022-09-09T23:09:54.557 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-24:4b05608c-7197-4e9b-bd29-7989607fa277.4354.24:1", 2022-09-09T23:09:54.557 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000075", 2022-09-09T23:09:54.557 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:0", 2022-09-09T23:09:54.557 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000120", 2022-09-09T23:09:54.558 INFO:teuthology.orchestra.run.smithi080.stdout: "obj_delete_at_hint.0000000030", 2022-09-09T23:09:54.558 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-49:4b05608c-7197-4e9b-bd29-7989607fa277.5371.1:3", 2022-09-09T23:09:54.558 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-18:4b05608c-7197-4e9b-bd29-7989607fa277.4354.18:4", 2022-09-09T23:09:54.558 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket.sync-status.4b05608c-7197-4e9b-bd29-7989607fa277:jkcgbj-23:4b05608c-7197-4e9b-bd29-7989607fa277.4354.23:4" 2022-09-09T23:09:54.559 INFO:teuthology.orchestra.run.smithi080.stdout:] 2022-09-09T23:09:54.559 INFO:teuthology.orchestra.run.smithi080.stdout: 2022-09-09T23:09:54.560 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone2 2022-09-09T23:09:54.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:54.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:54.562 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:09:54.610 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:09:54.610 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:09:54.708 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T23:09:54.709 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:09:54.709 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-09T23:09:54.710 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T23:09:53.013758Z" 2022-09-09T23:09:54.711 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:09:54.711 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:09:54.712 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000003609", 2022-09-09T23:09:54.712 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T23:09:53.006301Z" 2022-09-09T23:09:54.713 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:09:54.713 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:09:54.714 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000005859", 2022-09-09T23:09:54.714 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T23:09:53.008700Z" 2022-09-09T23:09:54.715 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:09:54.716 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:09:54.716 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000007551", 2022-09-09T23:09:54.717 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T23:09:53.011152Z" 2022-09-09T23:09:54.717 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:09:54.718 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T23:09:54.720 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000002060', 1: '00000000000000000000:00000000000000003609', 2: '00000000000000000000:00000000000000005859', 3: '00000000000000000000:00000000000000007551'} 2022-09-09T23:09:54.721 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2022-09-09T23:09:54.722 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:54.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:54.725 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:09:54.776 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:09:54.777 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:09:54.876 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:09:54.876 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:09:54.876 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:09:54.877 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:09:54.877 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:09:54.877 INFO:teuthology.orchestra.run.smithi080.stdout: "instance_id": 315074542659551751 2022-09-09T23:09:54.877 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:54.878 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:09:54.878 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:54.878 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:09:54.879 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:54.879 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:09:54.879 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-09T23:09:54.879 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:54.879 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:54.880 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:54.880 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:59:53.537383Z" 2022-09-09T23:09:54.880 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:54.880 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:54.881 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:54.881 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:09:54.881 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:54.881 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:09:54.882 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000002768", 2022-09-09T23:09:54.882 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:54.882 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:54.882 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:54.883 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:38:39.082329Z" 2022-09-09T23:09:54.883 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:54.883 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:54.883 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:54.884 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:09:54.884 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:54.884 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:09:54.884 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000004599", 2022-09-09T23:09:54.885 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:54.885 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:54.885 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:54.885 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:00:13.517424Z" 2022-09-09T23:09:54.886 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:54.886 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:54.886 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:09:54.886 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:09:54.887 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:09:54.887 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:09:54.887 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000006291", 2022-09-09T23:09:54.887 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:09:54.887 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:09:54.888 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:09:54.888 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:03:45.134880Z" 2022-09-09T23:09:54.888 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:54.888 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:09:54.889 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:09:54.889 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:54.889 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:09:54.889 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 0, 2022-09-09T23:09:54.890 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 0 2022-09-09T23:09:54.890 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:09:54.890 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:09:54Z" 2022-09-09T23:09:54.890 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:09:54.892 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 315074542659551751 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:59:53.537383Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002768", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:38:39.082329Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004599", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:00:13.517424Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006291", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:03:45.134880Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-09T23:09:54Z" } 2022-09-09T23:09:54.892 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:59:53.537383Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002768', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:38:39.082329Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004599', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:00:13.517424Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006291', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:03:45.134880Z'}}] 2022-09-09T23:09:54.893 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002060', 1: '00000000000000000000:00000000000000003609', 2: '00000000000000000000:00000000000000005859', 3: '00000000000000000000:00000000000000007551'} 2022-09-09T23:09:54.893 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002768', 2: '00000000000000000000:00000000000000004599', 3: '00000000000000000000:00000000000000006291'} 2022-09-09T23:09:54.894 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002060 target=00000000000000000000:00000000000000000796, shard=1 master=00000000000000000000:00000000000000003609 target=00000000000000000000:00000000000000002768, shard=2 master=00000000000000000000:00000000000000005859 target=00000000000000000000:00000000000000004599, shard=3 master=00000000000000000000:00000000000000007551 target=00000000000000000000:00000000000000006291 2022-09-09T23:09:59.895 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:59.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:09:59.897 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:09:59.947 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:09:59.948 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:10:00.048 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:10:00.048 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:10:00.049 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:10:00.049 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:10:00.050 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:10:00.051 INFO:teuthology.orchestra.run.smithi080.stdout: "instance_id": 315074542659551751 2022-09-09T23:10:00.051 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:00.051 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:10:00.052 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:00.052 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:10:00.052 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:00.053 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:10:00.053 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-09T23:10:00.053 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:00.054 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:00.054 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:00.054 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:09:53.013758Z" 2022-09-09T23:10:00.055 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:00.055 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:00.055 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:00.056 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:10:00.056 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:00.056 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:10:00.057 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000003609", 2022-09-09T23:10:00.057 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:00.057 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:00.058 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:00.058 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:09:53.006301Z" 2022-09-09T23:10:00.058 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:00.059 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:00.061 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:00.062 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:10:00.062 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:00.063 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:10:00.063 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000005859", 2022-09-09T23:10:00.064 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:00.064 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:00.065 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:00.066 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:09:53.008700Z" 2022-09-09T23:10:00.067 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:00.067 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:00.067 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:00.068 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:10:00.069 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:00.069 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:10:00.069 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000007551", 2022-09-09T23:10:00.070 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:00.070 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:00.070 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:00.070 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:09:53.011152Z" 2022-09-09T23:10:00.071 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:00.072 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:00.072 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:10:00.073 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:00.073 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:10:00.073 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 0, 2022-09-09T23:10:00.074 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 0 2022-09-09T23:10:00.074 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:00.074 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:10:00Z" 2022-09-09T23:10:00.074 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:10:00.076 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 315074542659551751 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002060", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:09:53.013758Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003609", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:09:53.006301Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005859", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:09:53.008700Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007551", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:09:53.011152Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-09T23:10:00Z" } 2022-09-09T23:10:00.077 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002060', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:09:53.013758Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003609', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:09:53.006301Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005859', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:09:53.008700Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007551', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:09:53.011152Z'}}] 2022-09-09T23:10:00.078 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002060', 1: '00000000000000000000:00000000000000003609', 2: '00000000000000000000:00000000000000005859', 3: '00000000000000000000:00000000000000007551'} 2022-09-09T23:10:00.078 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002060', 1: '00000000000000000000:00000000000000003609', 2: '00000000000000000000:00000000000000005859', 3: '00000000000000000000:00000000000000007551'} 2022-09-09T23:10:00.079 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2022-09-09T23:10:00.080 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone3 2022-09-09T23:10:00.080 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:00.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:00.083 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:10:00.131 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:10:00.131 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:10:00.231 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T23:10:00.232 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:00.233 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-09T23:10:00.233 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T23:09:53.013758Z" 2022-09-09T23:10:00.234 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:00.234 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:00.235 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000003609", 2022-09-09T23:10:00.235 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T23:09:53.006301Z" 2022-09-09T23:10:00.236 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:00.236 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:00.237 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000005859", 2022-09-09T23:10:00.237 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T23:09:53.008700Z" 2022-09-09T23:10:00.238 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:00.238 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:00.239 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000007551", 2022-09-09T23:10:00.239 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T23:09:53.011152Z" 2022-09-09T23:10:00.239 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:00.240 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T23:10:00.242 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000002060', 1: '00000000000000000000:00000000000000003609', 2: '00000000000000000000:00000000000000005859', 3: '00000000000000000000:00000000000000007551'} 2022-09-09T23:10:00.243 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2022-09-09T23:10:00.243 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:00.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:00.244 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:10:00.293 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:10:00.293 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:10:00.380 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T23:10:00.380 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T23:10:00.380 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T23:10:00.381 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T23:10:00.381 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T23:10:00.381 INFO:teuthology.orchestra.run.smithi005.stdout: "instance_id": 7350307779150546029 2022-09-09T23:10:00.381 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:00.382 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T23:10:00.382 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:00.382 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T23:10:00.382 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:10:00.383 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:10:00.383 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-09T23:10:00.383 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:10:00.384 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:10:00.384 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:10:00.384 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T23:09:53.013758Z" 2022-09-09T23:10:00.385 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:00.385 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:00.385 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:00.385 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T23:10:00.386 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:10:00.386 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:10:00.386 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000003609", 2022-09-09T23:10:00.386 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:10:00.387 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:10:00.387 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:10:00.387 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T23:09:53.006301Z" 2022-09-09T23:10:00.387 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:00.388 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:00.388 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:00.388 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T23:10:00.389 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:10:00.389 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:10:00.389 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000005859", 2022-09-09T23:10:00.389 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:10:00.390 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:10:00.390 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:10:00.390 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T23:09:53.008700Z" 2022-09-09T23:10:00.390 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:00.391 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:00.391 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:00.391 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T23:10:00.391 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:10:00.392 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:10:00.392 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000007551", 2022-09-09T23:10:00.392 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:10:00.392 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:10:00.393 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:10:00.393 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T23:09:53.011152Z" 2022-09-09T23:10:00.393 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:00.393 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:00.394 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:10:00.394 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:00.394 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T23:10:00.394 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 0, 2022-09-09T23:10:00.395 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 0 2022-09-09T23:10:00.395 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:00.395 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T23:10:00Z" 2022-09-09T23:10:00.395 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T23:10:00.397 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7350307779150546029 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002060", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:09:53.013758Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003609", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:09:53.006301Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005859", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:09:53.008700Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007551", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:09:53.011152Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-09T23:10:00Z" } 2022-09-09T23:10:00.398 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002060', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:09:53.013758Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003609', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:09:53.006301Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005859', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:09:53.008700Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007551', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:09:53.011152Z'}}] 2022-09-09T23:10:00.398 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002060', 1: '00000000000000000000:00000000000000003609', 2: '00000000000000000000:00000000000000005859', 3: '00000000000000000000:00000000000000007551'} 2022-09-09T23:10:00.399 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002060', 1: '00000000000000000000:00000000000000003609', 2: '00000000000000000000:00000000000000005859', 3: '00000000000000000000:00000000000000007551'} 2022-09-09T23:10:00.399 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2022-09-09T23:10:00.400 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone4 2022-09-09T23:10:00.400 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:00.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:00.401 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:10:00.447 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:10:00.448 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:10:00.546 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T23:10:00.547 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:00.547 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-09T23:10:00.547 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T23:09:53.013758Z" 2022-09-09T23:10:00.548 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:00.548 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:00.549 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000003609", 2022-09-09T23:10:00.549 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T23:09:53.006301Z" 2022-09-09T23:10:00.549 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:00.550 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:00.551 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000005859", 2022-09-09T23:10:00.551 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T23:09:53.008700Z" 2022-09-09T23:10:00.551 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:00.552 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:00.552 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000007551", 2022-09-09T23:10:00.552 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T23:09:53.011152Z" 2022-09-09T23:10:00.553 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:00.553 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T23:10:00.555 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000002060', 1: '00000000000000000000:00000000000000003609', 2: '00000000000000000000:00000000000000005859', 3: '00000000000000000000:00000000000000007551'} 2022-09-09T23:10:00.556 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2022-09-09T23:10:00.557 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:00.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:00.558 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:10:00.606 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:10:00.607 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:10:00.687 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:10:00.687 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:10:00.687 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:10:00.688 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:10:00.688 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:10:00.688 INFO:teuthology.orchestra.run.smithi080.stdout: "instance_id": 4950540593819681557 2022-09-09T23:10:00.689 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:00.689 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:10:00.689 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:00.690 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:10:00.690 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:00.690 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:10:00.690 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-09T23:10:00.691 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:00.691 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:00.691 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:00.692 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:09:53.013758Z" 2022-09-09T23:10:00.692 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:00.692 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:00.692 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:00.693 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:10:00.693 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:00.693 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:10:00.693 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000003609", 2022-09-09T23:10:00.694 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:00.694 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:00.694 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:00.694 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:09:53.006301Z" 2022-09-09T23:10:00.695 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:00.695 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:00.695 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:00.695 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:10:00.696 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:00.696 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:10:00.696 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000005859", 2022-09-09T23:10:00.696 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:00.697 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:00.697 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:00.697 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:09:53.008700Z" 2022-09-09T23:10:00.698 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:00.698 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:00.698 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:00.698 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:10:00.699 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:00.700 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:10:00.700 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000007551", 2022-09-09T23:10:00.701 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:00.701 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:00.701 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:00.702 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:09:53.011152Z" 2022-09-09T23:10:00.702 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:00.702 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:00.702 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:10:00.703 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:00.703 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:10:00.703 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 0, 2022-09-09T23:10:00.704 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 0 2022-09-09T23:10:00.704 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:00.704 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:10:00Z" 2022-09-09T23:10:00.704 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:10:00.706 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 4950540593819681557 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002060", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:09:53.013758Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003609", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:09:53.006301Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005859", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:09:53.008700Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007551", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:09:53.011152Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-09T23:10:00Z" } 2022-09-09T23:10:00.706 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002060', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:09:53.013758Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003609', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:09:53.006301Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005859', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:09:53.008700Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007551', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:09:53.011152Z'}}] 2022-09-09T23:10:00.707 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002060', 1: '00000000000000000000:00000000000000003609', 2: '00000000000000000000:00000000000000005859', 3: '00000000000000000000:00000000000000007551'} 2022-09-09T23:10:00.708 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002060', 1: '00000000000000000000:00000000000000003609', 2: '00000000000000000000:00000000000000005859', 3: '00000000000000000000:00000000000000007551'} 2022-09-09T23:10:00.708 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2022-09-09T23:10:00.709 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone1 2022-09-09T23:10:00.709 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:00.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:00.711 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:10:00.758 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:10:00.759 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:10:00.877 INFO:teuthology.orchestra.run.smithi080.stdout:[ 2022-09-09T23:10:00.878 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:00.879 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-09T23:10:00.879 INFO:teuthology.orchestra.run.smithi080.stdout: "last_update": "2022-09-09T23:00:24.570307Z" 2022-09-09T23:10:00.880 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:00.880 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:00.881 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000002627", 2022-09-09T23:10:00.881 INFO:teuthology.orchestra.run.smithi080.stdout: "last_update": "2022-09-09T22:38:18.735605Z" 2022-09-09T23:10:00.882 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:00.882 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:00.883 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000006291", 2022-09-09T23:10:00.883 INFO:teuthology.orchestra.run.smithi080.stdout: "last_update": "2022-09-09T23:00:24.580881Z" 2022-09-09T23:10:00.883 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:00.884 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:00.884 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000006715", 2022-09-09T23:10:00.884 INFO:teuthology.orchestra.run.smithi080.stdout: "last_update": "2022-09-09T23:04:07.178264Z" 2022-09-09T23:10:00.885 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:00.885 INFO:teuthology.orchestra.run.smithi080.stdout:] 2022-09-09T23:10:00.886 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002627', 2: '00000000000000000000:00000000000000006291', 3: '00000000000000000000:00000000000000006715'} 2022-09-09T23:10:00.887 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2022-09-09T23:10:00.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:00.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:00.889 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:10:00.939 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:10:00.940 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:10:01.029 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T23:10:01.029 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T23:10:01.029 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T23:10:01.030 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T23:10:01.030 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T23:10:01.030 INFO:teuthology.orchestra.run.smithi005.stdout: "instance_id": 5471899075723489543 2022-09-09T23:10:01.030 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:01.031 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T23:10:01.031 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:01.031 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T23:10:01.031 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:10:01.032 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:10:01.032 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-09T23:10:01.033 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:10:01.033 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:10:01.034 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:10:01.034 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T23:00:24.570307Z" 2022-09-09T23:10:01.035 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:01.035 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:01.036 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:01.036 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T23:10:01.037 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:10:01.038 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:10:01.038 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000002627", 2022-09-09T23:10:01.039 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:10:01.039 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:10:01.040 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:10:01.040 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:38:18.735605Z" 2022-09-09T23:10:01.040 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:01.041 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:01.042 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:01.042 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T23:10:01.042 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:10:01.043 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:10:01.044 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000006291", 2022-09-09T23:10:01.044 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:10:01.045 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:10:01.045 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:10:01.045 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T23:00:24.580881Z" 2022-09-09T23:10:01.046 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:01.046 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:01.046 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:01.046 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T23:10:01.047 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:10:01.047 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:10:01.047 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000006715", 2022-09-09T23:10:01.047 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:10:01.048 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:10:01.048 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:10:01.048 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T23:04:07.178264Z" 2022-09-09T23:10:01.048 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:01.049 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:01.049 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:10:01.049 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:01.049 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T23:10:01.050 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 0, 2022-09-09T23:10:01.050 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 0 2022-09-09T23:10:01.050 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:01.051 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T23:10:01Z" 2022-09-09T23:10:01.051 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T23:10:01.053 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5471899075723489543 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:00:24.570307Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002627", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:38:18.735605Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006291", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:00:24.580881Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006715", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:04:07.178264Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-09T23:10:01Z" } 2022-09-09T23:10:01.053 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:00:24.570307Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002627', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:38:18.735605Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006291', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:00:24.580881Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006715', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:04:07.178264Z'}}] 2022-09-09T23:10:01.054 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002627', 2: '00000000000000000000:00000000000000006291', 3: '00000000000000000000:00000000000000006715'} 2022-09-09T23:10:01.054 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002627', 2: '00000000000000000000:00000000000000006291', 3: '00000000000000000000:00000000000000006715'} 2022-09-09T23:10:01.055 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2022-09-09T23:10:01.055 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone3 2022-09-09T23:10:01.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:01.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:01.057 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:10:01.104 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:10:01.104 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:10:01.224 INFO:teuthology.orchestra.run.smithi080.stdout:[ 2022-09-09T23:10:01.225 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:01.225 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-09T23:10:01.226 INFO:teuthology.orchestra.run.smithi080.stdout: "last_update": "2022-09-09T23:00:24.570307Z" 2022-09-09T23:10:01.226 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:01.226 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:01.226 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000002627", 2022-09-09T23:10:01.227 INFO:teuthology.orchestra.run.smithi080.stdout: "last_update": "2022-09-09T22:38:18.735605Z" 2022-09-09T23:10:01.227 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:01.227 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:01.227 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000006291", 2022-09-09T23:10:01.228 INFO:teuthology.orchestra.run.smithi080.stdout: "last_update": "2022-09-09T23:00:24.580881Z" 2022-09-09T23:10:01.228 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:01.228 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:01.228 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000006715", 2022-09-09T23:10:01.229 INFO:teuthology.orchestra.run.smithi080.stdout: "last_update": "2022-09-09T23:04:07.178264Z" 2022-09-09T23:10:01.229 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:01.229 INFO:teuthology.orchestra.run.smithi080.stdout:] 2022-09-09T23:10:01.230 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002627', 2: '00000000000000000000:00000000000000006291', 3: '00000000000000000000:00000000000000006715'} 2022-09-09T23:10:01.230 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2022-09-09T23:10:01.231 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:01.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:01.232 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:10:01.284 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:10:01.285 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:10:01.367 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T23:10:01.368 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T23:10:01.369 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T23:10:01.369 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T23:10:01.370 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T23:10:01.370 INFO:teuthology.orchestra.run.smithi005.stdout: "instance_id": 5977856695850153756 2022-09-09T23:10:01.371 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:01.371 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T23:10:01.372 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:01.372 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T23:10:01.373 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:10:01.373 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:10:01.373 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-09T23:10:01.374 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:10:01.374 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:10:01.375 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:10:01.375 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T23:00:24.570307Z" 2022-09-09T23:10:01.375 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:01.375 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:01.376 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:01.376 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T23:10:01.376 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:10:01.377 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:10:01.377 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000002627", 2022-09-09T23:10:01.377 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:10:01.377 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:10:01.378 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:10:01.378 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:38:18.735605Z" 2022-09-09T23:10:01.378 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:01.378 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:01.379 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:01.379 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T23:10:01.379 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:10:01.379 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:10:01.380 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000006291", 2022-09-09T23:10:01.380 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:10:01.380 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:10:01.380 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:10:01.381 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T23:00:24.580881Z" 2022-09-09T23:10:01.381 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:01.381 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:01.381 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:01.382 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T23:10:01.382 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:10:01.382 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:10:01.382 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000006715", 2022-09-09T23:10:01.383 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:10:01.383 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:10:01.383 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:10:01.384 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T23:04:07.178264Z" 2022-09-09T23:10:01.384 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:01.384 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:01.384 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:10:01.385 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:01.385 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T23:10:01.385 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 0, 2022-09-09T23:10:01.385 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 0 2022-09-09T23:10:01.386 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:01.386 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T23:10:01Z" 2022-09-09T23:10:01.386 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T23:10:01.387 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5977856695850153756 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:00:24.570307Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002627", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:38:18.735605Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006291", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:00:24.580881Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006715", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:04:07.178264Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-09T23:10:01Z" } 2022-09-09T23:10:01.388 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:00:24.570307Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002627', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:38:18.735605Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006291', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:00:24.580881Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006715', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:04:07.178264Z'}}] 2022-09-09T23:10:01.389 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002627', 2: '00000000000000000000:00000000000000006291', 3: '00000000000000000000:00000000000000006715'} 2022-09-09T23:10:01.389 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002627', 2: '00000000000000000000:00000000000000006291', 3: '00000000000000000000:00000000000000006715'} 2022-09-09T23:10:01.390 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2022-09-09T23:10:01.390 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone4 2022-09-09T23:10:01.390 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone1 2022-09-09T23:10:01.391 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:01.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:01.392 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:10:01.439 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:10:01.440 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:10:01.538 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T23:10:01.538 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:01.539 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-09T23:10:01.539 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T23:09:58.734191Z" 2022-09-09T23:10:01.539 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:01.539 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:01.540 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000003750", 2022-09-09T23:10:01.540 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T23:09:58.727452Z" 2022-09-09T23:10:01.541 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:01.541 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:01.541 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000006000", 2022-09-09T23:10:01.542 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T23:09:58.729913Z" 2022-09-09T23:10:01.542 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:01.542 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:01.543 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000006847", 2022-09-09T23:10:01.543 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T23:09:58.732233Z" 2022-09-09T23:10:01.543 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:01.543 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T23:10:01.545 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000002060', 1: '00000000000000000000:00000000000000003750', 2: '00000000000000000000:00000000000000006000', 3: '00000000000000000000:00000000000000006847'} 2022-09-09T23:10:01.545 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2022-09-09T23:10:01.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:01.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:01.547 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:10:01.635 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:10:01.636 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:10:01.719 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T23:10:01.720 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T23:10:01.721 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T23:10:01.721 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T23:10:01.721 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T23:10:01.722 INFO:teuthology.orchestra.run.smithi005.stdout: "instance_id": 8378826292454742394 2022-09-09T23:10:01.722 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:01.723 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T23:10:01.723 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:01.723 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T23:10:01.724 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:10:01.724 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:10:01.725 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-09T23:10:01.725 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:10:01.725 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:10:01.726 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:10:01.726 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:57:20.177301Z" 2022-09-09T23:10:01.726 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:01.726 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:01.727 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:01.727 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T23:10:01.727 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:10:01.728 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:10:01.728 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-09T23:10:01.728 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:10:01.728 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:10:01.729 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:10:01.729 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:38:20.103388Z" 2022-09-09T23:10:01.729 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:01.729 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:01.730 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:01.730 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T23:10:01.730 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:10:01.730 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:10:01.731 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000004740", 2022-09-09T23:10:01.731 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:10:01.731 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:10:01.731 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:10:01.732 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T23:00:13.748880Z" 2022-09-09T23:10:01.732 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:01.732 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:01.732 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:01.733 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T23:10:01.733 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:10:01.733 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:10:01.733 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000005587", 2022-09-09T23:10:01.734 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:10:01.734 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:10:01.734 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:10:01.734 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T23:04:45.677379Z" 2022-09-09T23:10:01.735 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:01.735 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:01.735 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:10:01.736 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:01.736 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T23:10:01.737 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 0, 2022-09-09T23:10:01.737 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 0 2022-09-09T23:10:01.738 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:01.738 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T23:10:01Z" 2022-09-09T23:10:01.739 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T23:10:01.740 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 8378826292454742394 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:57:20.177301Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002909", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:38:20.103388Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004740", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:00:13.748880Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005587", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:04:45.677379Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-09T23:10:01Z" } 2022-09-09T23:10:01.741 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:57:20.177301Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:38:20.103388Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004740', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:00:13.748880Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005587', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:04:45.677379Z'}}] 2022-09-09T23:10:01.741 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002060', 1: '00000000000000000000:00000000000000003750', 2: '00000000000000000000:00000000000000006000', 3: '00000000000000000000:00000000000000006847'} 2022-09-09T23:10:01.742 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002909', 2: '00000000000000000000:00000000000000004740', 3: '00000000000000000000:00000000000000005587'} 2022-09-09T23:10:01.742 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone3: shard=0 master=00000000000000000000:00000000000000002060 target=00000000000000000000:00000000000000000796, shard=1 master=00000000000000000000:00000000000000003750 target=00000000000000000000:00000000000000002909, shard=2 master=00000000000000000000:00000000000000006000 target=00000000000000000000:00000000000000004740, shard=3 master=00000000000000000000:00000000000000006847 target=00000000000000000000:00000000000000005587 2022-09-09T23:10:06.744 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:06.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:06.747 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:10:06.796 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:10:06.797 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:10:06.877 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T23:10:06.877 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T23:10:06.877 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T23:10:06.878 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T23:10:06.878 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T23:10:06.878 INFO:teuthology.orchestra.run.smithi005.stdout: "instance_id": 8378826292454742394 2022-09-09T23:10:06.879 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:06.879 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T23:10:06.879 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:06.879 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T23:10:06.880 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:10:06.880 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:10:06.880 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-09T23:10:06.880 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:10:06.881 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:10:06.881 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:10:06.881 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T23:09:58.734191Z" 2022-09-09T23:10:06.881 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:06.882 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:06.882 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:06.882 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T23:10:06.882 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:10:06.883 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:10:06.883 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000003750", 2022-09-09T23:10:06.883 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:10:06.884 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:10:06.884 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:10:06.884 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T23:09:58.727452Z" 2022-09-09T23:10:06.884 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:06.885 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:06.885 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:06.885 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T23:10:06.885 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:10:06.886 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:10:06.886 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000006000", 2022-09-09T23:10:06.886 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:10:06.886 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:10:06.887 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:10:06.887 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T23:09:58.729913Z" 2022-09-09T23:10:06.887 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:06.887 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:06.888 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:06.888 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T23:10:06.888 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:10:06.888 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "incremental-sync", 2022-09-09T23:10:06.889 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000006847", 2022-09-09T23:10:06.889 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:10:06.889 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:10:06.889 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:10:06.890 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T23:09:58.732233Z" 2022-09-09T23:10:06.890 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:06.890 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:06.890 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:10:06.891 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:06.891 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T23:10:06.891 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 0, 2022-09-09T23:10:06.892 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 0 2022-09-09T23:10:06.892 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:06.892 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T23:10:06Z" 2022-09-09T23:10:06.892 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T23:10:06.894 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 8378826292454742394 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002060", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:09:58.734191Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003750", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:09:58.727452Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006000", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:09:58.729913Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006847", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:09:58.732233Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-09T23:10:06Z" } 2022-09-09T23:10:06.894 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002060', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:09:58.734191Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003750', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:09:58.727452Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006000', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:09:58.729913Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006847', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:09:58.732233Z'}}] 2022-09-09T23:10:06.895 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002060', 1: '00000000000000000000:00000000000000003750', 2: '00000000000000000000:00000000000000006000', 3: '00000000000000000000:00000000000000006847'} 2022-09-09T23:10:06.895 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002060', 1: '00000000000000000000:00000000000000003750', 2: '00000000000000000000:00000000000000006000', 3: '00000000000000000000:00000000000000006847'} 2022-09-09T23:10:06.896 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2022-09-09T23:10:06.896 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone2 2022-09-09T23:10:06.897 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:06.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:06.898 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:10:06.944 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:10:06.944 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:10:07.042 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T23:10:07.042 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:07.042 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-09T23:10:07.043 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T23:09:58.734191Z" 2022-09-09T23:10:07.043 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:07.043 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:07.044 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000003750", 2022-09-09T23:10:07.044 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T23:09:58.727452Z" 2022-09-09T23:10:07.044 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:07.044 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:07.045 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000006000", 2022-09-09T23:10:07.045 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T23:09:58.729913Z" 2022-09-09T23:10:07.045 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:07.045 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:07.046 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "00000000000000000000:00000000000000006847", 2022-09-09T23:10:07.046 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T23:09:58.732233Z" 2022-09-09T23:10:07.047 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:07.047 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T23:10:07.049 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000002060', 1: '00000000000000000000:00000000000000003750', 2: '00000000000000000000:00000000000000006000', 3: '00000000000000000000:00000000000000006847'} 2022-09-09T23:10:07.050 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2022-09-09T23:10:07.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:07.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:07.052 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:10:07.101 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:10:07.102 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:10:07.188 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:10:07.188 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:10:07.189 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:10:07.189 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:10:07.189 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:10:07.190 INFO:teuthology.orchestra.run.smithi080.stdout: "instance_id": 11996126732425202593 2022-09-09T23:10:07.190 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:07.190 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:10:07.190 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:07.191 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:10:07.191 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:07.191 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:10:07.191 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-09T23:10:07.192 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:07.192 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:07.192 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:07.192 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:57:20.177301Z" 2022-09-09T23:10:07.193 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:07.193 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:07.193 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:07.193 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:10:07.194 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:07.194 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:10:07.194 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-09T23:10:07.195 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:07.195 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:07.195 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:07.195 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:38:20.103388Z" 2022-09-09T23:10:07.196 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:07.196 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:07.196 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:07.196 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:10:07.196 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:07.197 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:10:07.197 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000004740", 2022-09-09T23:10:07.197 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:07.198 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:07.198 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:07.198 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:00:13.748880Z" 2022-09-09T23:10:07.198 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:07.199 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:07.199 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:07.199 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:10:07.199 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:07.200 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:10:07.200 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000005587", 2022-09-09T23:10:07.200 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:07.200 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:07.201 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:07.201 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:04:45.677379Z" 2022-09-09T23:10:07.201 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:07.201 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:07.201 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:10:07.202 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:07.202 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:10:07.202 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 0, 2022-09-09T23:10:07.203 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 0 2022-09-09T23:10:07.203 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:07.203 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:10:07Z" 2022-09-09T23:10:07.203 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:10:07.205 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11996126732425202593 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:57:20.177301Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002909", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:38:20.103388Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004740", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:00:13.748880Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005587", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:04:45.677379Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-09T23:10:07Z" } 2022-09-09T23:10:07.206 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:57:20.177301Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:38:20.103388Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004740', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:00:13.748880Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005587', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:04:45.677379Z'}}] 2022-09-09T23:10:07.206 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002060', 1: '00000000000000000000:00000000000000003750', 2: '00000000000000000000:00000000000000006000', 3: '00000000000000000000:00000000000000006847'} 2022-09-09T23:10:07.207 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002909', 2: '00000000000000000000:00000000000000004740', 3: '00000000000000000000:00000000000000005587'} 2022-09-09T23:10:07.207 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone3: shard=0 master=00000000000000000000:00000000000000002060 target=00000000000000000000:00000000000000000796, shard=1 master=00000000000000000000:00000000000000003750 target=00000000000000000000:00000000000000002909, shard=2 master=00000000000000000000:00000000000000006000 target=00000000000000000000:00000000000000004740, shard=3 master=00000000000000000000:00000000000000006847 target=00000000000000000000:00000000000000005587 2022-09-09T23:10:12.209 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:12.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:12.211 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:10:12.259 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:10:12.259 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:10:12.349 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:10:12.350 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:10:12.350 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:10:12.351 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:10:12.351 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:10:12.351 INFO:teuthology.orchestra.run.smithi080.stdout: "instance_id": 11996126732425202593 2022-09-09T23:10:12.351 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:12.352 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:10:12.352 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:12.352 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:10:12.352 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:12.353 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:10:12.353 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-09T23:10:12.353 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:12.353 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:12.354 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:12.354 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:57:20.177301Z" 2022-09-09T23:10:12.355 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:12.355 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:12.355 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:12.355 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:10:12.356 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:12.356 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:10:12.356 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-09T23:10:12.356 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:12.357 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:12.357 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:12.357 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:38:20.103388Z" 2022-09-09T23:10:12.357 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:12.358 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:12.358 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:12.358 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:10:12.358 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:12.359 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:10:12.359 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000004740", 2022-09-09T23:10:12.359 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:12.359 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:12.360 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:12.360 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:00:13.748880Z" 2022-09-09T23:10:12.360 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:12.360 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:12.361 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:12.361 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:10:12.362 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:12.362 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:10:12.362 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000005587", 2022-09-09T23:10:12.362 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:12.363 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:12.363 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:12.363 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:04:45.677379Z" 2022-09-09T23:10:12.363 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:12.364 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:12.364 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:10:12.364 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:12.364 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:10:12.365 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 0, 2022-09-09T23:10:12.365 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 0 2022-09-09T23:10:12.365 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:12.365 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:10:12Z" 2022-09-09T23:10:12.366 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:10:12.367 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11996126732425202593 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:57:20.177301Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002909", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:38:20.103388Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004740", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:00:13.748880Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005587", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:04:45.677379Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-09T23:10:12Z" } 2022-09-09T23:10:12.368 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:57:20.177301Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:38:20.103388Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004740', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:00:13.748880Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005587', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:04:45.677379Z'}}] 2022-09-09T23:10:12.368 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002060', 1: '00000000000000000000:00000000000000003750', 2: '00000000000000000000:00000000000000006000', 3: '00000000000000000000:00000000000000006847'} 2022-09-09T23:10:12.369 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000796', 1: '00000000000000000000:00000000000000002909', 2: '00000000000000000000:00000000000000004740', 3: '00000000000000000000:00000000000000005587'} 2022-09-09T23:10:12.369 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone3: shard=0 master=00000000000000000000:00000000000000002060 target=00000000000000000000:00000000000000000796, shard=1 master=00000000000000000000:00000000000000003750 target=00000000000000000000:00000000000000002909, shard=2 master=00000000000000000000:00000000000000006000 target=00000000000000000000:00000000000000004740, shard=3 master=00000000000000000000:00000000000000006847 target=00000000000000000000:00000000000000005587 2022-09-09T23:10:17.379 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:17.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:17.381 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:10:17.430 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:10:17.431 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:10:17.538 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:10:17.538 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:10:17.539 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:10:17.539 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:10:17.539 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:10:17.539 INFO:teuthology.orchestra.run.smithi080.stdout: "instance_id": 11996126732425202593 2022-09-09T23:10:17.540 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:17.540 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:10:17.540 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:17.541 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:10:17.541 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:17.541 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:10:17.542 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-09T23:10:17.542 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:17.542 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:17.542 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:17.543 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:09:58.734191Z" 2022-09-09T23:10:17.543 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:17.543 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:17.544 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:17.544 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:10:17.544 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:17.544 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:10:17.545 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000003750", 2022-09-09T23:10:17.545 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:17.545 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:17.545 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:17.546 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:09:58.727452Z" 2022-09-09T23:10:17.546 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:17.546 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:17.546 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:17.547 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:10:17.547 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:17.548 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:10:17.548 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000006000", 2022-09-09T23:10:17.548 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:17.548 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:17.549 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:17.549 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:09:58.729913Z" 2022-09-09T23:10:17.549 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:17.550 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:17.550 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:17.550 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:10:17.551 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:17.551 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "incremental-sync", 2022-09-09T23:10:17.551 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "00000000000000000000:00000000000000006847", 2022-09-09T23:10:17.551 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:17.552 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:17.552 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:17.552 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:09:58.732233Z" 2022-09-09T23:10:17.553 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:17.553 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:17.553 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:10:17.553 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:17.554 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:10:17.554 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 0, 2022-09-09T23:10:17.554 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 0 2022-09-09T23:10:17.554 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:17.555 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:10:17Z" 2022-09-09T23:10:17.555 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:10:17.556 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11996126732425202593 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002060", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:09:58.734191Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003750", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:09:58.727452Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006000", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:09:58.729913Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006847", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:09:58.732233Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-09T23:10:17Z" } 2022-09-09T23:10:17.557 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002060', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:09:58.734191Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003750', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:09:58.727452Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006000', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:09:58.729913Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006847', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:09:58.732233Z'}}] 2022-09-09T23:10:17.557 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002060', 1: '00000000000000000000:00000000000000003750', 2: '00000000000000000000:00000000000000006000', 3: '00000000000000000000:00000000000000006847'} 2022-09-09T23:10:17.558 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002060', 1: '00000000000000000000:00000000000000003750', 2: '00000000000000000000:00000000000000006000', 3: '00000000000000000000:00000000000000006847'} 2022-09-09T23:10:17.558 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2022-09-09T23:10:17.559 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone4 2022-09-09T23:10:17.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... ok 2022-09-09T23:10:17.566 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jkcgbj-59 2022-09-09T23:10:17.567 DEBUG:boto:path=/jkcgbj-59/ 2022-09-09T23:10:17.568 DEBUG:boto:auth_path=/jkcgbj-59/ 2022-09-09T23:10:17.569 DEBUG:boto:Method: PUT 2022-09-09T23:10:17.570 DEBUG:boto:Path: /jkcgbj-59/ 2022-09-09T23:10:17.572 DEBUG:boto:Data: 2022-09-09T23:10:17.573 DEBUG:boto:Headers: {} 2022-09-09T23:10:17.573 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T23:10:17.574 DEBUG:boto:Port: 8000 2022-09-09T23:10:17.574 DEBUG:boto:Params: {} 2022-09-09T23:10:17.576 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:10:17.576 DEBUG:boto:Token: None 2022-09-09T23:10:17.577 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:10:17 GMT /jkcgbj-59/ 2022-09-09T23:10:17.578 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:tVGveKfwHAWaAu8k7pyJBliiRUY= 2022-09-09T23:10:17.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:10:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:tVGveKfwHAWaAu8k7pyJBliiRUY=', 'Content-Length': '0'} 2022-09-09T23:10:17.615 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000163ad9a68a63f96-00631bc7d9-157e-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:10:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:10:17.616 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jkcgbj-60 2022-09-09T23:10:17.616 DEBUG:boto:path=/jkcgbj-60/ 2022-09-09T23:10:17.617 DEBUG:boto:auth_path=/jkcgbj-60/ 2022-09-09T23:10:17.617 DEBUG:boto:Method: PUT 2022-09-09T23:10:17.619 DEBUG:boto:Path: /jkcgbj-60/ 2022-09-09T23:10:17.620 DEBUG:boto:Data: 2022-09-09T23:10:17.620 DEBUG:boto:Headers: {} 2022-09-09T23:10:17.621 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T23:10:17.621 DEBUG:boto:Port: 8000 2022-09-09T23:10:17.622 DEBUG:boto:Params: {} 2022-09-09T23:10:17.623 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:10:17.623 DEBUG:boto:Token: None 2022-09-09T23:10:17.625 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:10:17 GMT /jkcgbj-60/ 2022-09-09T23:10:17.626 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UTmyLuJ6AO+s4XLB/9upjflNE5k= 2022-09-09T23:10:17.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:10:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UTmyLuJ6AO+s4XLB/9upjflNE5k=', 'Content-Length': '0'} 2022-09-09T23:10:17.699 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000432dfaeb72b2146e-00631bc7d9-16ce-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:10:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:10:17.700 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jkcgbj-61 2022-09-09T23:10:17.701 DEBUG:boto:path=/jkcgbj-61/ 2022-09-09T23:10:17.701 DEBUG:boto:auth_path=/jkcgbj-61/ 2022-09-09T23:10:17.702 DEBUG:boto:Method: PUT 2022-09-09T23:10:17.702 DEBUG:boto:Path: /jkcgbj-61/ 2022-09-09T23:10:17.703 DEBUG:boto:Data: 2022-09-09T23:10:17.703 DEBUG:boto:Headers: {} 2022-09-09T23:10:17.704 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T23:10:17.704 DEBUG:boto:Port: 8001 2022-09-09T23:10:17.705 DEBUG:boto:Params: {} 2022-09-09T23:10:17.706 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-09T23:10:17.707 DEBUG:boto:Token: None 2022-09-09T23:10:17.709 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:10:17 GMT /jkcgbj-61/ 2022-09-09T23:10:17.710 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:PED44PH3vISOGpl8K+og7gjgKmE= 2022-09-09T23:10:17.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:10:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:PED44PH3vISOGpl8K+og7gjgKmE=', 'Content-Length': '0'} 2022-09-09T23:10:17.767 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000065d87dd038f77ed5-00631bc7d9-157c-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:10:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:10:17.768 DEBUG:boto:path=/jkcgbj-59/ 2022-09-09T23:10:17.769 DEBUG:boto:auth_path=/jkcgbj-59/ 2022-09-09T23:10:17.769 DEBUG:boto:Method: HEAD 2022-09-09T23:10:17.770 DEBUG:boto:Path: /jkcgbj-59/ 2022-09-09T23:10:17.770 DEBUG:boto:Data: 2022-09-09T23:10:17.771 DEBUG:boto:Headers: {} 2022-09-09T23:10:17.772 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T23:10:17.772 DEBUG:boto:Port: 8000 2022-09-09T23:10:17.772 DEBUG:boto:Params: {} 2022-09-09T23:10:17.773 DEBUG:boto:Token: None 2022-09-09T23:10:17.774 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 23:10:17 GMT /jkcgbj-59/ 2022-09-09T23:10:17.774 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qM8DRbLj3fLg5hl4SvBbXgsEcnQ= 2022-09-09T23:10:17.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:10:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qM8DRbLj3fLg5hl4SvBbXgsEcnQ=', 'Content-Length': '0'} 2022-09-09T23:10:17.779 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8519298059bf8c8-00631bc7d9-157e-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:10:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:10:17.780 DEBUG:boto:path=/jkcgbj-59/obj1 2022-09-09T23:10:17.781 DEBUG:boto:auth_path=/jkcgbj-59/obj1 2022-09-09T23:10:17.781 DEBUG:boto:Method: PUT 2022-09-09T23:10:17.782 DEBUG:boto:Path: /jkcgbj-59/obj1 2022-09-09T23:10:17.782 DEBUG:boto:Data: 2022-09-09T23:10:17.783 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-09T23:10:17.784 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T23:10:17.784 DEBUG:boto:Port: 8000 2022-09-09T23:10:17.785 DEBUG:boto:Params: {} 2022-09-09T23:10:17.785 DEBUG:boto:Token: None 2022-09-09T23:10:17.786 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 09 Sep 2022 23:10:17 GMT /jkcgbj-59/obj1 2022-09-09T23:10:17.786 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6/CnYnwVz0p5PJdFJxSMVO6SyCA= 2022-09-09T23:10:17.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 23:10:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6/CnYnwVz0p5PJdFJxSMVO6SyCA='} 2022-09-09T23:10:17.802 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e470ad20312d27b4-00631bc7d9-157e-test-zone1'), ('Date', 'Fri, 09 Sep 2022 23:10:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:10:17.803 DEBUG:boto:path=/jkcgbj-59/ 2022-09-09T23:10:17.803 DEBUG:boto:auth_path=/jkcgbj-59/ 2022-09-09T23:10:17.804 DEBUG:boto:Method: HEAD 2022-09-09T23:10:17.804 DEBUG:boto:Path: /jkcgbj-59/ 2022-09-09T23:10:17.805 DEBUG:boto:Data: 2022-09-09T23:10:17.805 DEBUG:boto:Headers: {} 2022-09-09T23:10:17.806 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T23:10:17.807 DEBUG:boto:Port: 8000 2022-09-09T23:10:17.808 DEBUG:boto:Params: {} 2022-09-09T23:10:17.808 DEBUG:boto:Token: None 2022-09-09T23:10:17.809 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 23:10:17 GMT /jkcgbj-59/ 2022-09-09T23:10:17.810 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qM8DRbLj3fLg5hl4SvBbXgsEcnQ= 2022-09-09T23:10:17.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:10:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qM8DRbLj3fLg5hl4SvBbXgsEcnQ=', 'Content-Length': '0'} 2022-09-09T23:10:17.816 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f2462dc3f1ce1d6-00631bc7d9-157e-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:10:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:10:17.816 DEBUG:boto:path=/jkcgbj-59/obj2 2022-09-09T23:10:17.817 DEBUG:boto:auth_path=/jkcgbj-59/obj2 2022-09-09T23:10:17.818 DEBUG:boto:Method: PUT 2022-09-09T23:10:17.818 DEBUG:boto:Path: /jkcgbj-59/obj2 2022-09-09T23:10:17.819 DEBUG:boto:Data: 2022-09-09T23:10:17.819 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-09T23:10:17.820 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T23:10:17.820 DEBUG:boto:Port: 8000 2022-09-09T23:10:17.821 DEBUG:boto:Params: {} 2022-09-09T23:10:17.822 DEBUG:boto:Token: None 2022-09-09T23:10:17.822 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 09 Sep 2022 23:10:17 GMT /jkcgbj-59/obj2 2022-09-09T23:10:17.823 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:oZjdP8LhZahTYdpQ2jbOGMZrvLE= 2022-09-09T23:10:17.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 23:10:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:oZjdP8LhZahTYdpQ2jbOGMZrvLE='} 2022-09-09T23:10:17.866 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ae385475af834a5b-00631bc7d9-157e-test-zone1'), ('Date', 'Fri, 09 Sep 2022 23:10:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:10:17.868 DEBUG:boto:path=/jkcgbj-59/ 2022-09-09T23:10:17.870 DEBUG:boto:auth_path=/jkcgbj-59/ 2022-09-09T23:10:17.871 DEBUG:boto:Method: HEAD 2022-09-09T23:10:17.872 DEBUG:boto:Path: /jkcgbj-59/ 2022-09-09T23:10:17.873 DEBUG:boto:Data: 2022-09-09T23:10:17.873 DEBUG:boto:Headers: {} 2022-09-09T23:10:17.874 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T23:10:17.874 DEBUG:boto:Port: 8000 2022-09-09T23:10:17.875 DEBUG:boto:Params: {} 2022-09-09T23:10:17.876 DEBUG:boto:Token: None 2022-09-09T23:10:17.876 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 23:10:17 GMT /jkcgbj-59/ 2022-09-09T23:10:17.877 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qM8DRbLj3fLg5hl4SvBbXgsEcnQ= 2022-09-09T23:10:17.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:10:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qM8DRbLj3fLg5hl4SvBbXgsEcnQ=', 'Content-Length': '0'} 2022-09-09T23:10:17.883 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000019b330a81fb2a638-00631bc7d9-157e-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:10:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:10:17.884 DEBUG:boto:path=/jkcgbj-59/obj3 2022-09-09T23:10:17.885 DEBUG:boto:auth_path=/jkcgbj-59/obj3 2022-09-09T23:10:17.886 DEBUG:boto:Method: PUT 2022-09-09T23:10:17.886 DEBUG:boto:Path: /jkcgbj-59/obj3 2022-09-09T23:10:17.887 DEBUG:boto:Data: 2022-09-09T23:10:17.887 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-09T23:10:17.888 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T23:10:17.889 DEBUG:boto:Port: 8000 2022-09-09T23:10:17.889 DEBUG:boto:Params: {} 2022-09-09T23:10:17.890 DEBUG:boto:Token: None 2022-09-09T23:10:17.891 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 09 Sep 2022 23:10:17 GMT /jkcgbj-59/obj3 2022-09-09T23:10:17.892 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:QjD1L04C8+iX0tUTStdrqJ68Ip4= 2022-09-09T23:10:17.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 23:10:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:QjD1L04C8+iX0tUTStdrqJ68Ip4='} 2022-09-09T23:10:17.907 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c8a2ed82dea60f28-00631bc7d9-157e-test-zone1'), ('Date', 'Fri, 09 Sep 2022 23:10:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:10:17.908 DEBUG:boto:path=/jkcgbj-59/ 2022-09-09T23:10:17.908 DEBUG:boto:auth_path=/jkcgbj-59/ 2022-09-09T23:10:17.909 DEBUG:boto:Method: HEAD 2022-09-09T23:10:17.910 DEBUG:boto:Path: /jkcgbj-59/ 2022-09-09T23:10:17.910 DEBUG:boto:Data: 2022-09-09T23:10:17.911 DEBUG:boto:Headers: {} 2022-09-09T23:10:17.911 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T23:10:17.912 DEBUG:boto:Port: 8000 2022-09-09T23:10:17.912 DEBUG:boto:Params: {} 2022-09-09T23:10:17.913 DEBUG:boto:Token: None 2022-09-09T23:10:17.913 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 23:10:17 GMT /jkcgbj-59/ 2022-09-09T23:10:17.914 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:qM8DRbLj3fLg5hl4SvBbXgsEcnQ= 2022-09-09T23:10:17.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:10:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:qM8DRbLj3fLg5hl4SvBbXgsEcnQ=', 'Content-Length': '0'} 2022-09-09T23:10:17.922 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d96b35e2b5a9b69-00631bc7d9-157e-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:10:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:10:17.923 DEBUG:boto:path=/jkcgbj-59/obj4 2022-09-09T23:10:17.924 DEBUG:boto:auth_path=/jkcgbj-59/obj4 2022-09-09T23:10:17.924 DEBUG:boto:Method: PUT 2022-09-09T23:10:17.925 DEBUG:boto:Path: /jkcgbj-59/obj4 2022-09-09T23:10:17.926 DEBUG:boto:Data: 2022-09-09T23:10:17.927 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-09T23:10:17.928 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T23:10:17.928 DEBUG:boto:Port: 8000 2022-09-09T23:10:17.930 DEBUG:boto:Params: {} 2022-09-09T23:10:17.931 DEBUG:boto:Token: None 2022-09-09T23:10:17.932 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 09 Sep 2022 23:10:17 GMT /jkcgbj-59/obj4 2022-09-09T23:10:17.933 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:6KArBMOJ9mE8ux3DGYocLQjeyuc= 2022-09-09T23:10:17.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 23:10:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:6KArBMOJ9mE8ux3DGYocLQjeyuc='} 2022-09-09T23:10:17.978 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c7a9f1532da09848-00631bc7d9-157e-test-zone1'), ('Date', 'Fri, 09 Sep 2022 23:10:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:10:17.979 DEBUG:boto:path=/jkcgbj-60/ 2022-09-09T23:10:17.980 DEBUG:boto:auth_path=/jkcgbj-60/ 2022-09-09T23:10:17.980 DEBUG:boto:Method: HEAD 2022-09-09T23:10:17.981 DEBUG:boto:Path: /jkcgbj-60/ 2022-09-09T23:10:17.981 DEBUG:boto:Data: 2022-09-09T23:10:17.982 DEBUG:boto:Headers: {} 2022-09-09T23:10:17.983 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T23:10:17.983 DEBUG:boto:Port: 8000 2022-09-09T23:10:17.984 DEBUG:boto:Params: {} 2022-09-09T23:10:17.985 DEBUG:boto:Token: None 2022-09-09T23:10:17.985 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 23:10:17 GMT /jkcgbj-60/ 2022-09-09T23:10:17.986 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:aIUixMu5Gu8uP08909HXkMiX/TI= 2022-09-09T23:10:17.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:10:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:aIUixMu5Gu8uP08909HXkMiX/TI=', 'Content-Length': '0'} 2022-09-09T23:10:17.991 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bfc26d72ec384e20-00631bc7d9-16ce-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:10:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:10:17.992 DEBUG:boto:path=/jkcgbj-60/obj1 2022-09-09T23:10:17.993 DEBUG:boto:auth_path=/jkcgbj-60/obj1 2022-09-09T23:10:17.993 DEBUG:boto:Method: PUT 2022-09-09T23:10:17.994 DEBUG:boto:Path: /jkcgbj-60/obj1 2022-09-09T23:10:17.994 DEBUG:boto:Data: 2022-09-09T23:10:17.995 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-09T23:10:17.996 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T23:10:17.996 DEBUG:boto:Port: 8000 2022-09-09T23:10:17.997 DEBUG:boto:Params: {} 2022-09-09T23:10:17.997 DEBUG:boto:Token: None 2022-09-09T23:10:17.999 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 09 Sep 2022 23:10:17 GMT /jkcgbj-60/obj1 2022-09-09T23:10:18.000 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:+e3CtEWO/ayFUYaAk4U9/ecXGW8= 2022-09-09T23:10:18.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 23:10:17 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:+e3CtEWO/ayFUYaAk4U9/ecXGW8='} 2022-09-09T23:10:18.017 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e12628251b9642c6-00631bc7d9-16ce-test-zone2'), ('Date', 'Fri, 09 Sep 2022 23:10:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:10:18.018 DEBUG:boto:path=/jkcgbj-60/ 2022-09-09T23:10:18.018 DEBUG:boto:auth_path=/jkcgbj-60/ 2022-09-09T23:10:18.019 DEBUG:boto:Method: HEAD 2022-09-09T23:10:18.019 DEBUG:boto:Path: /jkcgbj-60/ 2022-09-09T23:10:18.020 DEBUG:boto:Data: 2022-09-09T23:10:18.021 DEBUG:boto:Headers: {} 2022-09-09T23:10:18.021 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T23:10:18.021 DEBUG:boto:Port: 8000 2022-09-09T23:10:18.022 DEBUG:boto:Params: {} 2022-09-09T23:10:18.023 DEBUG:boto:Token: None 2022-09-09T23:10:18.023 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 23:10:18 GMT /jkcgbj-60/ 2022-09-09T23:10:18.024 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:uu/Pv1OJCbFfGZ/VAyc6I+3lC3g= 2022-09-09T23:10:18.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:10:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:uu/Pv1OJCbFfGZ/VAyc6I+3lC3g=', 'Content-Length': '0'} 2022-09-09T23:10:18.029 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d1f545e133c3558-00631bc7da-16ce-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:10:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:10:18.030 DEBUG:boto:path=/jkcgbj-60/obj2 2022-09-09T23:10:18.031 DEBUG:boto:auth_path=/jkcgbj-60/obj2 2022-09-09T23:10:18.031 DEBUG:boto:Method: PUT 2022-09-09T23:10:18.032 DEBUG:boto:Path: /jkcgbj-60/obj2 2022-09-09T23:10:18.032 DEBUG:boto:Data: 2022-09-09T23:10:18.033 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-09T23:10:18.033 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T23:10:18.034 DEBUG:boto:Port: 8000 2022-09-09T23:10:18.034 DEBUG:boto:Params: {} 2022-09-09T23:10:18.035 DEBUG:boto:Token: None 2022-09-09T23:10:18.035 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 09 Sep 2022 23:10:18 GMT /jkcgbj-60/obj2 2022-09-09T23:10:18.036 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:OE6h9qHjZ1oLec0LmHCgIj8SzAM= 2022-09-09T23:10:18.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 23:10:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:OE6h9qHjZ1oLec0LmHCgIj8SzAM='} 2022-09-09T23:10:18.086 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c373d7f2c700dc91-00631bc7da-16ce-test-zone2'), ('Date', 'Fri, 09 Sep 2022 23:10:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:10:18.087 DEBUG:boto:path=/jkcgbj-60/ 2022-09-09T23:10:18.088 DEBUG:boto:auth_path=/jkcgbj-60/ 2022-09-09T23:10:18.088 DEBUG:boto:Method: HEAD 2022-09-09T23:10:18.089 DEBUG:boto:Path: /jkcgbj-60/ 2022-09-09T23:10:18.090 DEBUG:boto:Data: 2022-09-09T23:10:18.090 DEBUG:boto:Headers: {} 2022-09-09T23:10:18.091 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T23:10:18.091 DEBUG:boto:Port: 8000 2022-09-09T23:10:18.092 DEBUG:boto:Params: {} 2022-09-09T23:10:18.092 DEBUG:boto:Token: None 2022-09-09T23:10:18.093 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 23:10:18 GMT /jkcgbj-60/ 2022-09-09T23:10:18.093 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:uu/Pv1OJCbFfGZ/VAyc6I+3lC3g= 2022-09-09T23:10:18.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:10:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:uu/Pv1OJCbFfGZ/VAyc6I+3lC3g=', 'Content-Length': '0'} 2022-09-09T23:10:18.099 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a85b19ed489c7f62-00631bc7da-16ce-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:10:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:10:18.100 DEBUG:boto:path=/jkcgbj-60/obj3 2022-09-09T23:10:18.101 DEBUG:boto:auth_path=/jkcgbj-60/obj3 2022-09-09T23:10:18.101 DEBUG:boto:Method: PUT 2022-09-09T23:10:18.102 DEBUG:boto:Path: /jkcgbj-60/obj3 2022-09-09T23:10:18.102 DEBUG:boto:Data: 2022-09-09T23:10:18.103 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-09T23:10:18.103 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T23:10:18.104 DEBUG:boto:Port: 8000 2022-09-09T23:10:18.104 DEBUG:boto:Params: {} 2022-09-09T23:10:18.105 DEBUG:boto:Token: None 2022-09-09T23:10:18.105 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 09 Sep 2022 23:10:18 GMT /jkcgbj-60/obj3 2022-09-09T23:10:18.106 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Hdp2xDAX0T839O40cpxTVsUyM3A= 2022-09-09T23:10:18.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 23:10:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Hdp2xDAX0T839O40cpxTVsUyM3A='} 2022-09-09T23:10:18.121 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a8c71ce12d583659-00631bc7da-16ce-test-zone2'), ('Date', 'Fri, 09 Sep 2022 23:10:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:10:18.122 DEBUG:boto:path=/jkcgbj-60/ 2022-09-09T23:10:18.123 DEBUG:boto:auth_path=/jkcgbj-60/ 2022-09-09T23:10:18.123 DEBUG:boto:Method: HEAD 2022-09-09T23:10:18.124 DEBUG:boto:Path: /jkcgbj-60/ 2022-09-09T23:10:18.124 DEBUG:boto:Data: 2022-09-09T23:10:18.125 DEBUG:boto:Headers: {} 2022-09-09T23:10:18.125 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T23:10:18.126 DEBUG:boto:Port: 8000 2022-09-09T23:10:18.126 DEBUG:boto:Params: {} 2022-09-09T23:10:18.127 DEBUG:boto:Token: None 2022-09-09T23:10:18.128 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 23:10:18 GMT /jkcgbj-60/ 2022-09-09T23:10:18.128 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:uu/Pv1OJCbFfGZ/VAyc6I+3lC3g= 2022-09-09T23:10:18.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:10:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:uu/Pv1OJCbFfGZ/VAyc6I+3lC3g=', 'Content-Length': '0'} 2022-09-09T23:10:18.133 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000003ad3383316d3249-00631bc7da-16ce-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:10:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:10:18.134 DEBUG:boto:path=/jkcgbj-60/obj4 2022-09-09T23:10:18.135 DEBUG:boto:auth_path=/jkcgbj-60/obj4 2022-09-09T23:10:18.136 DEBUG:boto:Method: PUT 2022-09-09T23:10:18.137 DEBUG:boto:Path: /jkcgbj-60/obj4 2022-09-09T23:10:18.138 DEBUG:boto:Data: 2022-09-09T23:10:18.139 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-09T23:10:18.140 DEBUG:boto:Host: smithi080.front.sepia.ceph.com:8000 2022-09-09T23:10:18.141 DEBUG:boto:Port: 8000 2022-09-09T23:10:18.142 DEBUG:boto:Params: {} 2022-09-09T23:10:18.143 DEBUG:boto:Token: None 2022-09-09T23:10:18.144 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 09 Sep 2022 23:10:18 GMT /jkcgbj-60/obj4 2022-09-09T23:10:18.145 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:54e5upk6m/fbMAOgRu7pFR3HUak= 2022-09-09T23:10:18.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 23:10:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:54e5upk6m/fbMAOgRu7pFR3HUak='} 2022-09-09T23:10:18.190 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005c9efe17a1fd208a-00631bc7da-16ce-test-zone2'), ('Date', 'Fri, 09 Sep 2022 23:10:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:10:18.192 DEBUG:boto:path=/jkcgbj-61/ 2022-09-09T23:10:18.193 DEBUG:boto:auth_path=/jkcgbj-61/ 2022-09-09T23:10:18.194 DEBUG:boto:Method: HEAD 2022-09-09T23:10:18.195 DEBUG:boto:Path: /jkcgbj-61/ 2022-09-09T23:10:18.195 DEBUG:boto:Data: 2022-09-09T23:10:18.196 DEBUG:boto:Headers: {} 2022-09-09T23:10:18.197 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T23:10:18.198 DEBUG:boto:Port: 8001 2022-09-09T23:10:18.199 DEBUG:boto:Params: {} 2022-09-09T23:10:18.200 DEBUG:boto:Token: None 2022-09-09T23:10:18.201 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 23:10:18 GMT /jkcgbj-61/ 2022-09-09T23:10:18.202 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UbjMciyxikQs5fENj23pYSC9CiA= 2022-09-09T23:10:18.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:10:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UbjMciyxikQs5fENj23pYSC9CiA=', 'Content-Length': '0'} 2022-09-09T23:10:18.209 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000889f84b69f0817ef-00631bc7da-157c-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:10:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:10:18.210 DEBUG:boto:path=/jkcgbj-61/obj1 2022-09-09T23:10:18.211 DEBUG:boto:auth_path=/jkcgbj-61/obj1 2022-09-09T23:10:18.212 DEBUG:boto:Method: PUT 2022-09-09T23:10:18.213 DEBUG:boto:Path: /jkcgbj-61/obj1 2022-09-09T23:10:18.214 DEBUG:boto:Data: 2022-09-09T23:10:18.215 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-09T23:10:18.216 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T23:10:18.217 DEBUG:boto:Port: 8001 2022-09-09T23:10:18.217 DEBUG:boto:Params: {} 2022-09-09T23:10:18.219 DEBUG:boto:Token: None 2022-09-09T23:10:18.220 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 09 Sep 2022 23:10:18 GMT /jkcgbj-61/obj1 2022-09-09T23:10:18.221 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9crCeWG9TeN0KzX2lfgkEkavkPY= 2022-09-09T23:10:18.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 23:10:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9crCeWG9TeN0KzX2lfgkEkavkPY='} 2022-09-09T23:10:18.240 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dbf69ab646778e5e-00631bc7da-157c-test-zone3'), ('Date', 'Fri, 09 Sep 2022 23:10:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:10:18.241 DEBUG:boto:path=/jkcgbj-61/ 2022-09-09T23:10:18.242 DEBUG:boto:auth_path=/jkcgbj-61/ 2022-09-09T23:10:18.242 DEBUG:boto:Method: HEAD 2022-09-09T23:10:18.243 DEBUG:boto:Path: /jkcgbj-61/ 2022-09-09T23:10:18.244 DEBUG:boto:Data: 2022-09-09T23:10:18.244 DEBUG:boto:Headers: {} 2022-09-09T23:10:18.244 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T23:10:18.245 DEBUG:boto:Port: 8001 2022-09-09T23:10:18.245 DEBUG:boto:Params: {} 2022-09-09T23:10:18.246 DEBUG:boto:Token: None 2022-09-09T23:10:18.247 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 23:10:18 GMT /jkcgbj-61/ 2022-09-09T23:10:18.248 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UbjMciyxikQs5fENj23pYSC9CiA= 2022-09-09T23:10:18.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:10:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UbjMciyxikQs5fENj23pYSC9CiA=', 'Content-Length': '0'} 2022-09-09T23:10:18.253 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3c4e0f2f608c8f3-00631bc7da-157c-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:10:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:10:18.254 DEBUG:boto:path=/jkcgbj-61/obj2 2022-09-09T23:10:18.254 DEBUG:boto:auth_path=/jkcgbj-61/obj2 2022-09-09T23:10:18.255 DEBUG:boto:Method: PUT 2022-09-09T23:10:18.255 DEBUG:boto:Path: /jkcgbj-61/obj2 2022-09-09T23:10:18.256 DEBUG:boto:Data: 2022-09-09T23:10:18.256 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-09T23:10:18.257 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T23:10:18.257 DEBUG:boto:Port: 8001 2022-09-09T23:10:18.258 DEBUG:boto:Params: {} 2022-09-09T23:10:18.259 DEBUG:boto:Token: None 2022-09-09T23:10:18.260 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 09 Sep 2022 23:10:18 GMT /jkcgbj-61/obj2 2022-09-09T23:10:18.261 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:2f6MaHLYWbSXOKfmLL4C1Rqc+nE= 2022-09-09T23:10:18.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 23:10:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:2f6MaHLYWbSXOKfmLL4C1Rqc+nE='} 2022-09-09T23:10:18.306 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000603e8d0e54449b02-00631bc7da-157c-test-zone3'), ('Date', 'Fri, 09 Sep 2022 23:10:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:10:18.308 DEBUG:boto:path=/jkcgbj-61/ 2022-09-09T23:10:18.309 DEBUG:boto:auth_path=/jkcgbj-61/ 2022-09-09T23:10:18.310 DEBUG:boto:Method: HEAD 2022-09-09T23:10:18.310 DEBUG:boto:Path: /jkcgbj-61/ 2022-09-09T23:10:18.311 DEBUG:boto:Data: 2022-09-09T23:10:18.312 DEBUG:boto:Headers: {} 2022-09-09T23:10:18.313 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T23:10:18.314 DEBUG:boto:Port: 8001 2022-09-09T23:10:18.315 DEBUG:boto:Params: {} 2022-09-09T23:10:18.316 DEBUG:boto:Token: None 2022-09-09T23:10:18.317 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 23:10:18 GMT /jkcgbj-61/ 2022-09-09T23:10:18.319 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UbjMciyxikQs5fENj23pYSC9CiA= 2022-09-09T23:10:18.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:10:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UbjMciyxikQs5fENj23pYSC9CiA=', 'Content-Length': '0'} 2022-09-09T23:10:18.326 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000754f576082a763f7-00631bc7da-157c-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:10:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:10:18.328 DEBUG:boto:path=/jkcgbj-61/obj3 2022-09-09T23:10:18.329 DEBUG:boto:auth_path=/jkcgbj-61/obj3 2022-09-09T23:10:18.329 DEBUG:boto:Method: PUT 2022-09-09T23:10:18.330 DEBUG:boto:Path: /jkcgbj-61/obj3 2022-09-09T23:10:18.331 DEBUG:boto:Data: 2022-09-09T23:10:18.332 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-09T23:10:18.332 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T23:10:18.333 DEBUG:boto:Port: 8001 2022-09-09T23:10:18.333 DEBUG:boto:Params: {} 2022-09-09T23:10:18.334 DEBUG:boto:Token: None 2022-09-09T23:10:18.334 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 09 Sep 2022 23:10:18 GMT /jkcgbj-61/obj3 2022-09-09T23:10:18.335 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:5HaoFHfFiOA2uqFHOYLUujRE4zc= 2022-09-09T23:10:18.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 23:10:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:5HaoFHfFiOA2uqFHOYLUujRE4zc='} 2022-09-09T23:10:18.352 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000079d4448f56730530-00631bc7da-157c-test-zone3'), ('Date', 'Fri, 09 Sep 2022 23:10:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:10:18.353 DEBUG:boto:path=/jkcgbj-61/ 2022-09-09T23:10:18.354 DEBUG:boto:auth_path=/jkcgbj-61/ 2022-09-09T23:10:18.355 DEBUG:boto:Method: HEAD 2022-09-09T23:10:18.356 DEBUG:boto:Path: /jkcgbj-61/ 2022-09-09T23:10:18.357 DEBUG:boto:Data: 2022-09-09T23:10:18.358 DEBUG:boto:Headers: {} 2022-09-09T23:10:18.359 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T23:10:18.360 DEBUG:boto:Port: 8001 2022-09-09T23:10:18.362 DEBUG:boto:Params: {} 2022-09-09T23:10:18.362 DEBUG:boto:Token: None 2022-09-09T23:10:18.363 DEBUG:boto:StringToSign: HEAD Fri, 09 Sep 2022 23:10:18 GMT /jkcgbj-61/ 2022-09-09T23:10:18.363 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UbjMciyxikQs5fENj23pYSC9CiA= 2022-09-09T23:10:18.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:10:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UbjMciyxikQs5fENj23pYSC9CiA=', 'Content-Length': '0'} 2022-09-09T23:10:18.369 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c16e35004efda8e5-00631bc7da-157c-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 09 Sep 2022 23:10:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:10:18.370 DEBUG:boto:path=/jkcgbj-61/obj4 2022-09-09T23:10:18.370 DEBUG:boto:auth_path=/jkcgbj-61/obj4 2022-09-09T23:10:18.371 DEBUG:boto:Method: PUT 2022-09-09T23:10:18.372 DEBUG:boto:Path: /jkcgbj-61/obj4 2022-09-09T23:10:18.373 DEBUG:boto:Data: 2022-09-09T23:10:18.373 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-09T23:10:18.374 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8001 2022-09-09T23:10:18.375 DEBUG:boto:Port: 8001 2022-09-09T23:10:18.375 DEBUG:boto:Params: {} 2022-09-09T23:10:18.376 DEBUG:boto:Token: None 2022-09-09T23:10:18.376 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 09 Sep 2022 23:10:18 GMT /jkcgbj-61/obj4 2022-09-09T23:10:18.377 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:Vr+vcjv87FudomSCunLFlCeWF4M= 2022-09-09T23:10:18.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 09 Sep 2022 23:10:18 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:Vr+vcjv87FudomSCunLFlCeWF4M='} 2022-09-09T23:10:18.427 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e5f3bc1f31bbfbf7-00631bc7da-157c-test-zone3'), ('Date', 'Fri, 09 Sep 2022 23:10:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-09T23:10:18.428 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:18.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:18.429 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:10:18.476 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:10:18.476 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:10:18.533 INFO:teuthology.orchestra.run.smithi005.stderr:No --period given, using current period=fd4bc158-06fd-4840-8a27-33fabda67b99 2022-09-09T23:10:18.561 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2022-09-09T23:10:18.561 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:18.562 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T23:10:18.562 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T23:10:18.562 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:18.563 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:18.563 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T23:10:18.563 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T23:10:18.563 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:18.564 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:18.564 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T23:10:18.564 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "0.000000" 2022-09-09T23:10:18.564 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:18.565 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:18.565 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662765017.749301_664.1", 2022-09-09T23:10:18.565 INFO:teuthology.orchestra.run.smithi005.stdout: "last_update": "2022-09-09T23:10:17.749301Z" 2022-09-09T23:10:18.565 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:18.566 INFO:teuthology.orchestra.run.smithi005.stdout:] 2022-09-09T23:10:18.567 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:10:18.568 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-09T23:10:18.569 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:18.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:18.571 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:10:18.614 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:10:18.615 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:10:18.675 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:10:18.671+0000 7f4eba649dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:10:18.677 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:10:18.675+0000 7f4eba649dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:10:18.703 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:10:18.704 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:10:18.704 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:10:18.704 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:10:18.704 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:10:18.705 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:10:18.705 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:10:18.705 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:18.706 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:10:18.706 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:18.706 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:10:18.706 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:18.707 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:18.707 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:10:18.707 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:18.707 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:10:18.707 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:18.708 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:10:18.708 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:10:18.708 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:18.709 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:18.709 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:18.709 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:10:18.710 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:18.710 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:18.711 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:10:18.711 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:18.712 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:18.712 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:18.713 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:10:18.713 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:10:18.714 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:18.714 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:18.715 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:18.715 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:10:18.715 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:18.716 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:18.716 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:10:18.716 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:18.716 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:18.717 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:18.717 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:10:18.717 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:10:18.717 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:18.718 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:18.718 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:18.718 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:10:18.719 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:18.719 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:18.719 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.749301_664.1", 2022-09-09T23:10:18.719 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:18.720 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:18.720 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:18.721 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.749301Z", 2022-09-09T23:10:18.721 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:10:18.721 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:18.722 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:18.722 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:10:18.723 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:18.723 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:10:18.724 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:10:18.724 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:10:18.725 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:18.725 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:10:18Z" 2022-09-09T23:10:18.725 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:10:18.727 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.749301_664.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.749301Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:10:18Z" } 2022-09-09T23:10:18.728 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.749301_664.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.749301Z', 'realm_epoch': 4}}] 2022-09-09T23:10:18.729 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:10:18.730 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:10:18.730 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-09T23:10:18.731 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-09T23:10:18.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:18.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:18.732 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:10:18.780 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:10:18.781 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:10:18.833 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T23:10:18.831+0000 7f455ba28dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:10:18.835 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T23:10:18.831+0000 7f455ba28dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:10:18.860 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2022-09-09T23:10:18.861 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": { 2022-09-09T23:10:18.861 INFO:teuthology.orchestra.run.smithi005.stdout: "info": { 2022-09-09T23:10:19.172 INFO:teuthology.orchestra.run.smithi005.stdout: "status": "sync", 2022-09-09T23:10:19.173 INFO:teuthology.orchestra.run.smithi005.stdout: "num_shards": 4, 2022-09-09T23:10:19.173 INFO:teuthology.orchestra.run.smithi005.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:10:19.173 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 4 2022-09-09T23:10:19.174 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:19.174 INFO:teuthology.orchestra.run.smithi005.stdout: "markers": [ 2022-09-09T23:10:19.174 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:19.175 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 0, 2022-09-09T23:10:19.175 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:10:19.175 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T23:10:19.176 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:10:19.176 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:10:19.176 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 1, 2022-09-09T23:10:19.177 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:10:19.177 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:10:19.177 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T23:10:19.178 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:19.178 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:19.178 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:19.179 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 1, 2022-09-09T23:10:19.179 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:10:19.179 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T23:10:19.180 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "", 2022-09-09T23:10:19.180 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:10:19.180 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:10:19.181 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:10:19.181 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "0.000000", 2022-09-09T23:10:19.181 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T23:10:19.182 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:19.182 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:19.182 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:19.183 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 2, 2022-09-09T23:10:19.183 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:10:19.183 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T23:10:19.184 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:10:19.184 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:10:19.184 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:10:19.185 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:10:19.185 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:10:19.185 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2022-09-09T23:10:19.186 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:19.186 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:19.186 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-09-09T23:10:19.187 INFO:teuthology.orchestra.run.smithi005.stdout: "key": 3, 2022-09-09T23:10:19.187 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2022-09-09T23:10:19.187 INFO:teuthology.orchestra.run.smithi005.stdout: "state": 1, 2022-09-09T23:10:19.188 INFO:teuthology.orchestra.run.smithi005.stdout: "marker": "1_1662765017.749301_664.1", 2022-09-09T23:10:19.188 INFO:teuthology.orchestra.run.smithi005.stdout: "next_step_marker": "", 2022-09-09T23:10:19.188 INFO:teuthology.orchestra.run.smithi005.stdout: "total_entries": 0, 2022-09-09T23:10:19.188 INFO:teuthology.orchestra.run.smithi005.stdout: "pos": 0, 2022-09-09T23:10:19.189 INFO:teuthology.orchestra.run.smithi005.stdout: "timestamp": "2022-09-09T23:10:17.749301Z", 2022-09-09T23:10:19.189 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 4 2022-09-09T23:10:19.189 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:19.189 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-09-09T23:10:19.190 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-09-09T23:10:19.190 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:19.190 INFO:teuthology.orchestra.run.smithi005.stdout: "full_sync": { 2022-09-09T23:10:19.191 INFO:teuthology.orchestra.run.smithi005.stdout: "total": 1, 2022-09-09T23:10:19.191 INFO:teuthology.orchestra.run.smithi005.stdout: "complete": 1 2022-09-09T23:10:19.191 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-09-09T23:10:19.192 INFO:teuthology.orchestra.run.smithi005.stdout: "current_time": "2022-09-09T23:10:18Z" 2022-09-09T23:10:19.192 INFO:teuthology.orchestra.run.smithi005.stdout:} 2022-09-09T23:10:19.194 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.749301_664.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.749301Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:10:18Z" } 2022-09-09T23:10:19.195 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.749301_664.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.749301Z', 'realm_epoch': 4}}] 2022-09-09T23:10:19.195 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:10:19.196 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:10:19.196 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-09T23:10:19.197 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-09T23:10:19.197 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:19.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:19.199 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:10:19.247 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:10:19.247 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:10:19.304 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:10:19.299+0000 7fc41037fdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:10:19.308 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:10:19.303+0000 7fc41037fdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:10:19.338 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:10:19.338 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:10:19.339 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:10:19.339 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:10:19.341 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:10:19.341 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:10:19.342 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:10:19.342 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:19.343 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:10:19.343 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:19.344 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:10:19.344 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:19.345 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:19.345 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:10:19.346 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:19.346 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:10:19.348 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:19.348 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:10:19.349 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:10:19.350 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:19.350 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:19.351 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:19.351 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:10:19.352 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:19.352 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:19.353 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:10:19.353 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:19.353 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:19.354 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:19.354 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:10:19.355 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:10:19.355 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:19.356 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:19.356 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:19.357 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:10:19.357 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:19.358 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:19.358 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:10:19.359 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:19.359 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:19.360 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:19.361 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:10:19.361 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:10:19.361 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:19.362 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:19.362 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:19.362 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:10:19.363 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:19.363 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:19.363 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:10:19.363 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:19.364 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:19.364 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:19.364 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:10:19.364 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:10:19.364 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:19.365 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:19.366 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:10:19.366 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:19.367 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:10:19.367 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:10:19.367 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:10:19.367 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:19.368 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:10:19Z" 2022-09-09T23:10:19.368 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:10:19.369 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:10:19Z" } 2022-09-09T23:10:19.370 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:10:19.371 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:10:19.372 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:10:19.373 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:10:20.236 INFO:tasks.rgw.c1.client.0.smithi005.stdout:*** Caught signal (Segmentation fault) ** 2022-09-09T23:10:20.237 INFO:tasks.rgw.c1.client.0.smithi005.stdout: in thread 7f1f731f8700 thread_name:lua_background 2022-09-09T23:10:20.243 INFO:tasks.rgw.c1.client.0.smithi005.stdout: ceph version 17.0.0-14894-g28758235 (28758235ba2ab1c1d8afa0f88d00f07b5687a51e) quincy (dev) 2022-09-09T23:10:20.243 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f20a7d81420] 2022-09-09T23:10:20.244 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 2: (rgw::sal::RadosLuaManager::get_script(DoutPrefixProvider const*, optional_yield, std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator >&)+0x178) [0x55bbd4e34c88] 2022-09-09T23:10:20.244 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 3: (rgw::lua::read_script(DoutPrefixProvider const*, rgw::sal::LuaManager*, std::__cxx11::basic_string, std::allocator > const&, optional_yield, rgw::lua::context, std::__cxx11::basic_string, std::allocator >&)+0x75) [0x55bbd4f518f5] 2022-09-09T23:10:20.245 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 4: (rgw::lua::Background::read_script()+0x9f) [0x55bbd4f5cd0f] 2022-09-09T23:10:20.245 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 5: (rgw::lua::Background::run()+0x6e4) [0x55bbd4f5dce4] 2022-09-09T23:10:20.245 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 6: /usr/lib/ceph/libceph-common.so.2(+0xa81eb4) [0x7f20a90c4eb4] 2022-09-09T23:10:20.246 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 7: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f20a7d75609] 2022-09-09T23:10:20.246 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 8: clone() 2022-09-09T23:10:20.246 INFO:tasks.rgw.c1.client.0.smithi005.stdout:2022-09-09T23:10:20.239+0000 7f1f731f8700 -1 *** Caught signal (Segmentation fault) ** 2022-09-09T23:10:20.247 INFO:tasks.rgw.c1.client.0.smithi005.stdout: in thread 7f1f731f8700 thread_name:lua_background 2022-09-09T23:10:20.247 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 2022-09-09T23:10:20.248 INFO:tasks.rgw.c1.client.0.smithi005.stdout: ceph version 17.0.0-14894-g28758235 (28758235ba2ab1c1d8afa0f88d00f07b5687a51e) quincy (dev) 2022-09-09T23:10:20.248 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f20a7d81420] 2022-09-09T23:10:20.248 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 2: (rgw::sal::RadosLuaManager::get_script(DoutPrefixProvider const*, optional_yield, std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator >&)+0x178) [0x55bbd4e34c88] 2022-09-09T23:10:20.249 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 3: (rgw::lua::read_script(DoutPrefixProvider const*, rgw::sal::LuaManager*, std::__cxx11::basic_string, std::allocator > const&, optional_yield, rgw::lua::context, std::__cxx11::basic_string, std::allocator >&)+0x75) [0x55bbd4f518f5] 2022-09-09T23:10:20.249 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 4: (rgw::lua::Background::read_script()+0x9f) [0x55bbd4f5cd0f] 2022-09-09T23:10:20.249 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 5: (rgw::lua::Background::run()+0x6e4) [0x55bbd4f5dce4] 2022-09-09T23:10:20.250 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 6: /usr/lib/ceph/libceph-common.so.2(+0xa81eb4) [0x7f20a90c4eb4] 2022-09-09T23:10:20.250 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 7: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f20a7d75609] 2022-09-09T23:10:20.251 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 8: clone() 2022-09-09T23:10:20.252 INFO:tasks.rgw.c1.client.0.smithi005.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-09-09T23:10:20.252 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 2022-09-09T23:10:20.260 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 0> 2022-09-09T23:10:20.239+0000 7f1f731f8700 -1 *** Caught signal (Segmentation fault) ** 2022-09-09T23:10:20.261 INFO:tasks.rgw.c1.client.0.smithi005.stdout: in thread 7f1f731f8700 thread_name:lua_background 2022-09-09T23:10:20.262 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 2022-09-09T23:10:20.262 INFO:tasks.rgw.c1.client.0.smithi005.stdout: ceph version 17.0.0-14894-g28758235 (28758235ba2ab1c1d8afa0f88d00f07b5687a51e) quincy (dev) 2022-09-09T23:10:20.262 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f20a7d81420] 2022-09-09T23:10:20.263 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 2: (rgw::sal::RadosLuaManager::get_script(DoutPrefixProvider const*, optional_yield, std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator >&)+0x178) [0x55bbd4e34c88] 2022-09-09T23:10:20.263 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 3: (rgw::lua::read_script(DoutPrefixProvider const*, rgw::sal::LuaManager*, std::__cxx11::basic_string, std::allocator > const&, optional_yield, rgw::lua::context, std::__cxx11::basic_string, std::allocator >&)+0x75) [0x55bbd4f518f5] 2022-09-09T23:10:20.264 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 4: (rgw::lua::Background::read_script()+0x9f) [0x55bbd4f5cd0f] 2022-09-09T23:10:20.264 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 5: (rgw::lua::Background::run()+0x6e4) [0x55bbd4f5dce4] 2022-09-09T23:10:20.264 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 6: /usr/lib/ceph/libceph-common.so.2(+0xa81eb4) [0x7f20a90c4eb4] 2022-09-09T23:10:20.265 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 7: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f20a7d75609] 2022-09-09T23:10:20.265 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 8: clone() 2022-09-09T23:10:20.265 INFO:tasks.rgw.c1.client.0.smithi005.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-09-09T23:10:20.266 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 2022-09-09T23:10:20.273 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 0> 2022-09-09T23:10:20.239+0000 7f1f731f8700 -1 *** Caught signal (Segmentation fault) ** 2022-09-09T23:10:20.274 INFO:tasks.rgw.c1.client.0.smithi005.stdout: in thread 7f1f731f8700 thread_name:lua_background 2022-09-09T23:10:20.275 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 2022-09-09T23:10:20.275 INFO:tasks.rgw.c1.client.0.smithi005.stdout: ceph version 17.0.0-14894-g28758235 (28758235ba2ab1c1d8afa0f88d00f07b5687a51e) quincy (dev) 2022-09-09T23:10:20.276 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f20a7d81420] 2022-09-09T23:10:20.276 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 2: (rgw::sal::RadosLuaManager::get_script(DoutPrefixProvider const*, optional_yield, std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator >&)+0x178) [0x55bbd4e34c88] 2022-09-09T23:10:20.277 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 3: (rgw::lua::read_script(DoutPrefixProvider const*, rgw::sal::LuaManager*, std::__cxx11::basic_string, std::allocator > const&, optional_yield, rgw::lua::context, std::__cxx11::basic_string, std::allocator >&)+0x75) [0x55bbd4f518f5] 2022-09-09T23:10:20.277 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 4: (rgw::lua::Background::read_script()+0x9f) [0x55bbd4f5cd0f] 2022-09-09T23:10:20.278 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 5: (rgw::lua::Background::run()+0x6e4) [0x55bbd4f5dce4] 2022-09-09T23:10:20.278 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 6: /usr/lib/ceph/libceph-common.so.2(+0xa81eb4) [0x7f20a90c4eb4] 2022-09-09T23:10:20.278 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 7: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f20a7d75609] 2022-09-09T23:10:20.279 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 8: clone() 2022-09-09T23:10:20.279 INFO:tasks.rgw.c1.client.0.smithi005.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-09-09T23:10:20.280 INFO:tasks.rgw.c1.client.0.smithi005.stdout: 2022-09-09T23:10:20.485 INFO:tasks.rgw.c1.client.0.smithi005.stderr:daemon-helper: command crashed with signal 11 2022-09-09T23:10:24.374 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:24.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:24.377 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:10:24.424 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:10:24.425 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:10:24.487 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:10:24.487+0000 7f3abe775dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:10:24.489 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:10:24.491+0000 7f3abe775dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:10:24.514 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:10:24.515 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:10:24.515 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:10:24.516 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:10:24.516 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:10:24.516 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:10:24.516 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:10:24.517 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:24.517 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:10:24.517 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:24.518 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:10:24.518 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:24.518 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:24.519 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:10:24.519 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:24.519 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:10:24.520 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:24.520 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:10:24.520 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:10:24.520 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:24.521 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:24.521 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:24.521 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:10:24.521 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:24.522 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:24.522 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:10:24.522 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:24.522 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:24.523 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:24.523 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:10:24.523 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:10:24.523 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:24.524 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:24.524 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:24.524 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:10:24.524 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:24.525 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:24.525 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:10:24.525 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:24.525 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:24.526 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:24.526 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:10:24.526 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:10:24.526 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:24.527 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:24.527 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:24.527 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:10:24.527 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:24.528 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:24.528 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:10:24.528 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:24.528 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:24.529 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:24.529 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:10:24.529 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:10:24.529 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:24.529 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:24.530 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:10:24.530 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:24.530 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:10:24.530 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:10:24.531 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:10:24.531 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:24.531 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:10:24Z" 2022-09-09T23:10:24.531 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:10:24.533 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:10:24Z" } 2022-09-09T23:10:24.533 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:10:24.534 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:10:24.534 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:10:24.535 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:10:25.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2022-09-09T23:10:29.541 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:29.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:29.544 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:10:29.594 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:10:29.594 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:10:29.650 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:10:29.651+0000 7fe2b5139dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:10:29.651 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:10:29.651+0000 7fe2b5139dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:10:29.681 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:10:29.681 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:10:29.681 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:10:29.682 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:10:29.682 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:10:29.682 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:10:29.682 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:10:29.683 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:29.683 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:10:29.683 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:29.683 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:10:29.684 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:29.684 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:29.684 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:10:29.684 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:29.685 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:10:29.685 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:29.686 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:10:29.686 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:10:29.686 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:29.687 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:29.687 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:29.687 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:10:29.688 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:29.688 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:29.688 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:10:29.688 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:29.689 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:29.689 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:29.689 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:10:29.689 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:10:29.690 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:29.690 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:29.690 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:29.690 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:10:29.691 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:29.691 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:29.691 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:10:29.691 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:29.692 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:29.692 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:29.692 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:10:29.692 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:10:29.693 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:29.693 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:29.693 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:29.693 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:10:29.694 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:29.694 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:29.694 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:10:29.695 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:29.695 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:29.695 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:29.695 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:10:29.696 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:10:29.696 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:29.696 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:29.696 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:10:29.697 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:29.697 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:10:29.697 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:10:29.698 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:10:29.698 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:29.698 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:10:29Z" 2022-09-09T23:10:29.698 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:10:29.700 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:10:29Z" } 2022-09-09T23:10:29.700 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:10:29.701 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:10:29.701 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:10:29.702 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:10:31.152 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~6s 2022-09-09T23:10:34.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:34.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:34.704 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:10:34.752 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:10:34.753 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:10:34.821 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:10:34.823+0000 7f92d5833dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:10:34.822 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:10:34.823+0000 7f92d5833dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:10:34.850 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:10:34.850 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:10:34.850 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:10:34.851 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:10:34.851 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:10:34.851 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:10:34.851 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:10:34.852 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:34.852 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:10:34.853 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:34.853 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:10:34.854 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:34.854 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:34.855 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:10:34.855 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:34.856 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:10:34.856 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:34.857 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:10:34.857 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:10:34.858 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:34.858 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:34.859 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:34.859 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:10:34.860 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:34.860 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:34.861 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:10:34.861 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:34.862 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:34.862 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:34.862 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:10:34.863 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:10:34.863 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:34.864 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:34.864 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:34.865 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:10:34.865 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:34.866 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:34.866 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:10:34.867 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:34.867 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:34.868 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:34.868 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:10:34.869 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:10:34.869 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:34.870 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:34.870 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:34.871 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:10:34.871 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:34.872 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:34.872 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:10:34.872 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:34.873 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:34.873 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:34.874 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:10:34.874 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:10:34.875 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:34.875 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:34.876 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:10:34.876 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:34.877 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:10:34.877 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:10:34.878 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:10:34.878 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:34.879 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:10:34Z" 2022-09-09T23:10:34.879 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:10:34.881 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:10:34Z" } 2022-09-09T23:10:34.882 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:10:34.883 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:10:34.884 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:10:34.885 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:10:36.758 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~11s 2022-09-09T23:10:39.886 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:39.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:39.888 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:10:39.936 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:10:39.936 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:10:39.999 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:10:39.999+0000 7fc7cc39fdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:10:40.002 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:10:40.003+0000 7fc7cc39fdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:10:40.027 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:10:40.027 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:10:40.028 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:10:40.028 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:10:40.029 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:10:40.029 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:10:40.030 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:10:40.030 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:40.031 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:10:40.032 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:40.032 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:10:40.033 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:40.033 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:40.034 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:10:40.034 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:40.034 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:10:40.035 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:40.035 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:10:40.036 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:10:40.036 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:40.037 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:40.037 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:40.037 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:10:40.038 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:40.038 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:40.039 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:10:40.039 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:40.040 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:40.040 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:40.041 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:10:40.041 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:10:40.041 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:40.043 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:40.043 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:40.044 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:10:40.044 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:40.044 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:40.045 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:10:40.045 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:40.046 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:40.046 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:40.047 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:10:40.047 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:10:40.048 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:40.048 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:40.048 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:40.049 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:10:40.049 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:40.050 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:40.050 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:10:40.051 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:40.051 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:40.052 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:40.053 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:10:40.053 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:10:40.054 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:40.054 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:40.055 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:10:40.055 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:40.056 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:10:40.056 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:10:40.057 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:10:40.057 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:40.058 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:10:40Z" 2022-09-09T23:10:40.058 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:10:40.060 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:10:40Z" } 2022-09-09T23:10:40.061 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:10:40.062 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:10:40.062 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:10:40.063 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:10:42.364 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~17s 2022-09-09T23:10:45.064 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:45.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:45.068 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:10:45.115 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:10:45.116 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:10:45.179 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:10:45.178+0000 7f478483ddc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:10:45.181 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:10:45.182+0000 7f478483ddc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:10:45.212 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:10:45.213 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:10:45.213 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:10:45.213 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:10:45.213 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:10:45.214 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:10:45.214 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:10:45.215 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:45.216 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:10:45.216 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:45.216 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:10:45.217 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:45.218 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:45.218 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:10:45.218 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:45.219 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:10:45.219 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:45.220 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:10:45.220 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:10:45.221 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:45.222 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:45.222 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:45.223 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:10:45.223 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:45.224 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:45.224 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:10:45.225 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:45.226 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:45.227 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:45.227 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:10:45.228 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:10:45.228 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:45.228 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:45.228 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:45.229 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:10:45.229 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:45.229 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:45.229 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:10:45.230 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:45.230 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:45.230 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:45.231 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:10:45.231 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:10:45.231 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:45.231 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:45.232 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:45.232 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:10:45.232 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:45.233 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:45.233 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:10:45.233 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:45.233 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:45.234 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:45.234 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:10:45.234 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:10:45.234 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:45.235 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:45.235 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:10:45.235 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:45.236 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:10:45.236 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:10:45.236 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:10:45.237 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:45.237 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:10:45Z" 2022-09-09T23:10:45.237 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:10:45.238 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:10:45Z" } 2022-09-09T23:10:45.239 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:10:45.240 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:10:45.240 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:10:45.241 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:10:47.972 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~22s 2022-09-09T23:10:50.242 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:50.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:50.244 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:10:50.292 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:10:50.293 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:10:50.347 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:10:50.346+0000 7fac67a6edc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:10:50.349 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:10:50.350+0000 7fac67a6edc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:10:50.375 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:10:50.376 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:10:50.376 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:10:50.377 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:10:50.377 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:10:50.377 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:10:50.378 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:10:50.378 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:50.379 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:10:50.380 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:50.380 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:10:50.381 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:50.381 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:50.382 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:10:50.382 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:50.383 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:10:50.383 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:50.384 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:10:50.384 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:10:50.385 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:50.385 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:50.386 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:50.388 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:10:50.388 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:50.388 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:50.389 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:10:50.389 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:50.389 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:50.389 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:50.390 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:10:50.390 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:10:50.390 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:50.390 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:50.391 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:50.391 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:10:50.391 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:50.391 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:50.392 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:10:50.392 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:50.392 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:50.393 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:50.393 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:10:50.393 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:10:50.394 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:50.394 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:50.394 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:50.394 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:10:50.395 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:50.395 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:50.395 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:10:50.396 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:50.396 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:50.396 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:50.398 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:10:50.398 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:10:50.399 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:50.399 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:50.400 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:10:50.400 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:50.401 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:10:50.401 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:10:50.402 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:10:50.402 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:50.403 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:10:50Z" 2022-09-09T23:10:50.403 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:10:50.405 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:10:50Z" } 2022-09-09T23:10:50.406 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:10:50.407 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:10:50.408 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:10:50.409 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:10:53.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~28s 2022-09-09T23:10:55.410 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:55.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:10:55.412 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:10:55.460 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:10:55.460 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:10:55.521 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:10:55.522+0000 7f2af886edc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:10:55.523 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:10:55.522+0000 7f2af886edc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:10:55.553 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:10:55.553 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:10:55.554 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:10:55.554 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:10:55.554 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:10:55.554 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:10:55.555 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:10:55.555 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:55.555 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:10:55.555 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:55.556 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:10:55.556 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:55.556 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:55.557 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:10:55.557 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:55.557 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:10:55.558 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:55.558 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:10:55.558 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:10:55.558 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:55.559 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:55.559 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:55.559 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:10:55.559 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:55.560 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:55.560 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:10:55.560 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:55.560 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:55.561 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:55.561 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:10:55.562 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:10:55.562 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:55.562 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:55.562 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:55.563 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:10:55.563 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:55.563 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:55.563 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:10:55.564 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:55.564 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:55.564 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:55.564 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:10:55.565 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:10:55.565 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:55.565 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:55.565 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:10:55.566 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:10:55.566 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:10:55.566 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:10:55.567 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:10:55.567 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:10:55.567 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:10:55.567 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:10:55.568 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:10:55.568 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:10:55.568 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:55.568 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:10:55.569 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:10:55.569 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:55.569 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:10:55.570 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:10:55.570 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:10:55.570 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:10:55.570 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:10:55Z" 2022-09-09T23:10:55.571 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:10:55.572 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:10:55Z" } 2022-09-09T23:10:55.573 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:10:55.574 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:10:55.574 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:10:55.575 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:10:59.187 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~34s 2022-09-09T23:11:00.577 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:11:00.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:11:00.581 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:11:00.632 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:11:00.633 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:11:00.684 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:11:00.682+0000 7f028f4d0dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:11:00.687 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:11:00.686+0000 7f028f4d0dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:11:00.710 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:11:00.711 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:11:00.712 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:11:00.712 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:11:00.713 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:11:00.713 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:11:00.715 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:11:00.715 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:00.716 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:11:00.716 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:00.716 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:11:00.717 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:00.717 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:00.718 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:11:00.718 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:00.718 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:11:00.719 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:00.719 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:11:00.720 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:00.720 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:00.721 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:00.721 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:00.721 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:11:00.722 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:00.722 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:00.723 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:11:00.723 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:00.723 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:00.724 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:00.725 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:11:00.725 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:00.725 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:00.726 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:00.726 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:00.726 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:11:00.727 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:00.728 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:00.728 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:11:00.729 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:00.730 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:00.730 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:00.730 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:11:00.731 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:00.732 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:00.734 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:00.734 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:00.735 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:11:00.735 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:00.736 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:00.736 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:11:00.737 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:00.737 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:00.738 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:00.738 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:11:00.739 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:11:00.739 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:00.739 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:00.740 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:11:00.741 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:00.741 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:11:00.742 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:11:00.742 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:11:00.742 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:00.742 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:11:00Z" 2022-09-09T23:11:00.743 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:11:00.744 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:11:00Z" } 2022-09-09T23:11:00.745 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:11:00.746 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:11:00.747 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:11:00.748 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:11:04.794 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~39s 2022-09-09T23:11:05.749 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:11:05.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:11:05.751 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:11:05.798 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:11:05.798 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:11:05.867 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:11:05.866+0000 7f9ffcdcbdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:11:05.869 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:11:05.870+0000 7f9ffcdcbdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:11:05.896 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:11:05.897 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:11:05.897 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:11:05.898 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:11:05.898 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:11:05.898 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:11:05.899 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:11:05.899 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:05.900 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:11:05.900 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:05.901 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:11:05.901 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:05.903 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:05.905 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:11:05.905 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:05.906 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:11:05.906 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:05.907 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:11:05.907 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:05.908 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:05.908 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:05.908 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:05.909 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:11:05.909 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:05.910 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:05.910 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:11:05.910 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:05.911 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:05.911 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:05.911 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:11:05.912 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:05.913 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:05.914 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:05.915 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:05.915 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:11:05.916 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:05.916 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:05.916 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:11:05.917 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:05.917 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:05.918 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:05.918 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:11:05.919 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:05.919 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:05.920 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:05.920 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:05.921 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:11:05.921 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:05.921 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:05.922 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:11:05.923 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:05.923 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:05.923 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:05.924 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:11:05.924 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:11:05.925 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:05.925 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:05.925 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:11:05.926 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:05.926 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:11:05.927 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:11:05.927 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:11:05.928 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:05.928 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:11:05Z" 2022-09-09T23:11:05.928 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:11:05.931 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:11:05Z" } 2022-09-09T23:11:05.932 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:11:05.932 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:11:05.933 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:11:05.933 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:11:10.405 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~45s 2022-09-09T23:11:10.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:11:10.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:11:10.936 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:11:10.985 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:11:10.985 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:11:11.042 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:11:11.042+0000 7f9cbe909dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:11:11.044 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:11:11.046+0000 7f9cbe909dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:11:11.071 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:11:11.072 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:11:11.072 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:11:11.072 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:11:11.072 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:11:11.073 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:11:11.073 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:11:11.073 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:11.073 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:11:11.074 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:11.074 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:11:11.074 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:11.075 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:11.075 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:11:11.075 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:11.075 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:11:11.076 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:11.076 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:11:11.076 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:11.076 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:11.077 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:11.077 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:11.077 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:11:11.078 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:11.078 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:11.078 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:11:11.078 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:11.079 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:11.079 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:11.079 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:11:11.079 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:11.080 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:11.080 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:11.080 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:11.080 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:11:11.081 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:11.081 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:11.081 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:11:11.081 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:11.082 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:11.082 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:11.082 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:11:11.082 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:11.083 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:11.083 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:11.083 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:11.084 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:11:11.084 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:11.084 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:11.084 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:11:11.084 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:11.085 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:11.085 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:11.085 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:11:11.085 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:11:11.086 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:11.086 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:11.087 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:11:11.087 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:11.087 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:11:11.088 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:11:11.088 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:11:11.088 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:11.088 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:11:11Z" 2022-09-09T23:11:11.089 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:11:11.090 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:11:11Z" } 2022-09-09T23:11:11.091 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:11:11.091 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:11:11.092 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:11:11.092 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:11:16.012 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~50s 2022-09-09T23:11:16.094 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:11:16.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:11:16.096 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:11:16.144 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:11:16.145 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:11:16.198 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:11:16.198+0000 7fc144dc9dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:11:16.199 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:11:16.198+0000 7fc144dc9dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:11:16.226 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:11:16.227 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:11:16.227 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:11:16.228 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:11:16.228 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:11:16.228 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:11:16.229 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:11:16.229 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:16.230 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:11:16.230 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:16.230 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:11:16.230 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:16.231 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:16.231 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:11:16.231 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:16.231 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:11:16.232 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:16.232 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:11:16.232 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:16.233 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:16.233 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:16.234 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:16.234 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:11:16.234 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:16.234 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:16.235 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:11:16.235 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:16.235 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:16.235 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:16.236 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:11:16.236 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:16.236 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:16.236 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:16.237 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:16.237 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:11:16.237 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:16.237 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:16.238 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:11:16.238 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:16.238 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:16.239 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:16.239 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:11:16.239 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:16.240 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:16.240 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:16.240 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:16.241 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:11:16.241 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:16.242 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:16.242 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:11:16.242 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:16.243 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:16.243 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:16.243 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:11:16.244 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:11:16.244 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:16.244 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:16.245 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:11:16.245 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:16.245 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:11:16.245 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:11:16.246 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:11:16.246 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:16.246 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:11:16Z" 2022-09-09T23:11:16.246 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:11:16.248 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:11:16Z" } 2022-09-09T23:11:16.248 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:11:16.249 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:11:16.249 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:11:16.250 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:11:21.251 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:11:21.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:11:21.254 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:11:21.303 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:11:21.304 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:11:21.364 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:11:21.362+0000 7f2f83556dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:11:21.366 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:11:21.366+0000 7f2f83556dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:11:21.394 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:11:21.394 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:11:21.395 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:11:21.395 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:11:21.396 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:11:21.396 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:11:21.399 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:11:21.400 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:21.400 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:11:21.401 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:21.401 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:11:21.401 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:21.402 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:21.402 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:11:21.403 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:21.405 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:11:21.406 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:21.406 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:11:21.407 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:21.407 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:21.407 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:21.408 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:21.410 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:11:21.411 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:21.411 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:21.412 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:11:21.412 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:21.413 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:21.413 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:21.415 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:11:21.416 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:21.416 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:21.416 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:21.417 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:21.417 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:11:21.418 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:21.419 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:21.420 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:11:21.421 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:21.421 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:21.422 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:21.422 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:11:21.423 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:21.423 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:21.424 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:21.424 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:21.427 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:11:21.427 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:21.428 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:21.428 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:11:21.429 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:21.429 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:21.430 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:21.432 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:11:21.433 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:11:21.434 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:21.434 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:21.435 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:11:21.435 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:21.438 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:11:21.448 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:11:21.449 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:11:21.449 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:21.449 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:11:21Z" 2022-09-09T23:11:21.450 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:11:21.451 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:11:21Z" } 2022-09-09T23:11:21.452 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:11:21.453 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:11:21.453 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:11:21.454 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:11:21.667 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~56s 2022-09-09T23:11:26.455 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:11:26.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:11:26.456 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:11:26.503 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:11:26.503 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:11:26.564 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:11:26.562+0000 7f09d6adedc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:11:26.566 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:11:26.566+0000 7f09d6adedc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:11:26.595 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:11:26.596 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:11:26.596 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:11:26.596 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:11:26.596 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:11:26.597 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:11:26.597 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:11:26.597 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:26.598 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:11:26.598 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:26.598 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:11:26.598 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:26.599 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:26.599 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:11:26.599 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:26.599 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:11:26.600 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:26.600 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:11:26.600 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:26.600 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:26.601 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:26.601 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:26.601 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:11:26.601 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:26.602 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:26.602 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:11:26.602 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:26.602 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:26.603 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:26.603 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:11:26.603 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:26.604 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:26.604 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:26.604 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:26.604 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:11:26.605 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:26.605 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:26.605 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:11:26.605 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:26.606 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:26.606 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:26.606 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:11:26.606 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:26.607 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:26.607 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:26.607 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:26.607 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:11:26.608 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:26.608 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:26.608 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:11:26.608 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:26.609 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:26.609 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:26.609 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:11:26.609 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:11:26.610 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:26.610 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:26.610 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:11:26.610 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:26.611 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:11:26.611 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:11:26.611 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:11:26.611 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:26.612 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:11:26Z" 2022-09-09T23:11:26.612 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:11:26.613 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:11:26Z" } 2022-09-09T23:11:26.614 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:11:26.615 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:11:26.615 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:11:26.616 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:11:27.274 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~62s 2022-09-09T23:11:31.617 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:11:31.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:11:31.621 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:11:31.668 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:11:31.669 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:11:31.727 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:11:31.726+0000 7fc3a615bdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:11:31.728 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:11:31.730+0000 7fc3a615bdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:11:31.750 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:11:31.751 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:11:31.751 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:11:31.751 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:11:31.752 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:11:31.752 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:11:31.753 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:11:31.754 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:31.754 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:11:31.755 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:31.755 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:11:31.756 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:31.756 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:31.757 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:11:31.757 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:31.758 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:11:31.758 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:31.759 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:11:31.759 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:31.760 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:31.760 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:31.761 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:31.761 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:11:31.762 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:31.762 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:31.763 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:11:31.763 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:31.764 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:31.764 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:31.765 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:11:31.765 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:31.766 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:31.766 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:31.767 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:31.767 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:11:31.768 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:31.768 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:31.769 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:11:31.769 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:31.770 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:31.771 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:31.771 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:11:31.772 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:31.772 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:31.773 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:31.773 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:31.774 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:11:31.774 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:31.775 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:31.775 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:11:31.775 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:31.776 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:31.777 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:31.777 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:11:31.778 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:11:31.779 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:31.779 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:31.780 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:11:31.780 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:31.781 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:11:31.781 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:11:31.782 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:11:31.782 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:31.782 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:11:31Z" 2022-09-09T23:11:31.783 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:11:31.784 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:11:31Z" } 2022-09-09T23:11:31.785 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:11:31.786 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:11:31.787 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:11:31.787 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:11:32.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~67s 2022-09-09T23:11:36.789 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:11:36.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:11:36.791 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:11:36.839 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:11:36.840 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:11:36.893 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:11:36.893+0000 7fcecdbb2dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:11:36.895 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:11:36.893+0000 7fcecdbb2dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:11:36.921 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:11:36.922 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:11:36.923 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:11:36.923 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:11:36.924 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:11:36.924 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:11:36.924 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:11:36.924 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:36.925 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:11:36.925 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:36.925 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:11:36.925 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:36.926 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:36.926 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:11:36.926 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:36.927 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:11:36.927 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:36.927 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:11:36.928 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:36.928 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:36.928 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:36.928 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:36.929 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:11:36.929 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:36.929 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:36.929 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:11:36.930 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:36.930 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:36.930 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:36.931 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:11:36.931 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:36.931 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:36.932 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:36.932 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:36.932 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:11:36.932 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:36.933 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:36.934 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:11:36.934 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:36.934 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:36.934 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:36.935 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:11:36.935 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:36.935 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:36.935 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:36.936 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:36.936 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:11:36.936 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:36.937 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:36.937 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:11:36.937 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:36.937 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:36.938 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:36.938 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:11:36.938 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:11:36.939 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:36.939 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:36.939 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:11:36.939 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:36.940 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:11:36.940 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:11:36.940 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:11:36.940 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:36.941 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:11:36Z" 2022-09-09T23:11:36.941 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:11:36.942 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:11:36Z" } 2022-09-09T23:11:36.943 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:11:36.944 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:11:36.945 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:11:36.945 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:11:38.488 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~73s 2022-09-09T23:11:41.947 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:11:41.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:11:41.949 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:11:41.998 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:11:41.999 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:11:42.071 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:11:42.069+0000 7f81a1558dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:11:42.073 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:11:42.073+0000 7f81a1558dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:11:42.103 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:11:42.104 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:11:42.105 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:11:42.106 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:11:42.112 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:11:42.113 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:11:42.114 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:11:42.114 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:42.115 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:11:42.115 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:42.116 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:11:42.116 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:42.117 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:42.117 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:11:42.118 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:42.118 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:11:42.119 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:42.119 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:11:42.120 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:42.120 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:42.121 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:42.122 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:42.122 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:11:42.123 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:42.124 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:42.124 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:11:42.125 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:42.125 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:42.126 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:42.126 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:11:42.127 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:42.127 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:42.127 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:42.128 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:42.128 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:11:42.129 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:42.129 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:42.130 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:11:42.130 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:42.131 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:42.131 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:42.132 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:11:42.132 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:42.133 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:42.133 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:42.133 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:42.134 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:11:42.134 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:42.134 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:42.135 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:11:42.135 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:42.135 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:42.135 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:42.136 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:11:42.136 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:11:42.136 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:42.136 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:42.137 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:11:42.137 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:42.137 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:11:42.138 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:11:42.138 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:11:42.138 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:42.138 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:11:42Z" 2022-09-09T23:11:42.139 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:11:42.141 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:11:42Z" } 2022-09-09T23:11:42.141 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:11:42.142 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:11:42.143 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:11:42.144 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:11:44.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~79s 2022-09-09T23:11:47.145 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:11:47.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:11:47.147 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:11:47.196 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:11:47.198 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:11:47.252 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:11:47.249+0000 7f19c2901dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:11:47.254 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:11:47.253+0000 7f19c2901dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:11:47.282 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:11:47.283 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:11:47.283 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:11:47.284 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:11:47.284 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:11:47.284 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:11:47.285 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:11:47.285 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:47.285 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:11:47.285 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:47.286 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:11:47.286 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:47.286 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:47.287 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:11:47.287 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:47.287 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:11:47.287 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:47.288 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:11:47.288 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:47.288 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:47.288 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:47.289 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:47.289 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:11:47.289 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:47.289 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:47.290 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:11:47.290 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:47.290 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:47.290 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:47.291 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:11:47.291 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:47.291 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:47.291 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:47.292 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:47.292 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:11:47.292 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:47.292 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:47.293 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:11:47.293 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:47.294 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:47.294 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:47.294 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:11:47.295 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:47.295 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:47.295 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:47.295 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:47.296 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:11:47.296 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:47.296 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:47.297 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:11:47.297 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:47.297 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:47.298 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:47.298 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:11:47.298 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:11:47.298 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:47.299 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:47.299 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:11:47.299 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:47.299 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:11:47.300 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:11:47.300 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:11:47.301 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:47.301 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:11:47Z" 2022-09-09T23:11:47.301 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:11:47.302 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:11:47Z" } 2022-09-09T23:11:47.303 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:11:47.304 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:11:47.304 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:11:47.305 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:11:49.704 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~84s 2022-09-09T23:11:52.306 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:11:52.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:11:52.308 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:11:52.358 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:11:52.358 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:11:52.428 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:11:52.425+0000 7f807df96dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:11:52.430 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:11:52.429+0000 7f807df96dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:11:52.456 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:11:52.457 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:11:52.458 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:11:52.458 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:11:52.459 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:11:52.459 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:11:52.460 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:11:52.460 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:52.461 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:11:52.461 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:52.462 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:11:52.463 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:52.463 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:52.463 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:11:52.464 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:52.464 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:11:52.464 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:52.465 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:11:52.465 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:52.465 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:52.466 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:52.466 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:52.467 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:11:52.467 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:52.467 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:52.468 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:11:52.468 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:52.468 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:52.469 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:52.469 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:11:52.469 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:52.470 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:52.470 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:52.470 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:52.471 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:11:52.471 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:52.472 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:52.472 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:11:52.473 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:52.473 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:52.473 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:52.475 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:11:52.475 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:52.476 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:52.476 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:52.477 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:52.477 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:11:52.478 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:52.479 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:52.479 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:11:52.480 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:52.480 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:52.481 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:52.481 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:11:52.482 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:11:52.482 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:52.483 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:52.483 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:11:52.484 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:52.484 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:11:52.485 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:11:52.485 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:11:52.486 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:52.486 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:11:52Z" 2022-09-09T23:11:52.487 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:11:52.489 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:11:52Z" } 2022-09-09T23:11:52.491 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:11:52.492 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:11:52.492 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:11:52.493 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:11:55.312 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~90s 2022-09-09T23:11:57.496 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:11:57.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:11:57.498 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:11:57.544 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:11:57.545 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:11:57.597 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:11:57.597+0000 7fcb05faedc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:11:57.599 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:11:57.597+0000 7fcb05faedc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:11:57.626 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:11:57.626 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:11:57.627 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:11:57.627 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:11:57.627 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:11:57.628 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:11:57.628 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:11:57.628 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:57.628 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:11:57.629 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:57.629 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:11:57.629 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:57.629 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:57.630 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:11:57.630 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:57.630 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:11:57.630 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:57.631 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:11:57.631 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:57.631 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:57.631 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:57.632 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:57.632 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:11:57.632 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:57.632 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:57.633 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:11:57.633 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:57.634 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:57.634 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:57.634 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:11:57.635 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:57.635 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:57.635 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:57.635 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:57.636 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:11:57.636 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:57.636 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:57.636 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:11:57.637 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:57.637 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:57.637 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:57.637 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:11:57.638 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:11:57.638 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:57.638 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:57.639 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:11:57.639 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:11:57.639 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:11:57.640 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:11:57.640 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:11:57.640 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:11:57.640 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:11:57.641 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:11:57.641 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:11:57.641 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:11:57.641 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:57.642 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:11:57.642 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:11:57.642 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:57.642 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:11:57.643 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:11:57.643 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:11:57.643 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:11:57.643 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:11:57Z" 2022-09-09T23:11:57.644 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:11:57.645 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:11:57Z" } 2022-09-09T23:11:57.646 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:11:57.646 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:11:57.647 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:11:57.647 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:12:00.918 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~95s 2022-09-09T23:12:02.648 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:12:02.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:12:02.650 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:12:02.701 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:12:02.702 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:12:02.765 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:12:02.765+0000 7f752a5a4dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:12:02.768 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:12:02.765+0000 7f752a5a4dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:12:02.793 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:12:02.794 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:12:02.794 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:12:02.794 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:12:02.795 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:12:02.795 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:12:02.795 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:12:02.795 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:02.796 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:12:02.796 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:02.796 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:12:02.797 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:02.797 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:02.797 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:12:02.797 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:02.798 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:12:02.798 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:02.799 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:12:02.799 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:02.799 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:02.800 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:02.800 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:02.801 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:12:02.801 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:02.802 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:02.802 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:12:02.803 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:02.803 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:02.803 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:02.804 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:12:02.804 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:02.805 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:02.805 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:02.805 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:02.806 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:12:02.806 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:02.806 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:02.807 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:12:02.807 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:02.807 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:02.808 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:02.808 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:12:02.809 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:02.809 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:02.810 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:02.810 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:02.810 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:12:02.811 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:02.811 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:02.811 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:12:02.812 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:02.812 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:02.812 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:02.812 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:12:02.813 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:12:02.813 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:02.813 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:02.813 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:12:02.814 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:02.814 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:12:02.814 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:12:02.815 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:12:02.815 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:02.815 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:12:02Z" 2022-09-09T23:12:02.816 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:12:02.817 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:12:02Z" } 2022-09-09T23:12:02.818 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:12:02.819 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:12:02.819 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:12:02.820 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:12:06.524 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~101s 2022-09-09T23:12:07.821 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:12:07.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:12:07.823 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:12:07.876 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:12:07.877 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:12:07.932 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:12:07.929+0000 7fba8dab8dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:12:07.934 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:12:07.933+0000 7fba8dab8dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:12:07.960 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:12:07.961 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:12:07.961 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:12:07.962 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:12:07.962 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:12:07.962 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:12:07.963 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:12:07.963 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:07.963 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:12:07.964 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:07.964 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:12:07.965 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:07.965 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:07.965 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:12:07.966 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:07.966 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:12:07.967 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:07.967 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:12:07.967 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:07.967 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:07.968 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:07.968 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:07.968 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:12:07.968 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:07.969 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:07.969 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:12:07.969 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:07.970 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:07.970 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:07.971 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:12:07.971 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:07.971 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:07.972 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:07.972 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:07.973 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:12:07.973 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:07.974 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:07.974 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:12:07.975 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:07.975 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:07.975 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:07.976 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:12:07.977 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:07.977 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:07.977 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:07.978 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:07.978 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:12:07.978 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:07.978 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:07.979 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:12:07.979 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:07.979 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:07.979 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:07.980 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:12:07.980 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:12:07.980 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:07.980 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:07.981 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:12:07.981 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:07.981 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:12:07.982 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:12:07.982 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:12:07.982 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:07.983 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:12:07Z" 2022-09-09T23:12:07.983 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:12:07.984 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:12:07Z" } 2022-09-09T23:12:07.985 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:12:07.986 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:12:07.986 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:12:07.987 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:12:12.131 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~107s 2022-09-09T23:12:12.988 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:12:12.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:12:12.991 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:12:13.042 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:12:13.043 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:12:13.119 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:12:13.113+0000 7f80dbaf6dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:12:13.119 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:12:13.113+0000 7f80dbaf6dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:12:13.145 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:12:13.146 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:12:13.146 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:12:13.146 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:12:13.147 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:12:13.147 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:12:13.147 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:12:13.148 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:13.148 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:12:13.148 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:13.148 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:12:13.149 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:13.149 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:13.149 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:12:13.149 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:13.150 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:12:13.150 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:13.150 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:12:13.151 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:13.151 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:13.151 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:13.152 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:13.152 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:12:13.152 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:13.153 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:13.153 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:12:13.153 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:13.153 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:13.155 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:13.155 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:12:13.155 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:13.156 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:13.156 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:13.156 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:13.156 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:12:13.157 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:13.157 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:13.157 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:12:13.157 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:13.158 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:13.158 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:13.158 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:12:13.159 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:13.159 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:13.159 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:13.159 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:13.160 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:12:13.160 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:13.160 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:13.161 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:12:13.161 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:13.161 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:13.161 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:13.162 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:12:13.162 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:12:13.162 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:13.163 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:13.163 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:12:13.163 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:13.163 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:12:13.163 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:12:13.164 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:12:13.164 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:13.164 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:12:13Z" 2022-09-09T23:12:13.164 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:12:13.166 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:12:13Z" } 2022-09-09T23:12:13.166 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:12:13.167 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:12:13.167 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:12:13.168 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:12:17.742 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~112s 2022-09-09T23:12:18.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:12:18.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:12:18.173 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:12:18.227 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:12:18.228 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:12:18.291 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:12:18.289+0000 7fed91927dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:12:18.301 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:12:18.293+0000 7fed91927dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:12:18.322 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:12:18.323 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:12:18.323 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:12:18.324 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:12:18.324 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:12:18.324 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:12:18.324 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:12:18.325 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:18.325 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:12:18.325 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:18.325 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:12:18.326 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:18.327 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:18.327 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:12:18.327 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:18.327 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:12:18.328 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:18.328 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:12:18.329 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:18.329 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:18.329 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:18.330 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:18.330 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:12:18.330 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:18.330 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:18.331 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:12:18.331 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:18.331 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:18.332 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:18.332 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:12:18.332 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:18.333 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:18.333 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:18.333 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:18.333 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:12:18.334 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:18.334 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:18.335 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:12:18.335 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:18.336 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:18.336 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:18.337 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:12:18.337 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:18.338 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:18.338 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:18.338 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:18.339 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:12:18.340 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:18.340 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:18.340 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:12:18.341 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:18.342 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:18.342 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:18.343 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:12:18.343 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:12:18.344 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:18.345 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:18.345 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:12:18.346 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:18.346 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:12:18.347 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:12:18.347 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:12:18.348 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:18.348 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:12:18Z" 2022-09-09T23:12:18.349 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:12:18.351 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:12:18Z" } 2022-09-09T23:12:18.352 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:12:18.353 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:12:18.354 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:12:18.355 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:12:23.352 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~118s 2022-09-09T23:12:23.357 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:12:23.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:12:23.358 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:12:23.406 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:12:23.406 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:12:23.462 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:12:23.460+0000 7f7c6c4fbdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:12:23.463 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:12:23.460+0000 7f7c6c4fbdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:12:23.498 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:12:23.498 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:12:23.499 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:12:23.499 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:12:23.500 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:12:23.500 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:12:23.501 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:12:23.501 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:23.502 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:12:23.502 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:23.503 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:12:23.503 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:23.504 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:23.504 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:12:23.505 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:23.505 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:12:23.506 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:23.506 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:12:23.507 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:23.507 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:23.508 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:23.509 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:23.509 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:12:23.510 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:23.510 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:23.511 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:12:23.511 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:23.512 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:23.512 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:23.513 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:12:23.513 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:23.514 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:23.514 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:23.515 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:23.515 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:12:23.516 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:23.516 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:23.517 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:12:23.517 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:23.518 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:23.518 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:23.519 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:12:23.519 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:23.520 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:23.520 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:23.521 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:23.521 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:12:23.522 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:23.522 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:23.523 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:12:23.523 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:23.524 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:23.524 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:23.525 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:12:23.525 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:12:23.526 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:23.526 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:23.527 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:12:23.527 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:23.528 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:12:23.528 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:12:23.529 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:12:23.529 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:23.530 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:12:23Z" 2022-09-09T23:12:23.530 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:12:23.531 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:12:23Z" } 2022-09-09T23:12:23.532 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:12:23.533 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:12:23.534 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:12:23.535 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:12:28.536 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:12:28.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:12:28.538 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:12:28.584 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:12:28.585 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:12:28.646 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:12:28.644+0000 7fece0f80dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:12:28.649 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:12:28.648+0000 7fece0f80dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:12:28.677 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:12:28.678 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:12:28.678 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:12:28.679 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:12:28.679 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:12:28.679 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:12:28.680 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:12:28.681 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:28.681 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:12:28.681 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:28.681 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:12:28.682 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:28.682 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:28.682 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:12:28.683 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:28.683 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:12:28.683 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:28.684 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:12:28.684 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:28.684 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:28.685 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:28.685 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:28.685 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:12:28.686 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:28.686 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:28.686 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:12:28.686 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:28.687 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:28.687 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:28.687 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:12:28.688 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:28.688 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:28.688 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:28.689 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:28.689 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:12:28.689 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:28.689 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:28.690 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:12:28.690 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:28.690 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:28.690 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:28.691 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:12:28.691 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:28.691 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:28.691 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:28.692 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:28.692 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:12:28.692 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:28.693 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:28.693 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:12:28.693 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:28.693 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:28.694 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:28.694 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:12:28.695 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:12:28.695 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:28.695 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:28.695 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:12:28.696 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:28.696 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:12:28.696 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:12:28.696 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:12:28.697 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:28.697 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:12:28Z" 2022-09-09T23:12:28.697 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:12:28.699 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:12:28Z" } 2022-09-09T23:12:28.700 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:12:28.700 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:12:28.701 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:12:28.701 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:12:28.957 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~123s 2022-09-09T23:12:33.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:12:33.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:12:33.705 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:12:33.750 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:12:33.751 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:12:33.820 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:12:33.820+0000 7faf12d19dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:12:33.823 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:12:33.820+0000 7faf12d19dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:12:33.850 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:12:33.851 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:12:33.851 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:12:33.852 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:12:33.852 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:12:33.853 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:12:33.853 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:12:33.853 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:33.856 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:12:33.856 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:33.857 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:12:33.857 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:33.857 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:33.858 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:12:33.858 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:33.858 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:12:33.858 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:33.859 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:12:33.859 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:33.859 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:33.860 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:33.860 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:33.860 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:12:33.860 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:33.861 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:33.861 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:12:33.861 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:33.861 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:33.862 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:33.862 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:12:33.862 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:33.863 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:33.863 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:33.863 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:33.863 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:12:33.864 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:33.864 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:33.864 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:12:33.864 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:33.865 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:33.865 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:33.865 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:12:33.865 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:33.866 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:33.868 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:33.868 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:33.868 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:12:33.869 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:33.869 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:33.869 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:12:33.869 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:33.870 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:33.870 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:33.870 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:12:33.870 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:12:33.871 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:33.871 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:33.871 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:12:33.871 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:33.872 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:12:33.872 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:12:33.872 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:12:33.872 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:33.873 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:12:33Z" 2022-09-09T23:12:33.873 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:12:33.875 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:12:33Z" } 2022-09-09T23:12:33.875 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:12:33.876 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:12:33.877 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:12:33.877 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:12:34.564 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~129s 2022-09-09T23:12:38.879 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:12:38.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:12:38.881 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:12:38.933 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:12:38.933 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:12:39.001 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:12:39.000+0000 7f705269cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:12:39.003 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:12:39.000+0000 7f705269cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:12:39.030 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:12:39.030 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:12:39.031 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:12:39.031 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:12:39.032 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:12:39.032 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:12:39.033 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:12:39.033 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:39.034 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:12:39.034 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:39.034 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:12:39.035 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:39.035 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:39.035 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:12:39.035 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:39.036 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:12:39.036 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:39.036 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:12:39.036 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:39.037 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:39.037 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:39.037 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:39.037 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:12:39.038 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:39.038 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:39.038 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:12:39.038 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:39.039 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:39.039 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:39.039 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:12:39.039 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:39.040 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:39.040 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:39.040 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:39.040 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:12:39.041 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:39.041 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:39.041 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:12:39.042 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:39.042 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:39.042 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:39.043 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:12:39.043 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:39.043 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:39.044 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:39.044 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:39.044 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:12:39.045 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:39.045 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:39.045 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:12:39.046 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:39.046 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:39.046 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:39.047 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:12:39.047 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:12:39.047 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:39.048 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:39.048 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:12:39.048 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:39.049 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:12:39.049 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:12:39.049 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:12:39.049 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:39.050 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:12:39Z" 2022-09-09T23:12:39.050 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:12:39.051 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:12:39Z" } 2022-09-09T23:12:39.052 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:12:39.052 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:12:39.053 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:12:39.055 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:12:40.171 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~135s 2022-09-09T23:12:44.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:12:44.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:12:44.059 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:12:44.107 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:12:44.107 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:12:44.164 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:12:44.160+0000 7fb591dc7dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:12:44.165 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:12:44.160+0000 7fb591dc7dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:12:44.192 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:12:44.192 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:12:44.192 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:12:44.193 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:12:44.193 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:12:44.193 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:12:44.194 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:12:44.194 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:44.195 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:12:44.195 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:44.195 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:12:44.195 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:44.196 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:44.196 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:12:44.196 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:44.196 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:12:44.197 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:44.197 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:12:44.197 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:44.197 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:44.198 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:44.198 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:44.198 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:12:44.198 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:44.199 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:44.199 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:12:44.199 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:44.200 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:44.200 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:44.200 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:12:44.200 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:44.201 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:44.201 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:44.201 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:44.201 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:12:44.202 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:44.202 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:44.202 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:12:44.202 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:44.203 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:44.203 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:44.203 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:12:44.203 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:44.204 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:44.204 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:44.204 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:44.204 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:12:44.205 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:44.205 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:44.205 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:12:44.205 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:44.206 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:44.206 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:44.206 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:12:44.206 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:12:44.207 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:44.207 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:44.207 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:12:44.207 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:44.208 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:12:44.208 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:12:44.208 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:12:44.208 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:44.209 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:12:44Z" 2022-09-09T23:12:44.209 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:12:44.210 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:12:44Z" } 2022-09-09T23:12:44.211 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:12:44.211 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:12:44.212 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:12:44.212 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:12:45.777 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~140s 2022-09-09T23:12:49.214 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:12:49.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:12:49.215 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:12:49.263 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:12:49.263 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:12:49.318 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:12:49.316+0000 7facb4a69dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:12:49.320 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:12:49.316+0000 7facb4a69dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:12:49.347 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:12:49.347 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:12:49.347 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:12:49.348 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:12:49.348 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:12:49.348 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:12:49.349 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:12:49.349 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:49.349 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:12:49.350 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:49.351 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:12:49.351 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:49.351 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:49.352 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:12:49.352 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:49.352 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:12:49.353 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:49.353 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:12:49.354 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:49.354 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:49.355 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:49.355 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:49.355 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:12:49.355 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:49.356 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:49.356 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:12:49.356 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:49.356 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:49.357 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:49.357 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:12:49.357 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:49.357 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:49.358 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:49.359 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:49.360 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:12:49.360 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:49.361 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:49.361 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:12:49.361 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:49.362 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:49.362 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:49.362 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:12:49.363 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:49.363 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:49.363 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:49.364 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:49.364 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:12:49.364 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:49.365 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:49.365 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:12:49.365 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:49.365 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:49.366 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:49.366 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:12:49.366 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:12:49.367 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:49.367 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:49.367 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:12:49.368 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:49.368 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:12:49.368 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:12:49.368 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:12:49.369 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:49.369 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:12:49Z" 2022-09-09T23:12:49.369 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:12:49.384 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:12:49Z" } 2022-09-09T23:12:49.385 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:12:49.387 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:12:49.389 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:12:49.390 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:12:51.386 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~146s 2022-09-09T23:12:54.396 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:12:54.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:12:54.398 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:12:54.448 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:12:54.448 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:12:54.515 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:12:54.512+0000 7efda5405dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:12:54.517 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:12:54.516+0000 7efda5405dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:12:54.547 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:12:54.548 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:12:54.548 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:12:54.548 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:12:54.549 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:12:54.549 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:12:54.549 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:12:54.549 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:54.550 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:12:54.550 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:54.550 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:12:54.551 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:54.551 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:54.551 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:12:54.552 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:54.552 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:12:54.552 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:54.553 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:12:54.553 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:54.553 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:54.553 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:54.554 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:54.554 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:12:54.554 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:54.554 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:54.555 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:12:54.555 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:54.555 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:54.555 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:54.556 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:12:54.556 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:54.556 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:54.556 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:54.557 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:54.558 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:12:54.558 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:54.558 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:54.559 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:12:54.559 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:54.559 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:54.559 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:54.560 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:12:54.560 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:54.560 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:54.560 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:54.561 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:54.561 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:12:54.561 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:54.561 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:54.562 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:12:54.562 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:54.562 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:54.563 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:54.563 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:12:54.563 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:12:54.564 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:54.564 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:54.565 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:12:54.565 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:54.566 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:12:54.566 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:12:54.567 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:12:54.568 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:54.568 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:12:54Z" 2022-09-09T23:12:54.568 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:12:54.570 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:12:54Z" } 2022-09-09T23:12:54.570 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:12:54.571 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:12:54.572 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:12:54.572 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:12:56.994 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~151s 2022-09-09T23:12:59.574 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:12:59.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:12:59.576 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:12:59.626 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:12:59.627 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:12:59.694 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:12:59.688+0000 7fe1419c7dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:12:59.694 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:12:59.688+0000 7fe1419c7dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:12:59.727 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:12:59.728 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:12:59.728 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:12:59.729 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:12:59.729 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:12:59.730 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:12:59.730 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:12:59.731 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:59.731 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:12:59.732 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:59.732 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:12:59.733 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:59.733 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:59.733 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:12:59.734 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:59.734 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:12:59.735 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:59.735 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:12:59.736 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:59.736 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:59.737 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:59.737 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:59.738 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:12:59.738 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:59.739 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:59.739 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:12:59.740 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:59.740 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:59.741 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:59.741 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:12:59.741 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:59.742 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:59.742 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:59.743 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:59.743 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:12:59.744 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:59.744 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:59.745 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:12:59.745 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:59.746 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:59.746 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:59.747 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:12:59.747 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:12:59.747 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:59.748 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:59.748 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:12:59.749 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:12:59.749 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:12:59.749 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:12:59.750 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:12:59.750 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:12:59.751 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:12:59.751 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:12:59.752 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:12:59.752 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:12:59.753 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:59.753 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:12:59.754 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:12:59.754 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:59.755 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:12:59.755 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:12:59.755 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:12:59.756 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:12:59.756 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:12:59Z" 2022-09-09T23:12:59.756 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:12:59.757 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:12:59Z" } 2022-09-09T23:12:59.757 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:12:59.758 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:12:59.759 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:12:59.760 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:13:02.610 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~157s 2022-09-09T23:13:04.762 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:13:04.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:13:04.763 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:13:04.811 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:13:04.811 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:13:04.867 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:13:04.864+0000 7fd191b44dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:13:04.869 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:13:04.868+0000 7fd191b44dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:13:04.903 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:13:04.904 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:13:04.904 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:13:04.905 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:13:04.905 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:13:04.905 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:13:04.906 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:13:04.906 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:04.907 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:13:04.907 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:04.908 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:13:04.908 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:04.909 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:04.910 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:13:04.911 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:04.911 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:13:04.911 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:04.912 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:13:04.912 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:04.912 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:04.912 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:04.913 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:04.913 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:13:04.913 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:04.913 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:04.914 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:13:04.914 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:04.915 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:04.915 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:04.916 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:13:04.916 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:04.917 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:04.917 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:04.918 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:04.919 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:13:04.919 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:04.920 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:04.920 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:13:04.921 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:04.921 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:04.922 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:04.922 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:13:04.923 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:04.923 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:04.924 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:04.924 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:04.925 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:13:04.925 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:04.926 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:04.926 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:13:04.927 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:04.927 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:04.928 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:04.929 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:13:04.929 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:13:04.930 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:04.930 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:04.930 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:13:04.931 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:04.931 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:13:04.932 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:13:04.932 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:13:04.932 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:04.932 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:13:04Z" 2022-09-09T23:13:04.933 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:13:04.933 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:13:04Z" } 2022-09-09T23:13:04.934 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:13:04.935 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:13:04.936 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:13:04.937 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:13:08.217 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~163s 2022-09-09T23:13:09.938 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:13:09.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:13:09.942 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:13:09.994 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:13:09.994 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:13:10.053 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:13:10.052+0000 7f0e52e22dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:13:10.055 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:13:10.052+0000 7f0e52e22dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:13:10.088 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:13:10.088 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:13:10.088 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:13:10.089 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:13:10.089 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:13:10.089 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:13:10.090 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:13:10.090 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:10.090 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:13:10.091 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:10.091 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:13:10.091 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:10.091 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:10.092 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:13:10.092 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:10.092 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:13:10.092 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:10.093 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:13:10.093 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:10.093 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:10.093 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:10.093 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:10.094 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:13:10.094 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:10.094 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:10.095 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:13:10.095 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:10.095 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:10.095 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:10.096 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:13:10.096 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:10.096 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:10.098 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:10.099 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:10.099 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:13:10.101 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:10.102 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:10.102 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:13:10.102 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:10.102 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:10.103 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:10.103 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:13:10.103 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:10.103 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:10.104 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:10.104 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:10.104 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:13:10.104 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:10.105 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:10.105 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:13:10.106 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:10.106 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:10.107 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:10.109 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:13:10.110 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:13:10.110 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:10.110 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:10.111 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:13:10.111 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:10.111 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:13:10.111 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:13:10.112 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:13:10.112 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:10.112 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:13:10Z" 2022-09-09T23:13:10.112 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:13:10.120 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:13:10Z" } 2022-09-09T23:13:10.120 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:13:10.121 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:13:10.121 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:13:10.122 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:13:13.822 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~168s 2022-09-09T23:13:15.124 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:13:15.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:13:15.125 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:13:15.174 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:13:15.174 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:13:15.233 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:13:15.231+0000 7fd4b9613dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:13:15.235 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:13:15.231+0000 7fd4b9613dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:13:15.261 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:13:15.262 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:13:15.262 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:13:15.262 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:13:15.263 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:13:15.263 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:13:15.263 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:13:15.263 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:15.264 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:13:15.264 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:15.264 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:13:15.264 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:15.265 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:15.265 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:13:15.265 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:15.265 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:13:15.266 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:15.266 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:13:15.266 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:15.266 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:15.267 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:15.267 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:15.267 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:13:15.267 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:15.268 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:15.268 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:13:15.268 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:15.268 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:15.269 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:15.269 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:13:15.269 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:15.273 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:15.274 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:15.274 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:15.274 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:13:15.275 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:15.276 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:15.276 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:13:15.276 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:15.276 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:15.277 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:15.277 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:13:15.277 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:15.277 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:15.278 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:15.278 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:15.278 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:13:15.279 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:15.279 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:15.279 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:13:15.279 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:15.280 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:15.281 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:15.281 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:13:15.282 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:13:15.282 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:15.282 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:15.283 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:13:15.283 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:15.283 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:13:15.283 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:13:15.284 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:13:15.284 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:15.285 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:13:15Z" 2022-09-09T23:13:15.285 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:13:15.286 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:13:15Z" } 2022-09-09T23:13:15.287 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:13:15.288 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:13:15.288 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:13:15.289 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:13:19.428 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~174s 2022-09-09T23:13:20.290 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:13:20.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:13:20.292 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:13:20.341 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:13:20.342 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:13:20.416 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:13:20.415+0000 7fabc593cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:13:20.418 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:13:20.415+0000 7fabc593cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:13:20.450 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:13:20.451 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:13:20.452 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:13:20.453 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:13:20.454 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:13:20.455 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:13:20.455 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:13:20.456 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:20.456 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:13:20.457 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:20.457 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:13:20.458 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:20.459 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:20.460 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:13:20.460 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:20.461 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:13:20.461 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:20.466 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:13:20.467 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:20.467 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:20.468 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:20.468 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:20.469 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:13:20.469 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:20.469 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:20.472 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:13:20.472 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:20.473 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:20.473 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:20.474 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:13:20.474 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:20.474 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:20.474 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:20.475 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:20.475 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:13:20.475 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:20.476 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:20.476 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:13:20.476 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:20.476 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:20.477 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:20.477 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:13:20.477 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:20.477 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:20.478 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:20.478 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:20.478 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:13:20.479 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:20.479 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:20.479 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:13:20.480 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:20.480 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:20.480 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:20.481 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:13:20.481 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:13:20.481 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:20.481 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:20.482 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:13:20.482 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:20.482 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:13:20.482 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:13:20.483 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:13:20.483 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:20.483 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:13:20Z" 2022-09-09T23:13:20.484 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:13:20.487 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:13:20Z" } 2022-09-09T23:13:20.488 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:13:20.489 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:13:20.490 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:13:20.491 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:13:25.037 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~179s 2022-09-09T23:13:25.494 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:13:25.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:13:25.496 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:13:25.546 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:13:25.547 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:13:25.622 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:13:25.611+0000 7f6b4ab58dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:13:25.623 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:13:25.615+0000 7f6b4ab58dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:13:25.635 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:13:25.636 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:13:25.636 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:13:25.637 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:13:25.637 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:13:25.638 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:13:25.638 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:13:25.639 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:25.639 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:13:25.640 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:25.640 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:13:25.641 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:25.642 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:25.642 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:13:25.642 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:25.643 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:13:25.643 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:25.644 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:13:25.644 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:25.645 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:25.645 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:25.646 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:25.647 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:13:25.647 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:25.648 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:25.648 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:13:25.649 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:25.649 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:25.650 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:25.650 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:13:25.651 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:25.651 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:25.652 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:25.652 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:25.653 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:13:25.653 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:25.654 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:25.654 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:13:25.655 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:25.655 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:25.656 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:25.656 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:13:25.657 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:25.659 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:25.659 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:25.659 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:25.659 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:13:25.661 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:25.662 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:25.662 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:13:25.663 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:25.663 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:25.664 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:25.664 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:13:25.665 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:13:25.665 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:25.665 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:25.666 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:13:25.667 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:25.667 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:13:25.668 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:13:25.668 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:13:25.669 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:25.669 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:13:25Z" 2022-09-09T23:13:25.669 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:13:25.671 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:13:25Z" } 2022-09-09T23:13:25.672 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:13:25.672 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:13:25.673 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:13:25.673 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:13:30.656 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~185s 2022-09-09T23:13:30.675 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:13:30.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:13:30.676 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:13:30.727 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:13:30.728 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:13:30.799 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:13:30.795+0000 7efe0154cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:13:30.800 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:13:30.799+0000 7efe0154cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:13:30.828 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:13:30.829 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:13:30.829 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:13:30.830 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:13:30.830 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:13:30.831 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:13:30.831 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:13:30.832 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:30.833 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:13:30.833 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:30.834 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:13:30.835 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:30.838 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:30.839 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:13:30.839 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:30.840 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:13:30.840 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:30.841 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:13:30.841 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:30.842 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:30.842 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:30.843 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:30.843 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:13:30.844 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:30.844 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:30.844 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:13:30.845 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:30.845 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:30.846 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:30.846 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:13:30.846 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:30.847 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:30.848 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:30.849 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:30.849 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:13:30.849 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:30.850 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:30.850 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:13:30.851 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:30.851 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:30.852 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:30.852 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:13:30.853 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:30.853 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:30.853 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:30.854 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:30.854 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:13:30.854 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:30.855 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:30.855 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:13:30.855 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:30.856 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:30.856 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:30.856 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:13:30.856 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:13:30.857 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:30.857 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:30.858 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:13:30.858 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:30.859 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:13:30.866 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:13:30.866 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:13:30.867 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:30.867 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:13:30Z" 2022-09-09T23:13:30.868 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:13:30.869 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:13:30Z" } 2022-09-09T23:13:30.870 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:13:30.871 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:13:30.871 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:13:30.872 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:13:35.873 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:13:35.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:13:35.875 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:13:35.915 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:13:35.916 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:13:35.985 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:13:35.983+0000 7fcf55e1cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:13:35.987 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:13:35.983+0000 7fcf55e1cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:13:36.015 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:13:36.015 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:13:36.016 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:13:36.016 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:13:36.016 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:13:36.016 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:13:36.017 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:13:36.017 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:36.017 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:13:36.017 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:36.018 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:13:36.018 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:36.018 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:36.019 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:13:36.019 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:36.019 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:13:36.019 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:36.020 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:13:36.020 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:36.020 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:36.020 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:36.021 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:36.021 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:13:36.021 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:36.021 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:36.022 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:13:36.023 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:36.023 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:36.023 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:36.024 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:13:36.024 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:36.024 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:36.025 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:36.025 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:36.025 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:13:36.025 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:36.026 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:36.026 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:13:36.026 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:36.026 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:36.027 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:36.027 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:13:36.027 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:36.027 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:36.028 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:36.028 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:36.028 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:13:36.028 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:36.029 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:36.029 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:13:36.029 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:36.029 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:36.030 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:36.030 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:13:36.030 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:13:36.031 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:36.031 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:36.031 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:13:36.031 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:36.032 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:13:36.032 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:13:36.032 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:13:36.032 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:36.033 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:13:35Z" 2022-09-09T23:13:36.034 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:13:36.035 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:13:35Z" } 2022-09-09T23:13:36.036 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:13:36.037 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:13:36.037 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:13:36.038 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:13:36.262 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~191s 2022-09-09T23:13:41.039 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:13:41.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:13:41.041 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:13:41.085 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:13:41.086 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:13:41.153 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:13:41.155+0000 7fc120729dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:13:41.155 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:13:41.155+0000 7fc120729dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:13:41.184 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:13:41.184 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:13:41.185 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:13:41.185 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:13:41.185 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:13:41.185 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:13:41.186 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:13:41.186 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:41.186 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:13:41.187 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:41.187 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:13:41.187 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:41.187 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:41.188 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:13:41.188 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:41.188 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:13:41.188 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:41.189 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:13:41.189 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:41.189 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:41.189 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:41.190 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:41.190 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:13:41.190 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:41.191 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:41.191 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:13:41.191 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:41.192 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:41.192 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:41.192 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:13:41.192 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:41.193 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:41.193 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:41.193 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:41.193 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:13:41.194 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:41.194 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:41.194 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:13:41.195 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:41.195 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:41.195 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:41.196 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:13:41.196 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:41.196 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:41.196 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:41.197 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:41.197 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:13:41.197 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:41.197 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:41.198 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:13:41.198 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:41.199 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:41.199 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:41.199 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:13:41.199 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:13:41.200 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:41.200 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:41.200 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:13:41.201 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:41.201 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:13:41.201 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:13:41.202 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:13:41.202 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:41.202 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:13:41Z" 2022-09-09T23:13:41.203 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:13:41.204 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:13:41Z" } 2022-09-09T23:13:41.204 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:13:41.205 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:13:41.205 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:13:41.206 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:13:41.868 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~196s 2022-09-09T23:13:46.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:13:46.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:13:46.209 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:13:46.252 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:13:46.254 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:13:46.326 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:13:46.323+0000 7fee8b475dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:13:46.327 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:13:46.327+0000 7fee8b475dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:13:46.355 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:13:46.357 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:13:46.357 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:13:46.358 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:13:46.359 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:13:46.359 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:13:46.360 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:13:46.361 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:46.361 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:13:46.362 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:46.363 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:13:46.364 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:46.366 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:46.366 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:13:46.367 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:46.368 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:13:46.369 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:46.369 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:13:46.371 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:46.372 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:46.373 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:46.373 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:46.374 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:13:46.374 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:46.375 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:46.375 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:13:46.376 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:46.376 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:46.376 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:46.377 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:13:46.377 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:46.378 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:46.378 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:46.379 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:46.379 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:13:46.381 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:46.381 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:46.382 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:13:46.383 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:46.383 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:46.383 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:46.384 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:13:46.384 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:46.384 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:46.384 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:46.385 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:46.385 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:13:46.385 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:46.385 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:46.386 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:13:46.386 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:46.386 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:46.386 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:46.387 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:13:46.387 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:13:46.387 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:46.387 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:46.388 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:13:46.388 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:46.388 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:13:46.388 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:13:46.389 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:13:46.389 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:46.390 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:13:46Z" 2022-09-09T23:13:46.390 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:13:46.396 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:13:46Z" } 2022-09-09T23:13:46.398 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:13:46.399 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:13:46.400 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:13:46.401 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:13:47.474 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~202s 2022-09-09T23:13:51.402 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:13:51.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:13:51.403 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:13:51.451 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:13:51.452 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:13:51.514 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:13:51.515+0000 7f73ed8e9dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:13:51.516 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:13:51.519+0000 7f73ed8e9dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:13:51.543 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:13:51.544 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:13:51.544 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:13:51.544 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:13:51.544 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:13:51.545 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:13:51.545 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:13:51.545 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:51.545 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:13:51.546 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:51.546 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:13:51.546 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:51.546 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:51.547 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:13:51.547 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:51.547 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:13:51.548 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:51.548 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:13:51.548 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:51.548 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:51.549 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:51.549 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:51.549 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:13:51.550 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:51.550 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:51.550 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:13:51.550 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:51.551 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:51.551 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:51.551 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:13:51.551 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:51.552 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:51.552 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:51.552 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:51.552 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:13:51.553 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:51.553 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:51.553 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:13:51.553 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:51.554 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:51.554 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:51.554 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:13:51.555 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:51.555 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:51.555 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:51.555 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:51.556 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:13:51.556 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:51.556 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:51.556 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:13:51.557 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:51.557 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:51.557 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:51.558 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:13:51.558 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:13:51.558 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:51.558 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:51.559 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:13:51.559 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:51.560 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:13:51.560 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:13:51.560 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:13:51.561 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:51.561 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:13:51Z" 2022-09-09T23:13:51.561 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:13:51.562 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:13:51Z" } 2022-09-09T23:13:51.564 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:13:51.564 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:13:51.565 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:13:51.565 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:13:53.079 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~208s 2022-09-09T23:13:56.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:13:56.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:13:56.569 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:13:56.620 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:13:56.621 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:13:56.675 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:13:56.675+0000 7fce7fdc5dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:13:56.677 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:13:56.679+0000 7fce7fdc5dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:13:56.705 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:13:56.705 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:13:56.706 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:13:56.706 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:13:56.707 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:13:56.707 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:13:56.708 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:13:56.708 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:56.709 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:13:56.709 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:56.710 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:13:56.711 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:56.711 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:56.711 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:13:56.712 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:56.712 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:13:56.713 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:56.713 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:13:56.713 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:56.713 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:56.714 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:56.714 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:56.714 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:13:56.714 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:56.715 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:56.715 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:13:56.715 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:56.716 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:56.716 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:56.716 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:13:56.716 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:56.717 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:56.717 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:56.717 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:56.717 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:13:56.718 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:56.718 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:56.718 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:13:56.718 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:56.719 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:56.719 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:56.719 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:13:56.720 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:13:56.720 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:56.720 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:56.721 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:13:56.721 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:13:56.721 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:13:56.721 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:13:56.722 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:13:56.722 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:13:56.722 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:13:56.722 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:13:56.723 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:13:56.723 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:13:56.723 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:56.724 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:13:56.724 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:13:56.724 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:56.724 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:13:56.724 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:13:56.725 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:13:56.725 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:13:56.725 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:13:56Z" 2022-09-09T23:13:56.725 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:13:56.727 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:13:56Z" } 2022-09-09T23:13:56.727 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:13:56.728 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:13:56.729 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:13:56.729 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:13:58.687 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~213s 2022-09-09T23:14:01.730 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:14:01.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:14:01.733 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:14:01.780 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:14:01.781 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:14:01.839 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:14:01.838+0000 7fb1a7236dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:14:01.841 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:14:01.842+0000 7fb1a7236dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:14:01.869 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:14:01.870 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:14:01.870 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:14:01.871 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:14:01.871 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:14:01.872 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:14:01.872 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:14:01.872 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:01.873 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:14:01.873 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:01.873 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:14:01.874 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:01.874 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:01.874 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:14:01.875 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:01.875 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:14:01.875 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:01.876 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:14:01.876 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:01.876 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:01.876 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:01.877 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:01.877 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:14:01.877 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:01.878 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:01.878 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:14:01.879 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:01.879 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:01.880 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:01.881 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:14:01.881 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:01.882 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:01.882 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:01.882 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:01.883 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:14:01.883 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:01.884 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:01.884 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:14:01.885 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:01.885 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:01.886 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:01.886 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:14:01.886 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:01.886 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:01.887 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:01.887 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:01.887 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:14:01.888 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:01.888 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:01.888 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:14:01.888 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:01.889 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:01.890 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:01.890 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:14:01.891 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:14:01.891 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:01.892 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:01.892 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:14:01.893 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:01.894 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:14:01.894 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:14:01.895 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:14:01.896 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:01.896 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:14:01Z" 2022-09-09T23:14:01.897 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:14:01.899 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:14:01Z" } 2022-09-09T23:14:01.899 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:14:01.900 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:14:01.900 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:14:01.902 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:14:04.295 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~219s 2022-09-09T23:14:06.904 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:14:06.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:14:06.905 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:14:06.952 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:14:06.953 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:14:07.008 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:14:07.006+0000 7f92566f7dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:14:07.010 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:14:07.010+0000 7f92566f7dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:14:07.035 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:14:07.036 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:14:07.036 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:14:07.036 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:14:07.037 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:14:07.037 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:14:07.037 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:14:07.038 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:07.038 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:14:07.039 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:07.039 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:14:07.039 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:07.039 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:07.040 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:14:07.040 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:07.040 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:14:07.041 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:07.041 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:14:07.042 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:07.042 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:07.044 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:07.045 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:07.046 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:14:07.047 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:07.052 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:07.056 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:14:07.056 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:07.057 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:07.057 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:07.057 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:14:07.057 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:07.058 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:07.058 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:07.058 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:07.059 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:14:07.059 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:07.059 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:07.059 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:14:07.060 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:07.060 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:07.061 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:07.061 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:14:07.061 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:07.061 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:07.062 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:07.062 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:07.062 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:14:07.062 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:07.063 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:07.063 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:14:07.063 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:07.064 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:07.064 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:07.064 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:14:07.064 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:14:07.065 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:07.065 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:07.065 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:14:07.065 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:07.066 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:14:07.066 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:14:07.066 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:14:07.066 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:07.067 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:14:07Z" 2022-09-09T23:14:07.067 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:14:07.068 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:14:07Z" } 2022-09-09T23:14:07.069 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:14:07.069 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:14:07.071 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:14:07.071 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:14:09.903 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~224s 2022-09-09T23:14:12.073 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:14:12.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:14:12.075 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:14:12.123 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:14:12.124 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:14:12.179 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:14:12.178+0000 7fd5778f7dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:14:12.183 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:14:12.182+0000 7fd5778f7dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:14:12.205 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:14:12.206 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:14:12.206 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:14:12.206 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:14:12.207 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:14:12.207 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:14:12.207 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:14:12.208 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:12.208 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:14:12.208 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:12.209 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:14:12.209 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:12.209 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:12.209 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:14:12.212 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:12.212 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:14:12.213 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:12.213 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:14:12.214 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:12.215 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:12.215 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:12.215 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:12.216 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:14:12.216 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:12.216 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:12.216 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:14:12.217 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:12.217 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:12.217 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:12.218 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:14:12.218 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:12.218 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:12.219 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:12.219 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:12.219 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:14:12.219 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:12.220 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:12.220 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:14:12.220 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:12.220 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:12.221 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:12.221 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:14:12.225 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:12.225 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:12.225 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:12.225 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:12.226 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:14:12.226 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:12.226 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:12.226 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:14:12.227 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:12.227 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:12.227 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:12.227 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:14:12.228 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:14:12.228 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:12.228 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:12.228 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:14:12.229 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:12.230 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:14:12.231 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:14:12.231 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:14:12.232 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:12.232 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:14:12Z" 2022-09-09T23:14:12.233 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:14:12.234 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:14:12Z" } 2022-09-09T23:14:12.236 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:14:12.237 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:14:12.237 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:14:12.238 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:14:15.510 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~230s 2022-09-09T23:14:17.239 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:14:17.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:14:17.242 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:14:17.307 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:14:17.310 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:14:17.373 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:14:17.374+0000 7fbfcf260dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:14:17.375 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:14:17.378+0000 7fbfcf260dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:14:17.407 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:14:17.408 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:14:17.408 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:14:17.409 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:14:17.409 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:14:17.410 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:14:17.410 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:14:17.412 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:17.412 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:14:17.414 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:17.416 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:14:17.417 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:17.417 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:17.419 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:14:17.419 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:17.420 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:14:17.421 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:17.427 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:14:17.429 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:17.429 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:17.430 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:17.430 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:17.431 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:14:17.431 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:17.432 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:17.432 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:14:17.433 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:17.436 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:17.437 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:17.437 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:14:17.438 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:17.438 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:17.439 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:17.439 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:17.439 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:14:17.440 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:17.440 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:17.440 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:14:17.440 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:17.441 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:17.441 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:17.441 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:14:17.441 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:17.442 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:17.442 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:17.443 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:17.443 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:14:17.443 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:17.443 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:17.444 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:14:17.444 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:17.444 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:17.444 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:17.445 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:14:17.445 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:14:17.446 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:17.446 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:17.447 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:14:17.447 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:17.447 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:14:17.448 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:14:17.448 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:14:17.448 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:17.448 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:14:17Z" 2022-09-09T23:14:17.449 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:14:17.449 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:14:17Z" } 2022-09-09T23:14:17.450 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:14:17.450 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:14:17.451 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:14:17.452 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:14:21.119 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~236s 2022-09-09T23:14:22.453 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:14:22.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:14:22.455 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:14:22.503 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:14:22.504 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:14:22.555 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:14:22.554+0000 7f0545e15dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:14:22.558 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:14:22.558+0000 7f0545e15dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:14:22.583 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:14:22.584 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:14:22.585 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:14:22.585 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:14:22.585 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:14:22.586 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:14:22.586 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:14:22.587 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:22.587 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:14:22.587 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:22.588 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:14:22.588 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:22.588 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:22.588 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:14:22.589 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:22.589 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:14:22.589 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:22.589 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:14:22.590 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:22.590 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:22.590 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:22.591 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:22.591 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:14:22.593 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:22.593 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:22.594 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:14:22.594 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:22.595 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:22.596 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:22.596 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:14:22.597 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:22.597 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:22.598 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:22.598 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:22.599 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:14:22.599 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:22.600 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:22.600 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:14:22.601 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:22.601 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:22.602 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:22.602 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:14:22.603 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:22.604 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:22.604 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:22.605 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:22.605 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:14:22.606 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:22.606 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:22.607 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:14:22.607 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:22.608 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:22.609 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:22.611 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:14:22.612 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:14:22.612 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:22.612 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:22.613 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:14:22.613 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:22.613 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:14:22.613 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:14:22.620 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:14:22.621 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:22.622 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:14:22Z" 2022-09-09T23:14:22.622 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:14:22.624 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:14:22Z" } 2022-09-09T23:14:22.625 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:14:22.625 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:14:22.626 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:14:22.626 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:14:26.727 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~241s 2022-09-09T23:14:27.628 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:14:27.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:14:27.630 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:14:27.679 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:14:27.679 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:14:27.734 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:14:27.734+0000 7f742c5b2dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:14:27.736 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:14:27.734+0000 7f742c5b2dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:14:27.763 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:14:27.764 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:14:27.764 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:14:27.765 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:14:27.765 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:14:27.765 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:14:27.766 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:14:27.766 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:27.767 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:14:27.767 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:27.768 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:14:27.768 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:27.769 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:27.769 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:14:27.770 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:27.770 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:14:27.771 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:27.771 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:14:27.772 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:27.772 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:27.773 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:27.773 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:27.774 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:14:27.775 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:27.775 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:27.776 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:14:27.777 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:27.781 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:27.782 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:27.782 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:14:27.783 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:27.783 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:27.784 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:27.785 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:27.786 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:14:27.786 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:27.787 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:27.787 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:14:27.788 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:27.788 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:27.789 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:27.789 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:14:27.790 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:27.790 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:27.791 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:27.791 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:27.792 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:14:27.792 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:27.793 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:27.794 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:14:27.794 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:27.795 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:27.795 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:27.796 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:14:27.797 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:14:27.797 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:27.799 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:27.800 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:14:27.800 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:27.812 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:14:27.812 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:14:27.812 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:14:27.813 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:27.813 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:14:27Z" 2022-09-09T23:14:27.814 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:14:27.817 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:14:27Z" } 2022-09-09T23:14:27.818 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:14:27.819 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:14:27.820 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:14:27.820 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:14:32.337 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~247s 2022-09-09T23:14:32.822 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:14:32.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:14:32.823 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:14:32.869 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:14:32.917 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:14:32.934 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:14:32.934+0000 7f585e058dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:14:32.942 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:14:32.934+0000 7f585e058dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:14:32.963 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:14:32.964 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:14:32.965 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:14:32.965 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:14:32.965 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:14:32.966 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:14:32.967 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:14:32.967 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:32.968 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:14:32.968 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:32.969 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:14:32.969 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:32.970 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:32.971 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:14:32.971 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:32.972 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:14:32.972 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:32.973 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:14:32.973 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:32.974 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:32.974 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:32.975 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:32.975 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:14:32.976 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:32.976 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:32.978 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:14:32.978 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:32.979 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:32.979 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:32.980 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:14:32.981 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:32.982 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:32.982 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:32.983 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:32.984 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:14:32.984 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:32.985 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:32.985 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:14:32.986 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:32.986 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:32.987 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:32.987 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:14:32.988 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:32.988 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:32.988 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:32.989 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:32.989 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:14:32.990 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:32.990 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:32.991 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:14:32.991 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:32.992 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:32.992 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:32.993 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:14:32.993 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:14:32.994 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:32.994 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:32.995 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:14:32.995 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:32.996 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:14:32.996 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:14:32.997 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:14:32.997 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:32.998 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:14:32Z" 2022-09-09T23:14:32.998 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:14:33.001 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:14:32Z" } 2022-09-09T23:14:33.002 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:14:33.003 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:14:33.004 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:14:33.005 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:14:37.942 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~252s 2022-09-09T23:14:38.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:14:38.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:14:38.008 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:14:38.058 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:14:38.059 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:14:38.129 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:14:38.130+0000 7fada0c4edc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:14:38.130 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:14:38.130+0000 7fada0c4edc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:14:38.159 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:14:38.160 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:14:38.160 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:14:38.161 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:14:38.161 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:14:38.162 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:14:38.163 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:14:38.163 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:38.164 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:14:38.164 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:38.165 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:14:38.165 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:38.166 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:38.166 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:14:38.167 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:38.167 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:14:38.167 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:38.167 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:14:38.168 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:38.168 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:38.168 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:38.168 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:38.169 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:14:38.169 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:38.169 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:38.169 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:14:38.170 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:38.170 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:38.170 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:38.170 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:14:38.171 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:38.171 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:38.171 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:38.171 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:38.172 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:14:38.172 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:38.172 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:38.172 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:14:38.173 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:38.173 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:38.173 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:38.173 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:14:38.174 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:38.174 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:38.174 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:38.174 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:38.175 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:14:38.175 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:38.175 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:38.176 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:14:38.176 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:38.176 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:38.177 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:38.177 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:14:38.177 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:14:38.177 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:38.178 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:38.178 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:14:38.179 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:38.179 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:14:38.179 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:14:38.180 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:14:38.180 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:38.180 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:14:38Z" 2022-09-09T23:14:38.180 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:14:38.182 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:14:38Z" } 2022-09-09T23:14:38.182 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:14:38.183 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:14:38.183 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:14:38.184 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:14:43.185 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:14:43.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:14:43.188 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:14:43.239 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:14:43.239 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:14:43.295 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:14:43.294+0000 7f36449f7dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:14:43.297 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:14:43.298+0000 7f36449f7dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:14:43.323 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:14:43.324 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:14:43.324 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:14:43.325 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:14:43.325 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:14:43.326 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:14:43.326 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:14:43.327 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:43.327 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:14:43.328 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:43.328 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:14:43.329 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:43.329 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:43.330 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:14:43.330 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:43.331 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:14:43.331 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:43.332 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:14:43.332 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:43.332 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:43.333 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:43.333 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:43.333 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:14:43.333 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:43.334 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:43.334 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:14:43.334 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:43.334 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:43.335 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:43.335 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:14:43.335 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:43.335 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:43.336 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:43.336 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:43.336 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:14:43.336 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:43.337 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:43.337 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:14:43.337 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:43.338 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:43.338 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:43.338 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:14:43.338 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:43.339 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:43.339 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:43.339 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:43.339 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:14:43.340 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:43.340 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:43.340 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:14:43.340 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:43.341 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:43.341 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:43.341 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:14:43.341 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:14:43.342 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:43.342 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:43.342 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:14:43.343 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:43.343 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:14:43.343 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:14:43.343 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:14:43.344 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:43.344 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:14:43Z" 2022-09-09T23:14:43.344 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:14:43.346 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:14:43Z" } 2022-09-09T23:14:43.346 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:14:43.347 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:14:43.347 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:14:43.348 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:14:43.550 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~258s 2022-09-09T23:14:48.350 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:14:48.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:14:48.353 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:14:48.392 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:14:48.393 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:14:48.459 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:14:48.458+0000 7ff07616cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:14:48.461 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:14:48.462+0000 7ff07616cdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:14:48.493 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:14:48.493 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:14:48.494 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:14:48.494 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:14:48.494 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:14:48.495 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:14:48.495 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:14:48.495 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:48.496 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:14:48.496 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:48.496 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:14:48.497 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:48.497 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:48.497 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:14:48.497 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:48.498 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:14:48.498 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:48.498 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:14:48.499 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:48.499 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:48.499 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:48.499 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:48.500 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:14:48.500 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:48.500 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:48.500 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:14:48.501 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:48.501 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:48.501 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:48.501 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:14:48.502 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:48.502 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:48.502 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:48.502 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:48.503 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:14:48.503 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:48.503 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:48.503 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:14:48.504 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:48.504 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:48.504 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:48.504 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:14:48.505 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:48.506 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:48.506 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:48.506 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:48.506 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:14:48.507 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:48.507 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:48.507 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:14:48.507 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:48.508 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:48.508 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:48.508 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:14:48.508 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:14:48.509 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:48.510 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:48.510 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:14:48.510 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:48.510 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:14:48.511 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:14:48.511 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:14:48.511 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:48.511 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:14:48Z" 2022-09-09T23:14:48.512 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:14:48.513 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:14:48Z" } 2022-09-09T23:14:48.514 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:14:48.515 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:14:48.515 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:14:48.516 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:14:49.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~264s 2022-09-09T23:14:53.518 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:14:53.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:14:53.522 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:14:53.576 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:14:53.576 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:14:53.641 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:14:53.641+0000 7f694dc83dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:14:53.643 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:14:53.645+0000 7f694dc83dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:14:53.672 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:14:53.673 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:14:53.673 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:14:53.673 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:14:53.674 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:14:53.674 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:14:53.674 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:14:53.674 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:53.675 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:14:53.675 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:53.675 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:14:53.675 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:53.676 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:53.676 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:14:53.676 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:53.677 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:14:53.677 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:53.677 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:14:53.677 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:53.678 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:53.678 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:53.678 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:53.678 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:14:53.679 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:53.679 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:53.679 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:14:53.679 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:53.680 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:53.680 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:53.680 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:14:53.680 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:53.681 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:53.681 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:53.681 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:53.682 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:14:53.682 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:53.682 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:53.682 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:14:53.683 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:53.683 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:53.683 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:53.683 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:14:53.684 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:53.684 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:53.684 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:53.684 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:53.685 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:14:53.685 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:53.685 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:53.685 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:14:53.686 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:53.686 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:53.686 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:53.686 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:14:53.687 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:14:53.687 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:53.687 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:53.688 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:14:53.688 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:53.688 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:14:53.688 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:14:53.689 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:14:53.689 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:53.689 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:14:53Z" 2022-09-09T23:14:53.689 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:14:53.691 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:14:53Z" } 2022-09-09T23:14:53.691 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:14:53.692 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:14:53.693 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:14:53.693 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:14:54.763 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~269s 2022-09-09T23:14:58.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:14:58.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:14:58.697 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:14:58.757 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:14:58.757 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:14:58.812 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:14:58.813+0000 7fbb612dcdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:14:58.813 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:14:58.813+0000 7fbb612dcdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:14:58.839 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:14:58.839 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:14:58.840 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:14:58.840 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:14:58.840 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:14:58.840 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:14:58.841 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:14:58.841 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:58.841 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:14:58.841 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:58.842 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:14:58.842 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:58.842 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:58.843 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:14:58.843 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:58.843 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:14:58.843 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:58.844 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:14:58.844 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:58.845 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:58.845 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:58.845 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:58.846 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:14:58.846 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:58.846 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:58.846 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:14:58.847 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:58.847 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:58.847 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:58.848 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:14:58.848 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:58.848 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:58.848 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:58.849 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:58.849 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:14:58.849 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:58.849 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:58.850 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:14:58.850 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:58.850 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:58.850 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:58.851 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:14:58.851 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:14:58.851 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:58.851 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:58.852 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:14:58.852 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:14:58.852 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:14:58.853 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:14:58.853 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:14:58.853 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:14:58.853 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:14:58.854 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:14:58.854 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:14:58.854 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:14:58.855 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:58.855 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:14:58.856 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:14:58.856 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:58.856 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:14:58.856 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:14:58.857 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:14:58.857 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:14:58.857 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:14:58Z" 2022-09-09T23:14:58.858 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:14:58.859 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:14:58Z" } 2022-09-09T23:14:58.860 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:14:58.860 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:14:58.861 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:14:58.861 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:15:00.369 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~275s 2022-09-09T23:15:03.863 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:15:03.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:15:03.865 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:15:03.907 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:15:03.907 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:15:03.986 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:15:03.985+0000 7f99089bddc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:15:03.988 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:15:03.989+0000 7f99089bddc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:15:04.016 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:15:04.017 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:15:04.017 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:15:04.019 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:15:04.020 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:15:04.020 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:15:04.021 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:15:04.021 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:15:04.022 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:15:04.022 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:15:04.023 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:15:04.023 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:15:04.023 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:15:04.024 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:15:04.024 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:15:04.025 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:15:04.025 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:15:04.027 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:15:04.027 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:15:04.028 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:15:04.028 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:15:04.029 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:15:04.029 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:15:04.030 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:15:04.030 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:15:04.031 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:15:04.031 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:15:04.032 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:15:04.032 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:15:04.032 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:15:04.033 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:15:04.033 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:15:04.034 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:15:04.035 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:15:04.035 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:15:04.036 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:15:04.036 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:15:04.038 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:15:04.038 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:15:04.039 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:15:04.039 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:15:04.040 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:15:04.040 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:15:04.040 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:15:04.041 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:15:04.041 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:15:04.042 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:15:04.043 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:15:04.043 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:15:04.043 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:15:04.044 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:15:04.044 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:15:04.045 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:15:04.046 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:15:04.046 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:15:04.046 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:15:04.047 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:15:04.047 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:15:04.048 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:15:04.049 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:15:04.049 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:15:04.049 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:15:04.050 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:15:04.050 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:15:03Z" 2022-09-09T23:15:04.050 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:15:04.051 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:15:03Z" } 2022-09-09T23:15:04.052 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:15:04.053 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:15:04.053 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:15:04.054 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:15:05.975 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~280s 2022-09-09T23:15:09.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:15:09.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:15:09.058 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:15:09.107 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:15:09.108 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:15:09.168 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:15:09.169+0000 7f6b40c19dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:15:09.170 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:15:09.169+0000 7f6b40c19dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:15:09.199 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:15:09.200 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:15:09.200 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:15:09.201 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:15:09.201 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:15:09.202 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:15:09.202 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:15:09.203 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:15:09.203 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:15:09.204 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:15:09.204 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:15:09.205 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:15:09.205 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:15:09.206 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:15:09.206 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:15:09.207 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:15:09.208 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:15:09.208 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:15:09.209 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:15:09.209 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:15:09.210 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:15:09.210 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:15:09.211 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:15:09.211 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:15:09.212 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:15:09.215 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:15:09.216 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:15:09.216 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:15:09.217 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:15:09.217 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:15:09.217 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:15:09.218 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:15:09.218 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:15:09.218 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:15:09.219 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:15:09.219 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:15:09.220 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:15:09.220 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:15:09.221 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:15:09.221 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:15:09.221 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:15:09.221 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:15:09.222 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:15:09.222 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:15:09.223 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:15:09.223 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:15:09.223 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:15:09.224 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:15:09.224 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:15:09.224 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:15:09.224 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:15:09.225 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:15:09.225 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:15:09.225 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:15:09.225 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:15:09.226 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:15:09.226 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:15:09.226 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:15:09.227 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:15:09.227 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:15:09.227 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:15:09.227 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:15:09.228 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:15:09.228 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:15:09Z" 2022-09-09T23:15:09.229 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:15:09.231 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:15:09Z" } 2022-09-09T23:15:09.232 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:15:09.233 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:15:09.234 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:15:09.235 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:15:11.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~286s 2022-09-09T23:15:14.237 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:15:14.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:15:14.239 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:15:14.289 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:15:14.290 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:15:14.351 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:15:14.349+0000 7fc403ddfdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:15:14.353 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:15:14.353+0000 7fc403ddfdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:15:14.381 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:15:14.382 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:15:14.382 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:15:14.382 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:15:14.383 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:15:14.383 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:15:14.383 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:15:14.383 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:15:14.384 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:15:14.384 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:15:14.384 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:15:14.384 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:15:14.385 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:15:14.385 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:15:14.385 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:15:14.385 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:15:14.386 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:15:14.386 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:15:14.386 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:15:14.387 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:15:14.387 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:15:14.387 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:15:14.387 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:15:14.388 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:15:14.388 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:15:14.388 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:15:14.388 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:15:14.389 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:15:14.389 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:15:14.389 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:15:14.389 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:15:14.390 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:15:14.390 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:15:14.390 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:15:14.390 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:15:14.391 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:15:14.391 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:15:14.391 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:15:14.391 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:15:14.392 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:15:14.392 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:15:14.392 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:15:14.392 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:15:14.393 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:15:14.393 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:15:14.393 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:15:14.393 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:15:14.394 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:15:14.394 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:15:14.394 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:15:14.395 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:15:14.395 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:15:14.395 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:15:14.395 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:15:14.395 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:15:14.396 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:15:14.396 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:15:14.396 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:15:14.397 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:15:14.397 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:15:14.397 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:15:14.397 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:15:14.398 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:15:14.398 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:15:14Z" 2022-09-09T23:15:14.398 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:15:14.399 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:15:14Z" } 2022-09-09T23:15:14.400 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:15:14.400 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:15:14.401 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:15:14.401 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:15:17.185 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~292s 2022-09-09T23:15:19.403 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:15:19.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:15:19.405 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:15:19.463 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:15:19.464 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:15:19.517 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:15:19.517+0000 7f9f3c1acdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:15:19.519 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:15:19.517+0000 7f9f3c1acdc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:15:19.545 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:15:19.546 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:15:19.546 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:15:19.547 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:15:19.547 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:15:19.547 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:15:19.547 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:15:19.548 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:15:19.548 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:15:19.549 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:15:19.549 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:15:19.549 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:15:19.550 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:15:19.550 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:15:19.551 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:15:19.551 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:15:19.552 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:15:19.552 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:15:19.553 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:15:19.553 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:15:19.554 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:15:19.554 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:15:19.555 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:15:19.555 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:15:19.556 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:15:19.556 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:15:19.557 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:15:19.557 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:15:19.558 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:15:19.558 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:15:19.558 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:15:19.559 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:15:19.559 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:15:19.560 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:15:19.560 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:15:19.561 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:15:19.561 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:15:19.562 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:15:19.562 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:15:19.563 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:15:19.563 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:15:19.564 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:15:19.564 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:15:19.565 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:15:19.565 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:15:19.565 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:15:19.566 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:15:19.566 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:15:19.567 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:15:19.567 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:15:19.568 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:15:19.568 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:15:19.569 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:15:19.569 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:15:19.569 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:15:19.570 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:15:19.570 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:15:19.571 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:15:19.571 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:15:19.572 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:15:19.572 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:15:19.573 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:15:19.573 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:15:19.574 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:15:19Z" 2022-09-09T23:15:19.574 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:15:19.576 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:15:19Z" } 2022-09-09T23:15:19.578 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:15:19.579 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:15:19.579 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:15:19.580 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:15:22.791 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~297s 2022-09-09T23:15:24.582 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:15:24.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:15:24.584 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:15:24.634 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:15:24.635 INFO:teuthology.orchestra.run.smithi080.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:15:24.709 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:15:24.709+0000 7f0dc6fd6dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:15:24.710 INFO:teuthology.orchestra.run.smithi080.stderr:2022-09-09T23:15:24.709+0000 7f0dc6fd6dc0 20 RGW-SYNC:meta: read sync status 2022-09-09T23:15:24.738 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2022-09-09T23:15:24.739 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": { 2022-09-09T23:15:24.739 INFO:teuthology.orchestra.run.smithi080.stdout: "info": { 2022-09-09T23:15:24.740 INFO:teuthology.orchestra.run.smithi080.stdout: "status": "sync", 2022-09-09T23:15:24.740 INFO:teuthology.orchestra.run.smithi080.stdout: "num_shards": 4, 2022-09-09T23:15:24.740 INFO:teuthology.orchestra.run.smithi080.stdout: "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", 2022-09-09T23:15:24.741 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:15:24.741 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:15:24.741 INFO:teuthology.orchestra.run.smithi080.stdout: "markers": [ 2022-09-09T23:15:24.741 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:15:24.742 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 0, 2022-09-09T23:15:24.742 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:15:24.742 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:15:24.743 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762486.146147_45.1", 2022-09-09T23:15:24.743 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:15:24.743 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 1, 2022-09-09T23:15:24.743 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:15:24.743 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:28:06.146147Z", 2022-09-09T23:15:24.744 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:15:24.744 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:15:24.744 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:15:24.745 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:15:24.745 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 1, 2022-09-09T23:15:24.745 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:15:24.745 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:15:24.746 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "", 2022-09-09T23:15:24.746 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:15:24.746 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:15:24.747 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:15:24.747 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "0.000000", 2022-09-09T23:15:24.747 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:15:24.747 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:15:24.748 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:15:24.748 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:15:24.748 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 2, 2022-09-09T23:15:24.749 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:15:24.749 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:15:24.749 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662762637.757014_85.1", 2022-09-09T23:15:24.749 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:15:24.750 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:15:24.750 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:15:24.750 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T22:30:37.757014Z", 2022-09-09T23:15:24.751 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2022-09-09T23:15:24.751 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:15:24.751 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:15:24.752 INFO:teuthology.orchestra.run.smithi080.stdout: { 2022-09-09T23:15:24.752 INFO:teuthology.orchestra.run.smithi080.stdout: "key": 3, 2022-09-09T23:15:24.752 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2022-09-09T23:15:24.753 INFO:teuthology.orchestra.run.smithi080.stdout: "state": 1, 2022-09-09T23:15:24.753 INFO:teuthology.orchestra.run.smithi080.stdout: "marker": "1_1662765017.660642_657.1", 2022-09-09T23:15:24.753 INFO:teuthology.orchestra.run.smithi080.stdout: "next_step_marker": "", 2022-09-09T23:15:24.753 INFO:teuthology.orchestra.run.smithi080.stdout: "total_entries": 0, 2022-09-09T23:15:24.754 INFO:teuthology.orchestra.run.smithi080.stdout: "pos": 0, 2022-09-09T23:15:24.754 INFO:teuthology.orchestra.run.smithi080.stdout: "timestamp": "2022-09-09T23:10:17.660642Z", 2022-09-09T23:15:24.754 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 4 2022-09-09T23:15:24.755 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:15:24.755 INFO:teuthology.orchestra.run.smithi080.stdout: } 2022-09-09T23:15:24.755 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2022-09-09T23:15:24.755 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:15:24.756 INFO:teuthology.orchestra.run.smithi080.stdout: "full_sync": { 2022-09-09T23:15:24.756 INFO:teuthology.orchestra.run.smithi080.stdout: "total": 1, 2022-09-09T23:15:24.756 INFO:teuthology.orchestra.run.smithi080.stdout: "complete": 1 2022-09-09T23:15:24.756 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2022-09-09T23:15:24.757 INFO:teuthology.orchestra.run.smithi080.stdout: "current_time": "2022-09-09T23:15:24Z" 2022-09-09T23:15:24.757 INFO:teuthology.orchestra.run.smithi080.stdout:} 2022-09-09T23:15:24.758 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "fd4bc158-06fd-4840-8a27-33fabda67b99", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662762486.146147_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-09T22:28:06.146147Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662762637.757014_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T22:30:37.757014Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662765017.660642_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-09T23:10:17.660642Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-09T23:15:24Z" } 2022-09-09T23:15:24.759 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662762486.146147_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-09T22:28:06.146147Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662762637.757014_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T22:30:37.757014Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662765017.660642_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-09T23:10:17.660642Z', 'realm_epoch': 4}}] 2022-09-09T23:15:24.760 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '', 3: '1_1662765017.749301_664.1'} 2022-09-09T23:15:24.760 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '', 3: '1_1662765017.660642_657.1'} 2022-09-09T23:15:24.761 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1662765017.749301_664.1 target=1_1662765017.660642_657.1 2022-09-09T23:15:28.398 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~303s 2022-09-09T23:15:28.398 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2022-09-09T23:15:29.007 INFO:tasks.ceph.c1.osd.0:Sent signal 15 2022-09-09T23:15:29.011 INFO:tasks.ceph.c1.osd.1:Sent signal 15 2022-09-09T23:15:29.011 INFO:tasks.ceph.c1.osd.2:Sent signal 15 2022-09-09T23:15:29.012 INFO:tasks.ceph.c1.mon.a:Sent signal 15 2022-09-09T23:15:29.013 INFO:tasks.rgw.c1.client.1:Sent signal 15 2022-09-09T23:15:29.013 INFO:tasks.ceph.c1.mgr.x:Sent signal 15 2022-09-09T23:15:29.016 INFO:tasks.ceph.c1.osd.0.smithi005.stderr:2022-09-09T23:15:29.009+0000 7f1d9e27b700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster c1 -i 0 (PID: 14732) UID: 0 2022-09-09T23:15:29.016 INFO:tasks.ceph.c1.osd.0.smithi005.stderr:2022-09-09T23:15:29.009+0000 7f1d9e27b700 -1 osd.0 58 *** Got signal Terminated *** 2022-09-09T23:15:29.016 INFO:tasks.ceph.c1.osd.0.smithi005.stderr:2022-09-09T23:15:29.009+0000 7f1d9e27b700 -1 osd.0 58 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-09-09T23:15:29.017 INFO:tasks.ceph.c1.osd.1.smithi005.stderr:2022-09-09T23:15:29.013+0000 7fc25445d700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster c1 -i 1 (PID: 14735) UID: 0 2022-09-09T23:15:29.018 INFO:tasks.ceph.c1.osd.1.smithi005.stderr:2022-09-09T23:15:29.013+0000 7fc25445d700 -1 osd.1 58 *** Got signal Terminated *** 2022-09-09T23:15:29.019 INFO:tasks.ceph.c1.osd.1.smithi005.stderr:2022-09-09T23:15:29.013+0000 7fc25445d700 -1 osd.1 58 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-09-09T23:15:29.020 INFO:tasks.ceph.c1.osd.2.smithi005.stderr:2022-09-09T23:15:29.013+0000 7f1000226700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster c1 -i 2 (PID: 14738) UID: 0 2022-09-09T23:15:29.020 INFO:tasks.ceph.c1.osd.2.smithi005.stderr:2022-09-09T23:15:29.013+0000 7f1000226700 -1 osd.2 58 *** Got signal Terminated *** 2022-09-09T23:15:29.020 INFO:tasks.ceph.c1.osd.2.smithi005.stderr:2022-09-09T23:15:29.013+0000 7f1000226700 -1 osd.2 58 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-09-09T23:15:29.021 INFO:tasks.ceph.c1.mon.a.smithi005.stderr:2022-09-09T23:15:29.013+0000 7f856b8cf700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster c1 -i a (PID: 14552) UID: 0 2022-09-09T23:15:29.022 INFO:tasks.ceph.c1.mon.a.smithi005.stderr:2022-09-09T23:15:29.013+0000 7f856b8cf700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2022-09-09T23:15:29.023 INFO:tasks.rgw.c1.client.1.smithi005.stdout:2022-09-09T23:15:29.013+0000 7f5fe8490700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 31950) UID: 0 2022-09-09T23:15:29.023 INFO:tasks.rgw.c1.client.1.smithi005.stdout:2022-09-09T23:15:29.013+0000 7f5febceedc0 -1 shutting down 2022-09-09T23:15:29.024 INFO:tasks.ceph.c1.mgr.x.smithi005.stderr:2022-09-09T23:15:29.013+0000 7eff216e0700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mgr -f --cluster c1 -i x (PID: 14555) UID: 0 2022-09-09T23:15:29.026 INFO:tasks.ceph.c1.mgr.x.smithi005.stderr:2022-09-09T23:15:29.013+0000 7eff216e0700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2022-09-09T23:15:29.770 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... FAIL 2022-09-09T23:15:29.774 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jkcgbj-62 2022-09-09T23:15:29.775 DEBUG:boto:path=/jkcgbj-62/ 2022-09-09T23:15:29.777 DEBUG:boto:auth_path=/jkcgbj-62/ 2022-09-09T23:15:29.778 DEBUG:boto:Method: PUT 2022-09-09T23:15:29.778 DEBUG:boto:Path: /jkcgbj-62/ 2022-09-09T23:15:29.779 DEBUG:boto:Data: 2022-09-09T23:15:29.780 DEBUG:boto:Headers: {} 2022-09-09T23:15:29.780 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T23:15:29.781 DEBUG:boto:Port: 8000 2022-09-09T23:15:29.781 DEBUG:boto:Params: {} 2022-09-09T23:15:29.782 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:15:29.783 DEBUG:boto:Token: None 2022-09-09T23:15:29.784 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:15:29 GMT /jkcgbj-62/ 2022-09-09T23:15:29.785 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:b4zZeY/Q9SKhXPw6aCdbJlIYvLo= 2022-09-09T23:15:29.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:15:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:b4zZeY/Q9SKhXPw6aCdbJlIYvLo=', 'Content-Length': '0'} 2022-09-09T23:15:29.789 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:15:29.794 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:15:29.870 DEBUG:boto:Token: None 2022-09-09T23:15:29.871 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:15:29 GMT /jkcgbj-62/ 2022-09-09T23:15:29.872 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:b4zZeY/Q9SKhXPw6aCdbJlIYvLo= 2022-09-09T23:15:29.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:15:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:b4zZeY/Q9SKhXPw6aCdbJlIYvLo=', 'Content-Length': '0'} 2022-09-09T23:15:29.874 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:15:29.875 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:15:30.780 DEBUG:boto:Token: None 2022-09-09T23:15:30.781 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:15:29 GMT /jkcgbj-62/ 2022-09-09T23:15:30.781 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:b4zZeY/Q9SKhXPw6aCdbJlIYvLo= 2022-09-09T23:15:30.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:15:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:b4zZeY/Q9SKhXPw6aCdbJlIYvLo=', 'Content-Length': '0'} 2022-09-09T23:15:30.784 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:15:30.785 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:15:34.751 DEBUG:boto:Token: None 2022-09-09T23:15:34.752 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:15:29 GMT /jkcgbj-62/ 2022-09-09T23:15:34.752 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:b4zZeY/Q9SKhXPw6aCdbJlIYvLo= 2022-09-09T23:15:34.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:15:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:b4zZeY/Q9SKhXPw6aCdbJlIYvLo=', 'Content-Length': '0'} 2022-09-09T23:15:34.755 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:15:34.755 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:15:34.760 DEBUG:boto:Token: None 2022-09-09T23:15:34.760 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:15:29 GMT /jkcgbj-62/ 2022-09-09T23:15:34.761 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:b4zZeY/Q9SKhXPw6aCdbJlIYvLo= 2022-09-09T23:15:34.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:15:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:b4zZeY/Q9SKhXPw6aCdbJlIYvLo=', 'Content-Length': '0'} 2022-09-09T23:15:34.765 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:15:34.766 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:15:39.274 DEBUG:boto:Token: None 2022-09-09T23:15:39.275 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:15:29 GMT /jkcgbj-62/ 2022-09-09T23:15:39.276 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:b4zZeY/Q9SKhXPw6aCdbJlIYvLo= 2022-09-09T23:15:39.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:15:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:b4zZeY/Q9SKhXPw6aCdbJlIYvLo=', 'Content-Length': '0'} 2022-09-09T23:15:39.285 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:15:39.286 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:15:54.892 DEBUG:boto:Token: None 2022-09-09T23:15:54.893 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:15:29 GMT /jkcgbj-62/ 2022-09-09T23:15:54.894 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:b4zZeY/Q9SKhXPw6aCdbJlIYvLo= 2022-09-09T23:15:54.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:15:29 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:b4zZeY/Q9SKhXPw6aCdbJlIYvLo=', 'Content-Length': '0'} 2022-09-09T23:15:54.897 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:15:54.897 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:15:57.246 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... ERROR 2022-09-09T23:15:57.247 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jkcgbj-63 2022-09-09T23:15:57.248 DEBUG:boto:path=/jkcgbj-63/ 2022-09-09T23:15:57.249 DEBUG:boto:auth_path=/jkcgbj-63/ 2022-09-09T23:15:57.249 DEBUG:boto:Method: PUT 2022-09-09T23:15:57.250 DEBUG:boto:Path: /jkcgbj-63/ 2022-09-09T23:15:57.251 DEBUG:boto:Data: 2022-09-09T23:15:57.251 DEBUG:boto:Headers: {} 2022-09-09T23:15:57.252 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T23:15:57.252 DEBUG:boto:Port: 8000 2022-09-09T23:15:57.252 DEBUG:boto:Params: {} 2022-09-09T23:15:57.253 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:15:57.254 DEBUG:boto:Token: None 2022-09-09T23:15:57.255 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:15:57 GMT /jkcgbj-63/ 2022-09-09T23:15:57.256 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:HrVuLleGR1Votw4czPeSBnddZdE= 2022-09-09T23:15:57.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:15:57 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:HrVuLleGR1Votw4czPeSBnddZdE=', 'Content-Length': '0'} 2022-09-09T23:15:57.262 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:15:57.263 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:15:58.077 DEBUG:boto:Token: None 2022-09-09T23:15:58.078 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:15:57 GMT /jkcgbj-63/ 2022-09-09T23:15:58.079 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:HrVuLleGR1Votw4czPeSBnddZdE= 2022-09-09T23:15:58.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:15:57 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:HrVuLleGR1Votw4czPeSBnddZdE=', 'Content-Length': '0'} 2022-09-09T23:15:58.082 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:15:58.083 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:15:58.666 DEBUG:boto:Token: None 2022-09-09T23:15:58.668 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:15:57 GMT /jkcgbj-63/ 2022-09-09T23:15:58.669 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:HrVuLleGR1Votw4czPeSBnddZdE= 2022-09-09T23:15:58.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:15:57 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:HrVuLleGR1Votw4czPeSBnddZdE=', 'Content-Length': '0'} 2022-09-09T23:15:58.674 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:15:58.675 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:16:02.242 DEBUG:boto:Token: None 2022-09-09T23:16:02.243 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:15:57 GMT /jkcgbj-63/ 2022-09-09T23:16:02.244 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:HrVuLleGR1Votw4czPeSBnddZdE= 2022-09-09T23:16:02.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:15:57 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:HrVuLleGR1Votw4czPeSBnddZdE=', 'Content-Length': '0'} 2022-09-09T23:16:02.246 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:16:02.247 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:16:05.051 DEBUG:boto:Token: None 2022-09-09T23:16:05.052 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:15:57 GMT /jkcgbj-63/ 2022-09-09T23:16:05.053 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:HrVuLleGR1Votw4czPeSBnddZdE= 2022-09-09T23:16:05.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:15:57 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:HrVuLleGR1Votw4czPeSBnddZdE=', 'Content-Length': '0'} 2022-09-09T23:16:05.055 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:16:05.056 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:16:14.167 DEBUG:boto:Token: None 2022-09-09T23:16:14.168 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:15:57 GMT /jkcgbj-63/ 2022-09-09T23:16:14.170 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:HrVuLleGR1Votw4czPeSBnddZdE= 2022-09-09T23:16:14.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:15:57 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:HrVuLleGR1Votw4czPeSBnddZdE=', 'Content-Length': '0'} 2022-09-09T23:16:14.173 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:16:14.174 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:16:41.764 DEBUG:boto:Token: None 2022-09-09T23:16:41.766 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:15:57 GMT /jkcgbj-63/ 2022-09-09T23:16:41.767 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:HrVuLleGR1Votw4czPeSBnddZdE= 2022-09-09T23:16:41.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:15:57 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:HrVuLleGR1Votw4czPeSBnddZdE=', 'Content-Length': '0'} 2022-09-09T23:16:41.769 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:16:41.770 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:16:44.447 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... ERROR 2022-09-09T23:16:44.448 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jkcgbj-64 2022-09-09T23:16:44.448 DEBUG:boto:path=/jkcgbj-64/ 2022-09-09T23:16:44.449 DEBUG:boto:auth_path=/jkcgbj-64/ 2022-09-09T23:16:44.449 DEBUG:boto:Method: PUT 2022-09-09T23:16:44.450 DEBUG:boto:Path: /jkcgbj-64/ 2022-09-09T23:16:44.451 DEBUG:boto:Data: 2022-09-09T23:16:44.452 DEBUG:boto:Headers: {} 2022-09-09T23:16:44.453 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T23:16:44.454 DEBUG:boto:Port: 8000 2022-09-09T23:16:44.454 DEBUG:boto:Params: {} 2022-09-09T23:16:44.455 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:16:44.456 DEBUG:boto:Token: None 2022-09-09T23:16:44.457 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:16:44 GMT /jkcgbj-64/ 2022-09-09T23:16:44.458 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0nZeKbGvh56eFxZIA/eu9PMTGlI= 2022-09-09T23:16:44.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:16:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0nZeKbGvh56eFxZIA/eu9PMTGlI=', 'Content-Length': '0'} 2022-09-09T23:16:44.461 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:16:44.463 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:16:45.082 DEBUG:boto:Token: None 2022-09-09T23:16:45.083 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:16:44 GMT /jkcgbj-64/ 2022-09-09T23:16:45.084 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0nZeKbGvh56eFxZIA/eu9PMTGlI= 2022-09-09T23:16:45.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:16:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0nZeKbGvh56eFxZIA/eu9PMTGlI=', 'Content-Length': '0'} 2022-09-09T23:16:45.086 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:16:45.086 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:16:45.494 DEBUG:boto:Token: None 2022-09-09T23:16:45.495 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:16:44 GMT /jkcgbj-64/ 2022-09-09T23:16:45.495 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0nZeKbGvh56eFxZIA/eu9PMTGlI= 2022-09-09T23:16:45.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:16:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0nZeKbGvh56eFxZIA/eu9PMTGlI=', 'Content-Length': '0'} 2022-09-09T23:16:45.498 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:16:45.498 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:16:49.339 DEBUG:boto:Token: None 2022-09-09T23:16:49.340 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:16:44 GMT /jkcgbj-64/ 2022-09-09T23:16:49.342 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0nZeKbGvh56eFxZIA/eu9PMTGlI= 2022-09-09T23:16:49.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:16:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0nZeKbGvh56eFxZIA/eu9PMTGlI=', 'Content-Length': '0'} 2022-09-09T23:16:49.345 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:16:49.346 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:16:54.064 DEBUG:boto:Token: None 2022-09-09T23:16:54.065 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:16:44 GMT /jkcgbj-64/ 2022-09-09T23:16:54.066 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0nZeKbGvh56eFxZIA/eu9PMTGlI= 2022-09-09T23:16:54.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:16:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0nZeKbGvh56eFxZIA/eu9PMTGlI=', 'Content-Length': '0'} 2022-09-09T23:16:54.068 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:16:54.069 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:17:04.250 DEBUG:boto:Token: None 2022-09-09T23:17:04.252 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:16:44 GMT /jkcgbj-64/ 2022-09-09T23:17:04.253 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0nZeKbGvh56eFxZIA/eu9PMTGlI= 2022-09-09T23:17:04.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:16:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0nZeKbGvh56eFxZIA/eu9PMTGlI=', 'Content-Length': '0'} 2022-09-09T23:17:04.258 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:17:04.259 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:17:21.176 DEBUG:boto:Token: None 2022-09-09T23:17:21.177 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:16:44 GMT /jkcgbj-64/ 2022-09-09T23:17:21.179 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:0nZeKbGvh56eFxZIA/eu9PMTGlI= 2022-09-09T23:17:21.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:16:44 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:0nZeKbGvh56eFxZIA/eu9PMTGlI=', 'Content-Length': '0'} 2022-09-09T23:17:21.181 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:17:21.182 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:18:09.939 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... ERROR 2022-09-09T23:18:09.941 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jkcgbj-65 2022-09-09T23:18:09.942 DEBUG:boto:path=/jkcgbj-65/ 2022-09-09T23:18:09.943 DEBUG:boto:auth_path=/jkcgbj-65/ 2022-09-09T23:18:09.943 DEBUG:boto:Method: PUT 2022-09-09T23:18:09.944 DEBUG:boto:Path: /jkcgbj-65/ 2022-09-09T23:18:09.944 DEBUG:boto:Data: 2022-09-09T23:18:09.945 DEBUG:boto:Headers: {} 2022-09-09T23:18:09.945 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T23:18:09.946 DEBUG:boto:Port: 8000 2022-09-09T23:18:09.947 DEBUG:boto:Params: {} 2022-09-09T23:18:09.948 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:18:09.949 DEBUG:boto:Token: None 2022-09-09T23:18:09.951 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:18:09 GMT /jkcgbj-65/ 2022-09-09T23:18:09.952 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:j5GxmRcUkFud3IQMudGC5ZY4yEE= 2022-09-09T23:18:09.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:18:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:j5GxmRcUkFud3IQMudGC5ZY4yEE=', 'Content-Length': '0'} 2022-09-09T23:18:09.956 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:18:09.958 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:18:10.665 DEBUG:boto:Token: None 2022-09-09T23:18:10.666 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:18:09 GMT /jkcgbj-65/ 2022-09-09T23:18:10.666 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:j5GxmRcUkFud3IQMudGC5ZY4yEE= 2022-09-09T23:18:10.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:18:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:j5GxmRcUkFud3IQMudGC5ZY4yEE=', 'Content-Length': '0'} 2022-09-09T23:18:10.668 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:18:10.669 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:18:11.940 DEBUG:boto:Token: None 2022-09-09T23:18:11.941 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:18:09 GMT /jkcgbj-65/ 2022-09-09T23:18:11.942 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:j5GxmRcUkFud3IQMudGC5ZY4yEE= 2022-09-09T23:18:11.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:18:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:j5GxmRcUkFud3IQMudGC5ZY4yEE=', 'Content-Length': '0'} 2022-09-09T23:18:11.945 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:18:11.946 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:18:15.344 DEBUG:boto:Token: None 2022-09-09T23:18:15.345 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:18:09 GMT /jkcgbj-65/ 2022-09-09T23:18:15.346 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:j5GxmRcUkFud3IQMudGC5ZY4yEE= 2022-09-09T23:18:15.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:18:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:j5GxmRcUkFud3IQMudGC5ZY4yEE=', 'Content-Length': '0'} 2022-09-09T23:18:15.349 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:18:15.350 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:18:22.437 DEBUG:boto:Token: None 2022-09-09T23:18:22.439 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:18:09 GMT /jkcgbj-65/ 2022-09-09T23:18:22.441 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:j5GxmRcUkFud3IQMudGC5ZY4yEE= 2022-09-09T23:18:22.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:18:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:j5GxmRcUkFud3IQMudGC5ZY4yEE=', 'Content-Length': '0'} 2022-09-09T23:18:22.445 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:18:22.446 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:18:24.146 DEBUG:boto:Token: None 2022-09-09T23:18:24.147 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:18:09 GMT /jkcgbj-65/ 2022-09-09T23:18:24.148 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:j5GxmRcUkFud3IQMudGC5ZY4yEE= 2022-09-09T23:18:24.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:18:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:j5GxmRcUkFud3IQMudGC5ZY4yEE=', 'Content-Length': '0'} 2022-09-09T23:18:24.152 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:18:24.153 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:18:47.196 DEBUG:boto:Token: None 2022-09-09T23:18:47.197 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:18:09 GMT /jkcgbj-65/ 2022-09-09T23:18:47.198 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:j5GxmRcUkFud3IQMudGC5ZY4yEE= 2022-09-09T23:18:47.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:18:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:j5GxmRcUkFud3IQMudGC5ZY4yEE=', 'Content-Length': '0'} 2022-09-09T23:18:47.201 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:18:47.201 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:19:02.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... ERROR 2022-09-09T23:19:02.464 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jkcgbj-66 2022-09-09T23:19:02.465 DEBUG:boto:path=/jkcgbj-66/ 2022-09-09T23:19:02.465 DEBUG:boto:auth_path=/jkcgbj-66/ 2022-09-09T23:19:02.466 DEBUG:boto:Method: PUT 2022-09-09T23:19:02.466 DEBUG:boto:Path: /jkcgbj-66/ 2022-09-09T23:19:02.467 DEBUG:boto:Data: 2022-09-09T23:19:02.467 DEBUG:boto:Headers: {} 2022-09-09T23:19:02.468 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T23:19:02.468 DEBUG:boto:Port: 8000 2022-09-09T23:19:02.469 DEBUG:boto:Params: {} 2022-09-09T23:19:02.469 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:19:02.470 DEBUG:boto:Token: None 2022-09-09T23:19:02.470 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:19:02 GMT /jkcgbj-66/ 2022-09-09T23:19:02.471 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9fe4Vno5//acAfbQwW/++hsB4hs= 2022-09-09T23:19:02.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:19:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9fe4Vno5//acAfbQwW/++hsB4hs=', 'Content-Length': '0'} 2022-09-09T23:19:02.473 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:19:02.474 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:19:02.823 DEBUG:boto:Token: None 2022-09-09T23:19:02.824 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:19:02 GMT /jkcgbj-66/ 2022-09-09T23:19:02.825 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9fe4Vno5//acAfbQwW/++hsB4hs= 2022-09-09T23:19:02.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:19:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9fe4Vno5//acAfbQwW/++hsB4hs=', 'Content-Length': '0'} 2022-09-09T23:19:02.827 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:19:02.829 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:19:02.997 DEBUG:boto:Token: None 2022-09-09T23:19:02.998 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:19:02 GMT /jkcgbj-66/ 2022-09-09T23:19:02.999 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9fe4Vno5//acAfbQwW/++hsB4hs= 2022-09-09T23:19:02.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:19:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9fe4Vno5//acAfbQwW/++hsB4hs=', 'Content-Length': '0'} 2022-09-09T23:19:03.001 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:19:03.002 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:19:04.188 DEBUG:boto:Token: None 2022-09-09T23:19:04.189 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:19:02 GMT /jkcgbj-66/ 2022-09-09T23:19:04.189 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9fe4Vno5//acAfbQwW/++hsB4hs= 2022-09-09T23:19:04.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:19:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9fe4Vno5//acAfbQwW/++hsB4hs=', 'Content-Length': '0'} 2022-09-09T23:19:04.192 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:19:04.192 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:19:08.808 DEBUG:boto:Token: None 2022-09-09T23:19:08.809 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:19:02 GMT /jkcgbj-66/ 2022-09-09T23:19:08.810 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9fe4Vno5//acAfbQwW/++hsB4hs= 2022-09-09T23:19:08.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:19:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9fe4Vno5//acAfbQwW/++hsB4hs=', 'Content-Length': '0'} 2022-09-09T23:19:08.813 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:19:08.814 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:19:20.509 DEBUG:boto:Token: None 2022-09-09T23:19:20.510 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:19:02 GMT /jkcgbj-66/ 2022-09-09T23:19:20.511 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9fe4Vno5//acAfbQwW/++hsB4hs= 2022-09-09T23:19:20.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:19:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9fe4Vno5//acAfbQwW/++hsB4hs=', 'Content-Length': '0'} 2022-09-09T23:19:20.514 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:19:20.515 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:19:23.076 DEBUG:boto:Token: None 2022-09-09T23:19:23.077 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:19:02 GMT /jkcgbj-66/ 2022-09-09T23:19:23.078 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:9fe4Vno5//acAfbQwW/++hsB4hs= 2022-09-09T23:19:23.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:19:02 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:9fe4Vno5//acAfbQwW/++hsB4hs=', 'Content-Length': '0'} 2022-09-09T23:19:23.082 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:19:23.083 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:19:53.948 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_index_log_trim ... ERROR 2022-09-09T23:19:53.951 DEBUG:boto:path=/jkcgbj-67/ 2022-09-09T23:19:53.952 DEBUG:boto:auth_path=/jkcgbj-67/ 2022-09-09T23:19:53.953 DEBUG:boto:Method: PUT 2022-09-09T23:19:53.954 DEBUG:boto:Path: /jkcgbj-67/ 2022-09-09T23:19:53.955 DEBUG:boto:Data: 2022-09-09T23:19:53.956 DEBUG:boto:Headers: {} 2022-09-09T23:19:53.958 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T23:19:53.959 DEBUG:boto:Port: 8000 2022-09-09T23:19:53.960 DEBUG:boto:Params: {} 2022-09-09T23:19:53.962 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:19:53.963 DEBUG:boto:Token: None 2022-09-09T23:19:53.964 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:19:53 GMT /jkcgbj-67/ 2022-09-09T23:19:53.965 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wfsx3RKJMwttT8wp/Qh+1wwq/Z0= 2022-09-09T23:19:53.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:19:53 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wfsx3RKJMwttT8wp/Qh+1wwq/Z0=', 'Content-Length': '0'} 2022-09-09T23:19:53.969 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:19:53.971 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:19:54.511 DEBUG:boto:Token: None 2022-09-09T23:19:54.512 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:19:53 GMT /jkcgbj-67/ 2022-09-09T23:19:54.512 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wfsx3RKJMwttT8wp/Qh+1wwq/Z0= 2022-09-09T23:19:54.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:19:53 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wfsx3RKJMwttT8wp/Qh+1wwq/Z0=', 'Content-Length': '0'} 2022-09-09T23:19:54.515 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:19:54.516 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:19:54.968 DEBUG:boto:Token: None 2022-09-09T23:19:54.969 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:19:53 GMT /jkcgbj-67/ 2022-09-09T23:19:54.970 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wfsx3RKJMwttT8wp/Qh+1wwq/Z0= 2022-09-09T23:19:54.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:19:53 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wfsx3RKJMwttT8wp/Qh+1wwq/Z0=', 'Content-Length': '0'} 2022-09-09T23:19:54.974 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:19:54.975 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:19:58.204 DEBUG:boto:Token: None 2022-09-09T23:19:58.204 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:19:53 GMT /jkcgbj-67/ 2022-09-09T23:19:58.205 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wfsx3RKJMwttT8wp/Qh+1wwq/Z0= 2022-09-09T23:19:58.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:19:53 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wfsx3RKJMwttT8wp/Qh+1wwq/Z0=', 'Content-Length': '0'} 2022-09-09T23:19:58.208 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:19:58.210 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:20:05.007 DEBUG:boto:Token: None 2022-09-09T23:20:05.008 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:19:53 GMT /jkcgbj-67/ 2022-09-09T23:20:05.008 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wfsx3RKJMwttT8wp/Qh+1wwq/Z0= 2022-09-09T23:20:05.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:19:53 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wfsx3RKJMwttT8wp/Qh+1wwq/Z0=', 'Content-Length': '0'} 2022-09-09T23:20:05.010 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:20:05.011 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:20:15.771 DEBUG:boto:Token: None 2022-09-09T23:20:15.773 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:19:53 GMT /jkcgbj-67/ 2022-09-09T23:20:15.774 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wfsx3RKJMwttT8wp/Qh+1wwq/Z0= 2022-09-09T23:20:15.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:19:53 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wfsx3RKJMwttT8wp/Qh+1wwq/Z0=', 'Content-Length': '0'} 2022-09-09T23:20:15.777 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:20:15.777 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:20:32.802 DEBUG:boto:Token: None 2022-09-09T23:20:32.803 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:19:53 GMT /jkcgbj-67/ 2022-09-09T23:20:32.805 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:wfsx3RKJMwttT8wp/Qh+1wwq/Z0= 2022-09-09T23:20:32.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:19:53 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:wfsx3RKJMwttT8wp/Qh+1wwq/Z0=', 'Content-Length': '0'} 2022-09-09T23:20:32.808 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:20:32.809 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:20:59.807 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_incremental ... ERROR 2022-09-09T23:20:59.810 DEBUG:boto:path=/jkcgbj-68/ 2022-09-09T23:20:59.811 DEBUG:boto:auth_path=/jkcgbj-68/ 2022-09-09T23:20:59.811 DEBUG:boto:Method: PUT 2022-09-09T23:20:59.812 DEBUG:boto:Path: /jkcgbj-68/ 2022-09-09T23:20:59.812 DEBUG:boto:Data: 2022-09-09T23:20:59.813 DEBUG:boto:Headers: {} 2022-09-09T23:20:59.814 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-09T23:20:59.814 DEBUG:boto:Port: 8000 2022-09-09T23:20:59.815 DEBUG:boto:Params: {} 2022-09-09T23:20:59.815 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:20:59.816 DEBUG:boto:Token: None 2022-09-09T23:20:59.817 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:20:59 GMT /jkcgbj-68/ 2022-09-09T23:20:59.817 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7GUTJN3EUR1nQYdmxpMI1jsbiAs= 2022-09-09T23:20:59.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:20:59 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7GUTJN3EUR1nQYdmxpMI1jsbiAs=', 'Content-Length': '0'} 2022-09-09T23:20:59.820 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:20:59.821 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:21:00.056 DEBUG:boto:Token: None 2022-09-09T23:21:00.057 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:20:59 GMT /jkcgbj-68/ 2022-09-09T23:21:00.058 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7GUTJN3EUR1nQYdmxpMI1jsbiAs= 2022-09-09T23:21:00.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:20:59 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7GUTJN3EUR1nQYdmxpMI1jsbiAs=', 'Content-Length': '0'} 2022-09-09T23:21:00.061 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:21:00.061 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:21:01.416 DEBUG:boto:Token: None 2022-09-09T23:21:01.417 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:20:59 GMT /jkcgbj-68/ 2022-09-09T23:21:01.418 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7GUTJN3EUR1nQYdmxpMI1jsbiAs= 2022-09-09T23:21:01.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:20:59 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7GUTJN3EUR1nQYdmxpMI1jsbiAs=', 'Content-Length': '0'} 2022-09-09T23:21:01.420 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:21:01.421 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:21:05.061 DEBUG:boto:Token: None 2022-09-09T23:21:05.066 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:20:59 GMT /jkcgbj-68/ 2022-09-09T23:21:05.067 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7GUTJN3EUR1nQYdmxpMI1jsbiAs= 2022-09-09T23:21:05.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:20:59 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7GUTJN3EUR1nQYdmxpMI1jsbiAs=', 'Content-Length': '0'} 2022-09-09T23:21:05.069 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:21:05.070 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:21:08.851 DEBUG:boto:Token: None 2022-09-09T23:21:08.852 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:20:59 GMT /jkcgbj-68/ 2022-09-09T23:21:08.852 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7GUTJN3EUR1nQYdmxpMI1jsbiAs= 2022-09-09T23:21:08.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:20:59 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7GUTJN3EUR1nQYdmxpMI1jsbiAs=', 'Content-Length': '0'} 2022-09-09T23:21:08.854 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:21:08.855 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:21:20.411 DEBUG:boto:Token: None 2022-09-09T23:21:20.412 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:20:59 GMT /jkcgbj-68/ 2022-09-09T23:21:20.413 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7GUTJN3EUR1nQYdmxpMI1jsbiAs= 2022-09-09T23:21:20.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:20:59 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7GUTJN3EUR1nQYdmxpMI1jsbiAs=', 'Content-Length': '0'} 2022-09-09T23:21:20.416 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:21:20.416 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:21:29.315 DEBUG:boto:Token: None 2022-09-09T23:21:29.316 DEBUG:boto:StringToSign: PUT Fri, 09 Sep 2022 23:20:59 GMT /jkcgbj-68/ 2022-09-09T23:21:29.317 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:7GUTJN3EUR1nQYdmxpMI1jsbiAs= 2022-09-09T23:21:29.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 09 Sep 2022 23:20:59 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:7GUTJN3EUR1nQYdmxpMI1jsbiAs=', 'Content-Length': '0'} 2022-09-09T23:21:29.319 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-09T23:21:29.320 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-09T23:21:35.121 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_full ... ERROR 2022-09-09T23:21:35.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:21:35.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-09T23:21:35.128 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-09T23:21:35.181 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-09T23:21:35.181 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-09T23:26:35.185 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T23:26:35.182+0000 7f6f4966bdc0 0 monclient(hunting): authenticate timed out after 300 2022-09-09T23:31:35.189 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T23:31:35.183+0000 7f6f4966bdc0 0 monclient(hunting): authenticate timed out after 300 2022-09-09T23:36:35.188 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T23:36:35.185+0000 7f6f4966bdc0 0 monclient(hunting): authenticate timed out after 300 2022-09-09T23:41:35.188 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T23:41:35.183+0000 7f6f4966bdc0 0 monclient(hunting): authenticate timed out after 300 2022-09-09T23:46:35.190 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T23:46:35.184+0000 7f6f4966bdc0 0 monclient(hunting): authenticate timed out after 300 2022-09-09T23:51:35.206 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T23:51:35.182+0000 7f6f4966bdc0 0 monclient(hunting): authenticate timed out after 300 2022-09-09T23:56:35.192 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-09T23:56:35.188+0000 7f6f4966bdc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T00:01:35.201 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T00:01:35.185+0000 7f6f4966bdc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T00:06:35.195 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T00:06:35.187+0000 7f6f4966bdc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T00:11:35.195 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T00:11:35.189+0000 7f6f4966bdc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T00:11:35.197 INFO:teuthology.orchestra.run.smithi005.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-10T00:11:35.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-10T00:11:35.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... FAIL 2022-09-10T00:11:35.211 DEBUG:boto:path=/jkcgbj-69/ 2022-09-10T00:11:35.212 DEBUG:boto:auth_path=/jkcgbj-69/ 2022-09-10T00:11:35.213 DEBUG:boto:Method: PUT 2022-09-10T00:11:35.214 DEBUG:boto:Path: /jkcgbj-69/ 2022-09-10T00:11:35.214 DEBUG:boto:Data: 2022-09-10T00:11:35.215 DEBUG:boto:Headers: {} 2022-09-10T00:11:35.216 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-10T00:11:35.217 DEBUG:boto:Port: 8000 2022-09-10T00:11:35.217 DEBUG:boto:Params: {} 2022-09-10T00:11:35.219 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:11:35.220 DEBUG:boto:Token: None 2022-09-10T00:11:35.223 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:11:35 GMT /jkcgbj-69/ 2022-09-10T00:11:35.224 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:znCPc6IkKpLHZpr4tVMZa1s4m4o= 2022-09-10T00:11:35.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:11:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:znCPc6IkKpLHZpr4tVMZa1s4m4o=', 'Content-Length': '0'} 2022-09-10T00:11:35.232 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:11:35.234 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:11:35.650 DEBUG:boto:Token: None 2022-09-10T00:11:35.652 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:11:35 GMT /jkcgbj-69/ 2022-09-10T00:11:35.652 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:znCPc6IkKpLHZpr4tVMZa1s4m4o= 2022-09-10T00:11:35.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:11:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:znCPc6IkKpLHZpr4tVMZa1s4m4o=', 'Content-Length': '0'} 2022-09-10T00:11:35.655 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:11:35.656 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:11:35.704 DEBUG:boto:Token: None 2022-09-10T00:11:35.705 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:11:35 GMT /jkcgbj-69/ 2022-09-10T00:11:35.706 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:znCPc6IkKpLHZpr4tVMZa1s4m4o= 2022-09-10T00:11:35.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:11:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:znCPc6IkKpLHZpr4tVMZa1s4m4o=', 'Content-Length': '0'} 2022-09-10T00:11:35.709 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:11:35.709 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:11:38.518 DEBUG:boto:Token: None 2022-09-10T00:11:38.519 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:11:35 GMT /jkcgbj-69/ 2022-09-10T00:11:38.520 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:znCPc6IkKpLHZpr4tVMZa1s4m4o= 2022-09-10T00:11:38.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:11:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:znCPc6IkKpLHZpr4tVMZa1s4m4o=', 'Content-Length': '0'} 2022-09-10T00:11:38.522 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:11:38.523 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:11:40.321 DEBUG:boto:Token: None 2022-09-10T00:11:40.322 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:11:35 GMT /jkcgbj-69/ 2022-09-10T00:11:40.322 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:znCPc6IkKpLHZpr4tVMZa1s4m4o= 2022-09-10T00:11:40.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:11:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:znCPc6IkKpLHZpr4tVMZa1s4m4o=', 'Content-Length': '0'} 2022-09-10T00:11:40.326 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:11:40.327 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:11:40.519 DEBUG:boto:Token: None 2022-09-10T00:11:40.520 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:11:35 GMT /jkcgbj-69/ 2022-09-10T00:11:40.521 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:znCPc6IkKpLHZpr4tVMZa1s4m4o= 2022-09-10T00:11:40.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:11:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:znCPc6IkKpLHZpr4tVMZa1s4m4o=', 'Content-Length': '0'} 2022-09-10T00:11:40.524 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:11:40.525 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:12:07.913 DEBUG:boto:Token: None 2022-09-10T00:12:07.914 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:11:35 GMT /jkcgbj-69/ 2022-09-10T00:12:07.915 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:znCPc6IkKpLHZpr4tVMZa1s4m4o= 2022-09-10T00:12:07.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:11:35 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:znCPc6IkKpLHZpr4tVMZa1s4m4o=', 'Content-Length': '0'} 2022-09-10T00:12:07.918 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:12:07.919 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:12:55.367 INFO:tasks.rgw_multisite_tests:Create several generations of objects, then run bucket sync ... ERROR 2022-09-10T00:12:55.370 DEBUG:boto:path=/jkcgbj-70/ 2022-09-10T00:12:55.371 DEBUG:boto:auth_path=/jkcgbj-70/ 2022-09-10T00:12:55.372 DEBUG:boto:Method: PUT 2022-09-10T00:12:55.373 DEBUG:boto:Path: /jkcgbj-70/ 2022-09-10T00:12:55.374 DEBUG:boto:Data: 2022-09-10T00:12:55.375 DEBUG:boto:Headers: {} 2022-09-10T00:12:55.376 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-10T00:12:55.377 DEBUG:boto:Port: 8000 2022-09-10T00:12:55.377 DEBUG:boto:Params: {} 2022-09-10T00:12:55.380 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:12:55.381 DEBUG:boto:Token: None 2022-09-10T00:12:55.382 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:12:55 GMT /jkcgbj-70/ 2022-09-10T00:12:55.382 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UrwoZdYVlov5lHpxaDs42RnhgCY= 2022-09-10T00:12:55.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:12:55 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UrwoZdYVlov5lHpxaDs42RnhgCY=', 'Content-Length': '0'} 2022-09-10T00:12:55.390 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:12:55.391 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:12:55.636 DEBUG:boto:Token: None 2022-09-10T00:12:55.637 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:12:55 GMT /jkcgbj-70/ 2022-09-10T00:12:55.638 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UrwoZdYVlov5lHpxaDs42RnhgCY= 2022-09-10T00:12:55.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:12:55 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UrwoZdYVlov5lHpxaDs42RnhgCY=', 'Content-Length': '0'} 2022-09-10T00:12:55.645 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:12:55.650 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:12:57.608 DEBUG:boto:Token: None 2022-09-10T00:12:57.609 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:12:55 GMT /jkcgbj-70/ 2022-09-10T00:12:57.610 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UrwoZdYVlov5lHpxaDs42RnhgCY= 2022-09-10T00:12:57.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:12:55 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UrwoZdYVlov5lHpxaDs42RnhgCY=', 'Content-Length': '0'} 2022-09-10T00:12:57.612 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:12:57.613 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:12:58.214 DEBUG:boto:Token: None 2022-09-10T00:12:58.216 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:12:55 GMT /jkcgbj-70/ 2022-09-10T00:12:58.217 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UrwoZdYVlov5lHpxaDs42RnhgCY= 2022-09-10T00:12:58.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:12:55 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UrwoZdYVlov5lHpxaDs42RnhgCY=', 'Content-Length': '0'} 2022-09-10T00:12:58.220 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:12:58.221 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:12:58.436 DEBUG:boto:Token: None 2022-09-10T00:12:58.437 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:12:55 GMT /jkcgbj-70/ 2022-09-10T00:12:58.437 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UrwoZdYVlov5lHpxaDs42RnhgCY= 2022-09-10T00:12:58.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:12:55 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UrwoZdYVlov5lHpxaDs42RnhgCY=', 'Content-Length': '0'} 2022-09-10T00:12:58.440 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:12:58.441 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:13:10.530 DEBUG:boto:Token: None 2022-09-10T00:13:10.532 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:12:55 GMT /jkcgbj-70/ 2022-09-10T00:13:10.533 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UrwoZdYVlov5lHpxaDs42RnhgCY= 2022-09-10T00:13:10.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:12:55 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UrwoZdYVlov5lHpxaDs42RnhgCY=', 'Content-Length': '0'} 2022-09-10T00:13:10.542 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:13:10.543 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:13:20.610 DEBUG:boto:Token: None 2022-09-10T00:13:20.612 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:12:55 GMT /jkcgbj-70/ 2022-09-10T00:13:20.612 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:UrwoZdYVlov5lHpxaDs42RnhgCY= 2022-09-10T00:13:20.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:12:55 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:UrwoZdYVlov5lHpxaDs42RnhgCY=', 'Content-Length': '0'} 2022-09-10T00:13:20.625 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:13:20.627 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:14:04.096 INFO:tasks.rgw_multisite_tests:Create several generations of objects, trash them, then run bucket sync init ... ERROR 2022-09-10T00:14:04.099 INFO:rgw_multi.tests:create role zone=test-zone1 name=roles-jkcgbj-1 2022-09-10T00:14:04.100 DEBUG:boto:Method: POST 2022-09-10T00:14:04.100 DEBUG:boto:Path: / 2022-09-10T00:14:04.101 DEBUG:boto:Data: 2022-09-10T00:14:04.101 DEBUG:boto:Headers: {} 2022-09-10T00:14:04.102 DEBUG:boto:Host: smithi005.front.sepia.ceph.com 2022-09-10T00:14:04.102 DEBUG:boto:Port: 8000 2022-09-10T00:14:04.103 DEBUG:boto:Params: {'RoleName': 'roles-jkcgbj-1', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2022-09-10T00:14:04.104 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:14:04.104 DEBUG:boto:Token: None 2022-09-10T00:14:04.105 DEBUG:boto:CanonicalRequest: POST / host:smithi005.front.sepia.ceph.com:8000 x-amz-date:20220910T001404Z host;x-amz-date 3f183ded43ce8ac736782d339c441f51026612ce95e155344d5bd632b9401d33 2022-09-10T00:14:04.106 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220910T001404Z 20220910/front/smithi005/aws4_request d9f13fd615c19be93f7b662d2a3a0f515ef47764fa56e567d4631f49f21e6ef3 2022-09-10T00:14:04.108 DEBUG:boto:Signature: 6226833602cebfcfb6eea97192910e780c60c36f983e0f32c6e5da7e973bce87 2022-09-10T00:14:04.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220910T001404Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=VQ6YV8EJ6TF7N01I9BCH/20220910/front/smithi005/aws4_request,SignedHeaders=host;x-amz-date,Signature=6226833602cebfcfb6eea97192910e780c60c36f983e0f32c6e5da7e973bce87', 'Host': 'smithi005.front.sepia.ceph.com:8000'} 2022-09-10T00:14:04.111 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:14:04.112 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:14:04.510 DEBUG:boto:Token: None 2022-09-10T00:14:04.511 DEBUG:boto:CanonicalRequest: POST / host:smithi005.front.sepia.ceph.com:8000 x-amz-date:20220910T001404Z host;x-amz-date 3f183ded43ce8ac736782d339c441f51026612ce95e155344d5bd632b9401d33 2022-09-10T00:14:04.512 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220910T001404Z 20220910/front/smithi005/aws4_request d9f13fd615c19be93f7b662d2a3a0f515ef47764fa56e567d4631f49f21e6ef3 2022-09-10T00:14:04.513 DEBUG:boto:Signature: 6226833602cebfcfb6eea97192910e780c60c36f983e0f32c6e5da7e973bce87 2022-09-10T00:14:04.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220910T001404Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=VQ6YV8EJ6TF7N01I9BCH/20220910/front/smithi005/aws4_request,SignedHeaders=host;x-amz-date,Signature=6226833602cebfcfb6eea97192910e780c60c36f983e0f32c6e5da7e973bce87', 'Host': 'smithi005.front.sepia.ceph.com:8000'} 2022-09-10T00:14:04.517 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:14:04.517 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:14:05.322 DEBUG:boto:Token: None 2022-09-10T00:14:05.323 DEBUG:boto:CanonicalRequest: POST / host:smithi005.front.sepia.ceph.com:8000 x-amz-date:20220910T001405Z host;x-amz-date 3f183ded43ce8ac736782d339c441f51026612ce95e155344d5bd632b9401d33 2022-09-10T00:14:05.323 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220910T001405Z 20220910/front/smithi005/aws4_request 23dbb7269e852184bd7221f960c13f7a5298e5c996e9e1c52c9151c31e61823b 2022-09-10T00:14:05.324 DEBUG:boto:Signature: 0c6cc7dbb9d3a19f6df234120233f11e61f9f7cdd741ca9755e16042044e07d7 2022-09-10T00:14:05.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220910T001405Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=VQ6YV8EJ6TF7N01I9BCH/20220910/front/smithi005/aws4_request,SignedHeaders=host;x-amz-date,Signature=0c6cc7dbb9d3a19f6df234120233f11e61f9f7cdd741ca9755e16042044e07d7', 'Host': 'smithi005.front.sepia.ceph.com:8000'} 2022-09-10T00:14:05.330 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:14:05.332 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:14:08.062 DEBUG:boto:Token: None 2022-09-10T00:14:08.063 DEBUG:boto:CanonicalRequest: POST / host:smithi005.front.sepia.ceph.com:8000 x-amz-date:20220910T001408Z host;x-amz-date 3f183ded43ce8ac736782d339c441f51026612ce95e155344d5bd632b9401d33 2022-09-10T00:14:08.064 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220910T001408Z 20220910/front/smithi005/aws4_request 28542e32639af0865a5127192444d7b46913854fb877ca9c2d95e838bae6517e 2022-09-10T00:14:08.064 DEBUG:boto:Signature: ad0964c979e62051c9a808ca5a7f660984bca22b68ceca8f647ab4ce0dc2c934 2022-09-10T00:14:08.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220910T001408Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=VQ6YV8EJ6TF7N01I9BCH/20220910/front/smithi005/aws4_request,SignedHeaders=host;x-amz-date,Signature=ad0964c979e62051c9a808ca5a7f660984bca22b68ceca8f647ab4ce0dc2c934', 'Host': 'smithi005.front.sepia.ceph.com:8000'} 2022-09-10T00:14:08.067 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:14:08.068 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:14:10.162 DEBUG:boto:Token: None 2022-09-10T00:14:10.164 DEBUG:boto:CanonicalRequest: POST / host:smithi005.front.sepia.ceph.com:8000 x-amz-date:20220910T001410Z host;x-amz-date 3f183ded43ce8ac736782d339c441f51026612ce95e155344d5bd632b9401d33 2022-09-10T00:14:10.165 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220910T001410Z 20220910/front/smithi005/aws4_request f30105ef795ff21679e4bd720f4acd95451d917472c0667035ec5b9f8783e11a 2022-09-10T00:14:10.165 DEBUG:boto:Signature: 79932842ef22563bee1a23647e4865823e5508015ddb577ab6767b63b593d116 2022-09-10T00:14:10.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220910T001410Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=VQ6YV8EJ6TF7N01I9BCH/20220910/front/smithi005/aws4_request,SignedHeaders=host;x-amz-date,Signature=79932842ef22563bee1a23647e4865823e5508015ddb577ab6767b63b593d116', 'Host': 'smithi005.front.sepia.ceph.com:8000'} 2022-09-10T00:14:10.168 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:14:10.169 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:14:25.374 DEBUG:boto:Token: None 2022-09-10T00:14:25.375 DEBUG:boto:CanonicalRequest: POST / host:smithi005.front.sepia.ceph.com:8000 x-amz-date:20220910T001425Z host;x-amz-date 3f183ded43ce8ac736782d339c441f51026612ce95e155344d5bd632b9401d33 2022-09-10T00:14:25.376 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220910T001425Z 20220910/front/smithi005/aws4_request 3f05edfe03ea63e5d199af088abb04a0fc868900bf0d1443215887e598b55d20 2022-09-10T00:14:25.377 DEBUG:boto:Signature: b6b3476a04204096c7988c03c2764822c5f144b4bd5146da16bf26d395e97a07 2022-09-10T00:14:25.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220910T001425Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=VQ6YV8EJ6TF7N01I9BCH/20220910/front/smithi005/aws4_request,SignedHeaders=host;x-amz-date,Signature=b6b3476a04204096c7988c03c2764822c5f144b4bd5146da16bf26d395e97a07', 'Host': 'smithi005.front.sepia.ceph.com:8000'} 2022-09-10T00:14:25.379 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:14:25.380 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:14:53.937 DEBUG:boto:Token: None 2022-09-10T00:14:53.940 DEBUG:boto:CanonicalRequest: POST / host:smithi005.front.sepia.ceph.com:8000 x-amz-date:20220910T001453Z host;x-amz-date 3f183ded43ce8ac736782d339c441f51026612ce95e155344d5bd632b9401d33 2022-09-10T00:14:53.941 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220910T001453Z 20220910/front/smithi005/aws4_request c374c19f4a73139cdf577f6d83b031ca6a577dd171a79607923772b678ec9cc3 2022-09-10T00:14:53.952 DEBUG:boto:Signature: f11363b6d2d284ddc02a8b2331a3518459c84a6842df87e4364ee418b9ff6e56 2022-09-10T00:14:53.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220910T001453Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=VQ6YV8EJ6TF7N01I9BCH/20220910/front/smithi005/aws4_request,SignedHeaders=host;x-amz-date,Signature=f11363b6d2d284ddc02a8b2331a3518459c84a6842df87e4364ee418b9ff6e56', 'Host': 'smithi005.front.sepia.ceph.com:8000'} 2022-09-10T00:14:53.956 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:14:53.957 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:15:14.222 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync ... ERROR 2022-09-10T00:15:14.224 DEBUG:boto:path=/jkcgbj-71/ 2022-09-10T00:15:14.225 DEBUG:boto:auth_path=/jkcgbj-71/ 2022-09-10T00:15:14.227 DEBUG:boto:Method: PUT 2022-09-10T00:15:14.227 DEBUG:boto:Path: /jkcgbj-71/ 2022-09-10T00:15:14.229 DEBUG:boto:Data: 2022-09-10T00:15:14.231 DEBUG:boto:Headers: {} 2022-09-10T00:15:14.232 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-10T00:15:14.233 DEBUG:boto:Port: 8000 2022-09-10T00:15:14.233 DEBUG:boto:Params: {} 2022-09-10T00:15:14.234 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:15:14.235 DEBUG:boto:Token: None 2022-09-10T00:15:14.236 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:15:14 GMT /jkcgbj-71/ 2022-09-10T00:15:14.238 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:WDprrpp2gSmU4j7oODQ4JVOR0hs= 2022-09-10T00:15:14.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:15:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:WDprrpp2gSmU4j7oODQ4JVOR0hs=', 'Content-Length': '0'} 2022-09-10T00:15:14.244 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:15:14.246 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:15:14.282 DEBUG:boto:Token: None 2022-09-10T00:15:14.283 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:15:14 GMT /jkcgbj-71/ 2022-09-10T00:15:14.284 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:WDprrpp2gSmU4j7oODQ4JVOR0hs= 2022-09-10T00:15:14.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:15:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:WDprrpp2gSmU4j7oODQ4JVOR0hs=', 'Content-Length': '0'} 2022-09-10T00:15:14.290 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:15:14.291 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:15:14.982 DEBUG:boto:Token: None 2022-09-10T00:15:14.984 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:15:14 GMT /jkcgbj-71/ 2022-09-10T00:15:14.985 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:WDprrpp2gSmU4j7oODQ4JVOR0hs= 2022-09-10T00:15:14.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:15:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:WDprrpp2gSmU4j7oODQ4JVOR0hs=', 'Content-Length': '0'} 2022-09-10T00:15:14.989 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:15:14.990 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:15:15.957 DEBUG:boto:Token: None 2022-09-10T00:15:15.968 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:15:14 GMT /jkcgbj-71/ 2022-09-10T00:15:15.969 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:WDprrpp2gSmU4j7oODQ4JVOR0hs= 2022-09-10T00:15:15.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:15:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:WDprrpp2gSmU4j7oODQ4JVOR0hs=', 'Content-Length': '0'} 2022-09-10T00:15:15.972 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:15:15.973 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:15:22.350 DEBUG:boto:Token: None 2022-09-10T00:15:22.351 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:15:14 GMT /jkcgbj-71/ 2022-09-10T00:15:22.352 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:WDprrpp2gSmU4j7oODQ4JVOR0hs= 2022-09-10T00:15:22.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:15:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:WDprrpp2gSmU4j7oODQ4JVOR0hs=', 'Content-Length': '0'} 2022-09-10T00:15:22.358 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:15:22.364 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:15:24.239 DEBUG:boto:Token: None 2022-09-10T00:15:24.240 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:15:14 GMT /jkcgbj-71/ 2022-09-10T00:15:24.241 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:WDprrpp2gSmU4j7oODQ4JVOR0hs= 2022-09-10T00:15:24.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:15:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:WDprrpp2gSmU4j7oODQ4JVOR0hs=', 'Content-Length': '0'} 2022-09-10T00:15:24.245 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:15:24.247 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:15:49.427 DEBUG:boto:Token: None 2022-09-10T00:15:49.428 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:15:14 GMT /jkcgbj-71/ 2022-09-10T00:15:49.428 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:WDprrpp2gSmU4j7oODQ4JVOR0hs= 2022-09-10T00:15:49.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:15:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:WDprrpp2gSmU4j7oODQ4JVOR0hs=', 'Content-Length': '0'} 2022-09-10T00:15:49.432 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:15:49.434 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:16:09.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_full_sync_after_data_sync_init ... ERROR 2022-09-10T00:16:09.420 DEBUG:boto:path=/jkcgbj-72/ 2022-09-10T00:16:09.420 DEBUG:boto:auth_path=/jkcgbj-72/ 2022-09-10T00:16:09.421 DEBUG:boto:Method: PUT 2022-09-10T00:16:09.422 DEBUG:boto:Path: /jkcgbj-72/ 2022-09-10T00:16:09.423 DEBUG:boto:Data: 2022-09-10T00:16:09.424 DEBUG:boto:Headers: {} 2022-09-10T00:16:09.425 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-10T00:16:09.426 DEBUG:boto:Port: 8000 2022-09-10T00:16:09.426 DEBUG:boto:Params: {} 2022-09-10T00:16:09.427 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:16:09.428 DEBUG:boto:Token: None 2022-09-10T00:16:09.429 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:16:09 GMT /jkcgbj-72/ 2022-09-10T00:16:09.430 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:q6mfQkZTj8Jd7uImzFmmurbfAwY= 2022-09-10T00:16:09.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:16:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:q6mfQkZTj8Jd7uImzFmmurbfAwY=', 'Content-Length': '0'} 2022-09-10T00:16:09.434 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:16:09.435 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:16:09.614 DEBUG:boto:Token: None 2022-09-10T00:16:09.615 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:16:09 GMT /jkcgbj-72/ 2022-09-10T00:16:09.616 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:q6mfQkZTj8Jd7uImzFmmurbfAwY= 2022-09-10T00:16:09.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:16:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:q6mfQkZTj8Jd7uImzFmmurbfAwY=', 'Content-Length': '0'} 2022-09-10T00:16:09.620 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:16:09.621 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:16:09.678 DEBUG:boto:Token: None 2022-09-10T00:16:09.679 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:16:09 GMT /jkcgbj-72/ 2022-09-10T00:16:09.680 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:q6mfQkZTj8Jd7uImzFmmurbfAwY= 2022-09-10T00:16:09.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:16:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:q6mfQkZTj8Jd7uImzFmmurbfAwY=', 'Content-Length': '0'} 2022-09-10T00:16:09.688 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:16:09.690 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:16:09.714 DEBUG:boto:Token: None 2022-09-10T00:16:09.714 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:16:09 GMT /jkcgbj-72/ 2022-09-10T00:16:09.715 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:q6mfQkZTj8Jd7uImzFmmurbfAwY= 2022-09-10T00:16:09.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:16:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:q6mfQkZTj8Jd7uImzFmmurbfAwY=', 'Content-Length': '0'} 2022-09-10T00:16:09.718 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:16:09.719 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:16:13.130 DEBUG:boto:Token: None 2022-09-10T00:16:13.131 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:16:09 GMT /jkcgbj-72/ 2022-09-10T00:16:13.132 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:q6mfQkZTj8Jd7uImzFmmurbfAwY= 2022-09-10T00:16:13.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:16:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:q6mfQkZTj8Jd7uImzFmmurbfAwY=', 'Content-Length': '0'} 2022-09-10T00:16:13.139 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:16:13.140 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:16:14.982 DEBUG:boto:Token: None 2022-09-10T00:16:14.983 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:16:09 GMT /jkcgbj-72/ 2022-09-10T00:16:14.984 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:q6mfQkZTj8Jd7uImzFmmurbfAwY= 2022-09-10T00:16:14.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:16:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:q6mfQkZTj8Jd7uImzFmmurbfAwY=', 'Content-Length': '0'} 2022-09-10T00:16:14.987 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:16:14.988 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:16:16.070 DEBUG:boto:Token: None 2022-09-10T00:16:16.071 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:16:09 GMT /jkcgbj-72/ 2022-09-10T00:16:16.072 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:q6mfQkZTj8Jd7uImzFmmurbfAwY= 2022-09-10T00:16:16.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:16:09 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:q6mfQkZTj8Jd7uImzFmmurbfAwY=', 'Content-Length': '0'} 2022-09-10T00:16:16.075 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:16:16.076 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:16:19.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init ... ERROR 2022-09-10T00:16:19.243 DEBUG:boto:path=/jkcgbj-73/ 2022-09-10T00:16:19.243 DEBUG:boto:auth_path=/jkcgbj-73/ 2022-09-10T00:16:19.245 DEBUG:boto:Method: PUT 2022-09-10T00:16:19.245 DEBUG:boto:Path: /jkcgbj-73/ 2022-09-10T00:16:19.246 DEBUG:boto:Data: 2022-09-10T00:16:19.247 DEBUG:boto:Headers: {} 2022-09-10T00:16:19.248 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-10T00:16:19.249 DEBUG:boto:Port: 8000 2022-09-10T00:16:19.250 DEBUG:boto:Params: {} 2022-09-10T00:16:19.251 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:16:19.252 DEBUG:boto:Token: None 2022-09-10T00:16:19.252 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:16:19 GMT /jkcgbj-73/ 2022-09-10T00:16:19.253 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:zmqt/YTV18appzjpj6pGudrvaIE= 2022-09-10T00:16:19.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:16:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:zmqt/YTV18appzjpj6pGudrvaIE=', 'Content-Length': '0'} 2022-09-10T00:16:19.255 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:16:19.257 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:16:19.334 DEBUG:boto:Token: None 2022-09-10T00:16:19.335 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:16:19 GMT /jkcgbj-73/ 2022-09-10T00:16:19.336 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:zmqt/YTV18appzjpj6pGudrvaIE= 2022-09-10T00:16:19.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:16:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:zmqt/YTV18appzjpj6pGudrvaIE=', 'Content-Length': '0'} 2022-09-10T00:16:19.339 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:16:19.341 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:16:20.695 DEBUG:boto:Token: None 2022-09-10T00:16:20.697 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:16:19 GMT /jkcgbj-73/ 2022-09-10T00:16:20.698 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:zmqt/YTV18appzjpj6pGudrvaIE= 2022-09-10T00:16:20.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:16:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:zmqt/YTV18appzjpj6pGudrvaIE=', 'Content-Length': '0'} 2022-09-10T00:16:20.701 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:16:20.702 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:16:23.808 DEBUG:boto:Token: None 2022-09-10T00:16:23.809 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:16:19 GMT /jkcgbj-73/ 2022-09-10T00:16:23.810 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:zmqt/YTV18appzjpj6pGudrvaIE= 2022-09-10T00:16:23.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:16:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:zmqt/YTV18appzjpj6pGudrvaIE=', 'Content-Length': '0'} 2022-09-10T00:16:23.814 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:16:23.815 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:16:26.272 DEBUG:boto:Token: None 2022-09-10T00:16:26.273 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:16:19 GMT /jkcgbj-73/ 2022-09-10T00:16:26.274 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:zmqt/YTV18appzjpj6pGudrvaIE= 2022-09-10T00:16:26.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:16:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:zmqt/YTV18appzjpj6pGudrvaIE=', 'Content-Length': '0'} 2022-09-10T00:16:26.279 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:16:26.280 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:16:32.838 DEBUG:boto:Token: None 2022-09-10T00:16:32.839 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:16:19 GMT /jkcgbj-73/ 2022-09-10T00:16:32.840 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:zmqt/YTV18appzjpj6pGudrvaIE= 2022-09-10T00:16:32.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:16:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:zmqt/YTV18appzjpj6pGudrvaIE=', 'Content-Length': '0'} 2022-09-10T00:16:32.842 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:16:32.843 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:16:49.675 DEBUG:boto:Token: None 2022-09-10T00:16:49.677 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:16:19 GMT /jkcgbj-73/ 2022-09-10T00:16:49.678 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:zmqt/YTV18appzjpj6pGudrvaIE= 2022-09-10T00:16:49.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:16:19 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:zmqt/YTV18appzjpj6pGudrvaIE=', 'Content-Length': '0'} 2022-09-10T00:16:49.680 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:16:49.682 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:16:53.925 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init ... ERROR 2022-09-10T00:16:53.926 DEBUG:boto:path=/jkcgbj-74/ 2022-09-10T00:16:53.927 DEBUG:boto:auth_path=/jkcgbj-74/ 2022-09-10T00:16:53.928 DEBUG:boto:Method: PUT 2022-09-10T00:16:53.929 DEBUG:boto:Path: /jkcgbj-74/ 2022-09-10T00:16:53.930 DEBUG:boto:Data: 2022-09-10T00:16:53.931 DEBUG:boto:Headers: {} 2022-09-10T00:16:53.931 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-10T00:16:53.932 DEBUG:boto:Port: 8000 2022-09-10T00:16:53.933 DEBUG:boto:Params: {} 2022-09-10T00:16:53.937 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:16:53.938 DEBUG:boto:Token: None 2022-09-10T00:16:53.939 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:16:53 GMT /jkcgbj-74/ 2022-09-10T00:16:53.940 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jpgz5ayBJURlN8KC7EANEgUj5A4= 2022-09-10T00:16:53.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:16:53 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jpgz5ayBJURlN8KC7EANEgUj5A4=', 'Content-Length': '0'} 2022-09-10T00:16:53.944 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:16:53.945 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:16:54.688 DEBUG:boto:Token: None 2022-09-10T00:16:54.689 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:16:53 GMT /jkcgbj-74/ 2022-09-10T00:16:54.690 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jpgz5ayBJURlN8KC7EANEgUj5A4= 2022-09-10T00:16:54.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:16:53 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jpgz5ayBJURlN8KC7EANEgUj5A4=', 'Content-Length': '0'} 2022-09-10T00:16:54.702 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:16:54.704 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:16:55.168 DEBUG:boto:Token: None 2022-09-10T00:16:55.169 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:16:53 GMT /jkcgbj-74/ 2022-09-10T00:16:55.169 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jpgz5ayBJURlN8KC7EANEgUj5A4= 2022-09-10T00:16:55.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:16:53 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jpgz5ayBJURlN8KC7EANEgUj5A4=', 'Content-Length': '0'} 2022-09-10T00:16:55.172 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:16:55.173 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:16:55.704 DEBUG:boto:Token: None 2022-09-10T00:16:55.706 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:16:53 GMT /jkcgbj-74/ 2022-09-10T00:16:55.707 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jpgz5ayBJURlN8KC7EANEgUj5A4= 2022-09-10T00:16:55.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:16:53 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jpgz5ayBJURlN8KC7EANEgUj5A4=', 'Content-Length': '0'} 2022-09-10T00:16:55.715 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:16:55.717 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:16:55.914 DEBUG:boto:Token: None 2022-09-10T00:16:55.915 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:16:53 GMT /jkcgbj-74/ 2022-09-10T00:16:55.916 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jpgz5ayBJURlN8KC7EANEgUj5A4= 2022-09-10T00:16:55.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:16:53 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jpgz5ayBJURlN8KC7EANEgUj5A4=', 'Content-Length': '0'} 2022-09-10T00:16:55.919 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:16:55.920 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:16:56.119 DEBUG:boto:Token: None 2022-09-10T00:16:56.120 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:16:53 GMT /jkcgbj-74/ 2022-09-10T00:16:56.121 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jpgz5ayBJURlN8KC7EANEgUj5A4= 2022-09-10T00:16:56.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:16:53 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jpgz5ayBJURlN8KC7EANEgUj5A4=', 'Content-Length': '0'} 2022-09-10T00:16:56.125 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:16:56.126 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:16:57.245 DEBUG:boto:Token: None 2022-09-10T00:16:57.246 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:16:53 GMT /jkcgbj-74/ 2022-09-10T00:16:57.247 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:jpgz5ayBJURlN8KC7EANEgUj5A4= 2022-09-10T00:16:57.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:16:53 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:jpgz5ayBJURlN8KC7EANEgUj5A4=', 'Content-Length': '0'} 2022-09-10T00:16:57.255 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:16:57.256 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:17:14.691 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init ... ERROR 2022-09-10T00:17:14.692 DEBUG:boto:path=/jkcgbj-75/ 2022-09-10T00:17:14.693 DEBUG:boto:auth_path=/jkcgbj-75/ 2022-09-10T00:17:14.693 DEBUG:boto:Method: PUT 2022-09-10T00:17:14.695 DEBUG:boto:Path: /jkcgbj-75/ 2022-09-10T00:17:14.695 DEBUG:boto:Data: 2022-09-10T00:17:14.695 DEBUG:boto:Headers: {} 2022-09-10T00:17:14.696 DEBUG:boto:Host: smithi005.front.sepia.ceph.com:8000 2022-09-10T00:17:14.697 DEBUG:boto:Port: 8000 2022-09-10T00:17:14.697 DEBUG:boto:Params: {} 2022-09-10T00:17:14.698 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:17:14.698 DEBUG:boto:Token: None 2022-09-10T00:17:14.699 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:17:14 GMT /jkcgbj-75/ 2022-09-10T00:17:14.700 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dkcpmCKDg/36gtHE/l2wx2rky9M= 2022-09-10T00:17:14.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:17:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dkcpmCKDg/36gtHE/l2wx2rky9M=', 'Content-Length': '0'} 2022-09-10T00:17:14.704 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:17:14.706 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:17:15.452 DEBUG:boto:Token: None 2022-09-10T00:17:15.455 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:17:14 GMT /jkcgbj-75/ 2022-09-10T00:17:15.456 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dkcpmCKDg/36gtHE/l2wx2rky9M= 2022-09-10T00:17:15.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:17:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dkcpmCKDg/36gtHE/l2wx2rky9M=', 'Content-Length': '0'} 2022-09-10T00:17:15.458 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:17:15.459 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:17:16.628 DEBUG:boto:Token: None 2022-09-10T00:17:16.629 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:17:14 GMT /jkcgbj-75/ 2022-09-10T00:17:16.630 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dkcpmCKDg/36gtHE/l2wx2rky9M= 2022-09-10T00:17:16.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:17:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dkcpmCKDg/36gtHE/l2wx2rky9M=', 'Content-Length': '0'} 2022-09-10T00:17:16.634 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:17:16.635 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:17:17.196 DEBUG:boto:Token: None 2022-09-10T00:17:17.197 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:17:14 GMT /jkcgbj-75/ 2022-09-10T00:17:17.199 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dkcpmCKDg/36gtHE/l2wx2rky9M= 2022-09-10T00:17:17.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:17:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dkcpmCKDg/36gtHE/l2wx2rky9M=', 'Content-Length': '0'} 2022-09-10T00:17:17.202 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:17:17.204 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:17:22.140 DEBUG:boto:Token: None 2022-09-10T00:17:22.141 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:17:14 GMT /jkcgbj-75/ 2022-09-10T00:17:22.143 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dkcpmCKDg/36gtHE/l2wx2rky9M= 2022-09-10T00:17:22.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:17:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dkcpmCKDg/36gtHE/l2wx2rky9M=', 'Content-Length': '0'} 2022-09-10T00:17:22.146 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:17:22.147 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:17:37.585 DEBUG:boto:Token: None 2022-09-10T00:17:37.587 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:17:14 GMT /jkcgbj-75/ 2022-09-10T00:17:37.588 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dkcpmCKDg/36gtHE/l2wx2rky9M= 2022-09-10T00:17:37.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:17:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dkcpmCKDg/36gtHE/l2wx2rky9M=', 'Content-Length': '0'} 2022-09-10T00:17:37.591 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:17:37.592 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:17:42.540 DEBUG:boto:Token: None 2022-09-10T00:17:42.541 DEBUG:boto:StringToSign: PUT Sat, 10 Sep 2022 00:17:14 GMT /jkcgbj-75/ 2022-09-10T00:17:42.542 DEBUG:boto:Signature: AWS VQ6YV8EJ6TF7N01I9BCH:dkcpmCKDg/36gtHE/l2wx2rky9M= 2022-09-10T00:17:42.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Sat, 10 Sep 2022 00:17:14 GMT', 'Authorization': 'AWS VQ6YV8EJ6TF7N01I9BCH:dkcpmCKDg/36gtHE/l2wx2rky9M=', 'Content-Length': '0'} 2022-09-10T00:17:42.546 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-10T00:17:42.547 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-10T00:18:32.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init ... ERROR 2022-09-10T00:18:32.613 INFO:tasks.rgw_multisite_tests:log information for manual testing ... ok 2022-09-10T00:18:32.614 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-10T00:18:32.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-10T00:18:32.626 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-10T00:18:32.683 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-10T00:18:32.683 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-10T00:23:32.682 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T00:23:32.677+0000 7f9539ce0dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T00:28:32.681 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T00:28:32.679+0000 7f9539ce0dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T00:33:32.686 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T00:33:32.676+0000 7f9539ce0dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T00:38:32.695 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T00:38:32.678+0000 7f9539ce0dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T00:43:32.683 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T00:43:32.679+0000 7f9539ce0dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T00:48:32.684 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T00:48:32.681+0000 7f9539ce0dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T00:53:32.689 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T00:53:32.679+0000 7f9539ce0dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T00:58:32.689 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T00:58:32.680+0000 7f9539ce0dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T01:03:32.685 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T01:03:32.682+0000 7f9539ce0dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T01:08:32.686 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T01:08:32.683+0000 7f9539ce0dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T01:08:32.689 INFO:teuthology.orchestra.run.smithi005.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-10T01:08:32.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-10T01:08:32.696 INFO:tasks.rgw_multisite_tests:test low level implementation of s3 notifications ... FAIL 2022-09-10T01:08:32.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-10T01:08:32.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-10T01:08:32.701 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-10T01:08:32.783 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-10T01:08:32.784 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-10T01:13:32.788 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T01:13:32.781+0000 7f1b1c413dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T01:18:32.787 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T01:18:32.782+0000 7f1b1c413dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T01:23:32.788 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T01:23:32.784+0000 7f1b1c413dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T01:28:32.788 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T01:28:32.785+0000 7f1b1c413dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T01:33:32.789 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T01:33:32.787+0000 7f1b1c413dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T01:38:32.790 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T01:38:32.784+0000 7f1b1c413dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T01:43:32.791 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T01:43:32.786+0000 7f1b1c413dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T01:48:32.793 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T01:48:32.787+0000 7f1b1c413dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T01:53:32.792 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T01:53:32.789+0000 7f1b1c413dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T01:58:32.793 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T01:58:32.786+0000 7f1b1c413dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T01:58:32.797 INFO:teuthology.orchestra.run.smithi005.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-10T01:58:32.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-10T01:58:32.838 INFO:tasks.rgw_multisite_tests:test s3 records fetching ... FAIL 2022-09-10T01:58:32.841 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-10T01:58:32.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-10T01:58:32.844 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-10T01:58:32.900 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-10T01:58:32.900 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-10T02:03:32.904 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T02:03:32.900+0000 7f98494bcdc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T02:08:32.906 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T02:08:32.901+0000 7f98494bcdc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T02:13:32.907 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T02:13:32.903+0000 7f98494bcdc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T02:18:32.908 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T02:18:32.900+0000 7f98494bcdc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T02:23:32.907 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T02:23:32.898+0000 7f98494bcdc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T02:28:32.908 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T02:28:32.903+0000 7f98494bcdc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T02:33:32.909 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T02:33:32.905+0000 7f98494bcdc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T02:38:32.909 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T02:38:32.902+0000 7f98494bcdc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T02:43:32.910 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T02:43:32.904+0000 7f98494bcdc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T02:48:32.913 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T02:48:32.905+0000 7f98494bcdc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T02:48:32.917 INFO:teuthology.orchestra.run.smithi005.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-10T02:48:32.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-10T02:48:32.960 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete ... FAIL 2022-09-10T02:48:32.967 INFO:tasks.rgw_multisite_tests:test s3 notification filter on master ... ok 2022-09-10T02:48:32.969 INFO:tasks.rgw_multisite_tests:rgw_multi.tests_ps.test_object_timing ... ok 2022-09-10T02:48:32.969 INFO:tasks.rgw_multisite_tests:test that opaque id set in topic, is sent in notification ... ok 2022-09-10T02:48:32.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-10T02:48:32.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-10T02:48:32.974 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-10T02:48:33.066 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-10T02:48:33.067 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-10T02:53:33.069 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T02:53:33.067+0000 7f0c3d392dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T02:58:33.071 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T02:58:33.064+0000 7f0c3d392dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T03:03:33.071 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T03:03:33.066+0000 7f0c3d392dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T03:08:33.072 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T03:08:33.067+0000 7f0c3d392dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T03:13:33.076 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T03:13:33.069+0000 7f0c3d392dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T03:18:33.075 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T03:18:33.070+0000 7f0c3d392dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T03:23:33.074 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T03:23:33.072+0000 7f0c3d392dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T03:28:33.075 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T03:28:33.069+0000 7f0c3d392dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T03:33:33.076 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T03:33:33.071+0000 7f0c3d392dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T03:38:33.077 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T03:38:33.072+0000 7f0c3d392dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T03:38:33.079 INFO:teuthology.orchestra.run.smithi005.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-10T03:38:33.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-10T03:38:33.111 INFO:tasks.rgw_multisite_tests:test set/get/delete of topic ... FAIL 2022-09-10T03:38:33.114 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-10T03:38:33.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-10T03:38:33.118 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-10T03:38:33.195 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-10T03:38:33.196 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-10T03:43:33.200 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T03:43:33.193+0000 7fdaaf01edc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T03:48:33.200 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T03:48:33.195+0000 7fdaaf01edc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T03:53:33.200 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T03:53:33.196+0000 7fdaaf01edc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T03:58:33.202 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T03:58:33.194+0000 7fdaaf01edc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T04:03:33.202 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T04:03:33.199+0000 7fdaaf01edc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T04:08:33.204 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T04:08:33.197+0000 7fdaaf01edc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T04:13:33.203 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T04:13:33.198+0000 7fdaaf01edc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T04:18:33.204 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T04:18:33.200+0000 7fdaaf01edc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T04:23:33.204 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T04:23:33.201+0000 7fdaaf01edc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T04:28:33.205 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T04:28:33.203+0000 7fdaaf01edc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T04:28:33.208 INFO:teuthology.orchestra.run.smithi005.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-10T04:28:33.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-10T04:28:33.252 INFO:tasks.rgw_multisite_tests:test set topic with endpoint ... FAIL 2022-09-10T04:28:33.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-10T04:28:33.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-10T04:28:33.256 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-10T04:28:33.305 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-10T04:28:33.306 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-10T04:33:33.316 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T04:33:33.304+0000 7f25425c3dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T04:38:33.310 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T04:38:33.306+0000 7f25425c3dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T04:43:33.311 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T04:43:33.307+0000 7f25425c3dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T04:48:33.311 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T04:48:33.304+0000 7f25425c3dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T04:53:33.312 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T04:53:33.306+0000 7f25425c3dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T04:58:33.313 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T04:58:33.307+0000 7f25425c3dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T05:03:33.313 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T05:03:33.309+0000 7f25425c3dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T05:08:33.314 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T05:08:33.310+0000 7f25425c3dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T05:13:33.315 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T05:13:33.312+0000 7f25425c3dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T05:18:33.316 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T05:18:33.309+0000 7f25425c3dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T05:18:33.351 INFO:teuthology.orchestra.run.smithi005.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-10T05:18:33.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-10T05:18:33.381 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification ... FAIL 2022-09-10T05:18:33.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-10T05:18:33.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-10T05:18:33.385 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-10T05:18:33.432 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-10T05:18:33.433 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-10T05:23:33.438 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T05:23:33.431+0000 7f1f73ca5dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T05:28:33.437 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T05:28:33.432+0000 7f1f73ca5dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T05:33:33.438 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T05:33:33.434+0000 7f1f73ca5dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T05:38:33.439 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T05:38:33.435+0000 7f1f73ca5dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T05:43:33.439 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T05:43:33.433+0000 7f1f73ca5dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T05:48:33.450 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T05:48:33.434+0000 7f1f73ca5dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T05:53:33.440 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T05:53:33.436+0000 7f1f73ca5dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T05:58:33.442 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T05:58:33.437+0000 7f1f73ca5dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T06:03:33.442 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T06:03:33.435+0000 7f1f73ca5dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T06:08:33.443 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T06:08:33.436+0000 7f1f73ca5dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T06:08:33.471 INFO:teuthology.orchestra.run.smithi005.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-10T06:08:33.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-10T06:08:33.506 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification on specific events ... FAIL 2022-09-10T06:08:33.508 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-10T06:08:33.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-10T06:08:33.511 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-10T06:08:33.593 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-10T06:08:33.593 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-10T06:13:33.598 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T06:13:33.590+0000 7f91169cedc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T06:18:33.597 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T06:18:33.591+0000 7f91169cedc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T06:23:33.599 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T06:23:33.593+0000 7f91169cedc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T06:28:33.599 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T06:28:33.595+0000 7f91169cedc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T06:33:33.599 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T06:33:33.592+0000 7f91169cedc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T06:38:33.599 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T06:38:33.590+0000 7f91169cedc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T06:43:33.600 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T06:43:33.591+0000 7f91169cedc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T06:48:33.601 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T06:48:33.597+0000 7f91169cedc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T06:53:33.601 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T06:53:33.595+0000 7f91169cedc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T06:58:33.602 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T06:58:33.596+0000 7f91169cedc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T06:58:33.637 INFO:teuthology.orchestra.run.smithi005.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-10T06:58:33.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-10T06:58:33.669 INFO:tasks.rgw_multisite_tests:test set/get/delete of subscription ... FAIL 2022-09-10T06:58:33.672 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-10T06:58:33.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-10T06:58:33.674 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-10T06:58:33.721 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-10T06:58:33.722 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-10T07:03:33.725 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T07:03:33.722+0000 7f05bb257dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T07:08:33.725 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T07:08:33.719+0000 7f05bb257dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T07:13:33.727 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T07:13:33.721+0000 7f05bb257dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T07:18:33.727 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T07:18:33.722+0000 7f05bb257dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T07:23:33.728 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T07:23:33.724+0000 7f05bb257dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T07:28:33.728 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T07:28:33.721+0000 7f05bb257dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T07:33:33.729 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T07:33:33.723+0000 7f05bb257dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T07:38:33.730 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T07:38:33.725+0000 7f05bb257dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T07:43:33.731 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T07:43:33.722+0000 7f05bb257dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T07:48:33.731 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T07:48:33.724+0000 7f05bb257dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T07:48:33.764 INFO:teuthology.orchestra.run.smithi005.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-10T07:48:33.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-10T07:48:33.808 INFO:tasks.rgw_multisite_tests:test radosgw-admin commands ... FAIL 2022-09-10T07:48:33.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-10T07:48:33.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-10T07:48:33.815 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-10T07:48:33.866 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-10T07:48:33.866 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-10T07:53:33.869 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T07:53:33.865+0000 7f043aeb8dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T07:58:33.871 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T07:58:33.867+0000 7f043aeb8dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T08:03:33.871 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T08:03:33.864+0000 7f043aeb8dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T08:08:33.872 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T08:08:33.865+0000 7f043aeb8dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T08:13:33.873 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T08:13:33.867+0000 7f043aeb8dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T08:18:33.874 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T08:18:33.868+0000 7f043aeb8dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T08:23:33.873 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T08:23:33.866+0000 7f043aeb8dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T08:28:33.875 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T08:28:33.871+0000 7f043aeb8dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T08:33:33.875 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T08:33:33.869+0000 7f043aeb8dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T08:38:33.876 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T08:38:33.870+0000 7f043aeb8dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T08:38:33.904 INFO:teuthology.orchestra.run.smithi005.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-10T08:38:33.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-10T08:38:33.945 INFO:tasks.rgw_multisite_tests:test that events are only sent on incremental sync ... FAIL 2022-09-10T08:38:33.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-10T08:38:33.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-10T08:38:33.952 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-10T08:38:34.000 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-10T08:38:34.000 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-10T08:43:34.003 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T08:43:33.996+0000 7fc2c48a7dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T08:48:34.004 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T08:48:34.001+0000 7fc2c48a7dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T08:53:34.006 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T08:53:33.998+0000 7fc2c48a7dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T08:58:34.007 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T08:58:34.000+0000 7fc2c48a7dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T09:03:34.007 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T09:03:33.997+0000 7fc2c48a7dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T09:08:34.008 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T09:08:34.003+0000 7fc2c48a7dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T09:13:34.008 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T09:13:34.004+0000 7fc2c48a7dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T09:18:34.009 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T09:18:34.002+0000 7fc2c48a7dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T09:23:34.011 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T09:23:34.003+0000 7fc2c48a7dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T09:28:34.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-10T09:28:34.047 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T09:28:34.005+0000 7fc2c48a7dc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T09:28:34.047 INFO:teuthology.orchestra.run.smithi005.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-10T09:28:34.051 INFO:tasks.rgw_multisite_tests:test subscriptions for different events ... FAIL 2022-09-10T09:28:34.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-10T09:28:34.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-10T09:28:34.057 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-10T09:28:34.107 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-10T09:28:34.108 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-10T09:33:34.112 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T09:33:34.106+0000 7fa5b462edc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T09:38:34.112 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T09:38:34.107+0000 7fa5b462edc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T09:43:34.114 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T09:43:34.105+0000 7fa5b462edc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T09:48:34.114 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T09:48:34.106+0000 7fa5b462edc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T09:53:34.114 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T09:53:34.108+0000 7fa5b462edc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T09:58:34.115 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T09:58:34.109+0000 7fa5b462edc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T10:03:34.116 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T10:03:34.111+0000 7fa5b462edc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T10:08:34.116 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T10:08:34.108+0000 7fa5b462edc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T10:13:34.118 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T10:13:34.110+0000 7fa5b462edc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T10:18:34.118 INFO:teuthology.orchestra.run.smithi005.stderr:2022-09-10T10:18:34.111+0000 7fa5b462edc0 0 monclient(hunting): authenticate timed out after 300 2022-09-10T10:18:34.167 INFO:teuthology.orchestra.run.smithi005.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-10T10:18:34.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-10T10:18:34.207 INFO:tasks.rgw_multisite_tests:test incremental fetching of events from a subscription ... FAIL 2022-09-10T10:18:34.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-10T10:18:34.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-10T10:18:34.214 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-10T10:18:34.264 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setuser ceph since I am not root 2022-09-10T10:18:34.265 INFO:teuthology.orchestra.run.smithi005.stderr:ignoring --setgroup ceph since I am not root 2022-09-10T10:21:34.812 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-09-10T10:21:34.817 DEBUG:teuthology.task.console_log:Killing console logger for smithi005 2022-09-10T10:21:34.818 DEBUG:teuthology.task.console_log:Killing console logger for smithi080 2022-09-10T10:21:34.818 DEBUG:teuthology.exit:Finished running handlers